- API-users - lists.gbif.org

What happens to previous data after dataset/crawl?
by Roderic Page 29 Aug '16

29 Aug '16

Just wanted to check the consequences of the following dataset operation. Say I have a dataset with 10 occurrences with occurrence ids 1-10. In my local database I now assign those 10 occurrences new identifiers a-j. If I create a new DwCA file for my data and crawl the new archive, my expectation is: 1. Old data with ids 1-10 is deleted from GBIF index 2. New data with ids a-j is indexed So, end result is dataset has 10 occurrences. I'm asking because I know in the past the some datasets have changed identifiers and this has resulted in records with old and new identifiers coexisting in GBIF index, resulting in duplicated data. Obviously it would be nice to have stable, unchanging identifiers for occurrences, but the for data set I'm working with the creators have changed their minds between versions of the data :( Regards, Rod Get Outlook for iOS<https://aka.ms/o0ukef>

3 8

org.gbif.api.vocabulary.License.UNSPECIFIED
by Nozomi James Ytow 19 Aug '16

19 Aug '16

Hi, I got the following exception when getting a dataset in my Java code: org.codehaus.jackson.map.JsonMappingException: Can not construct instance of org.gbif.api.vocabulary.License from String value 'unspecified': value not one of declared Enum instance names at [Source: http://api.gbif.org/v1/dataset/7ddf754f-d193-4cc9-b351-99906754a03b; line: 1, column: 18567] (through reference chain: org.gbif.api.model.registry.Dataset["license"]) Because Java Enum is case sensitive, either is necessary to rename the entry to lower case, to change the value to upper case, or provide conversion code in org.gbif.api.vocabulary.License. Or, of course, modify application codes.... Cheers, James

2 3

Version of GBIF Backbone taxonomy
by Nozomi James Ytow 19 Aug '16

19 Aug '16

Hi, how one can refer version of GBIF Backbone taxonomy? Works using the backbone need to specify its version for comparison of rsults between works. Different backbone may give different conclusion. Or, are older backbones availabe with different datase ID? Cheers, James

2 1

Crawling paused and new taxonomic backbone
by Tim Robertson 09 Aug '16

09 Aug '16

Dear GBIF API users, We are pleased to announce that we are preparing to move a new taxonomic backbone into production. This is the first revision of the backbone for quite some years, which will bring the GBIF backbone up to date with latest data sources, such as the Catalogue of Life 2016 (January edition) along with fixes for the many issues you have reported since 2013. You can read more about the backbone taxonomy rewriting on http://gbif.blogspot.com/2016/04/updating-gbif-backbone.html This requires some significant processing within the GBIF.org system as we'll be rerunning all quality control routines on all data - this means other improvements may also take effect across older data. While GBIF.org will remain operational to users during this time, crawling will be paused. We expect to start this process today, and during next week the changes will be visible in production, after which crawling will resume. Thank you all for your patience, and providing the data, feedback and submitting the issues that have allowed this. We'll report here as changes come into effect. More soon, The GBIF dev team

7 12

typeSpecimens query for species is broken
by Roderic D. M. Page 23 Jul '16

23 Jul '16

According to the docs I should be able to get the type specimens for a species using: http://api.gbif.org/v1/species/5231190/typeSpecimens (the example given on http://www.gbif.org/developer/species ) But this returns {"offset":0,"limit":20,"endOfRecords":true,"results":[]} This species has A LOT of types, see http://www.gbif.org/species/5231190, so something seems amiss... Regards, Rod

2 2

New Portal/Occurrece API functionalities
by Federico Mendez 20 Jun '16

20 Jun '16

Dear GBIFApiers, we are excited to announce some new features that are now supported by there GBIF Portal and Occurrence API: 1. Full text search is now supported. Some examples of this new function are: * Search by collector (recorded by): * http://api.gbif.org/v1/occurrence/search?q=hobern<http://www.gbif.org/occurrence/search?q=hobern> * http://api.gbif.org/v1/occurrence/search?q=desmet * Search by locality shows accurate results for most of the cases: http://api.gbif.org/v1/occurrence/search?q=santa%20ana * This service provides suggestions when a 'similar/corrected' version of the query could show more accurate results: http://api.gbif.org/v1/occurrence/search?q=kajsdkla&spellCheck=true . * This functionality is also supported by the GBIF portal: http://www.gbif.org/occurrence/search?q=cobra<http://www.gbif.org/occurrence/search> 2. Faceted search * The GBIF portal now supports faceted navigation by: basis of record, type status, data set, country, year and month. Please, see this example: http://www.gbif.org/occurrence/search?basis_of_record=FOSSIL_SPECIMEN . * The GBIF API supports faceted results by: basisOfRecord, catalogNumber, classKey, collectionCode, country, datasetKey, decimalLatitude, decimalLongitude, depth, elevation, establishmentMeans, familyKey, genusKey, hasCoordinate, hasGeospatialIssue, institutionCode, issue, kingdomKey, mediaType, month, occurrenceID, orderKey, phylumKey, publishingCountry, recordedBy, recordNumber, repatriated, scientificName, speciesKey, subgenusKey, taxonKey, typeStatus and year. * The occurrence API supports pagination request in facets using the parameters facetLimit and facetOffset, see this example: * First three country counts/facets: http://api.gbif.org/v1/occurrence/search?facet=COUNTRY&COUNTRY.facetLimit=3… * Next three country counts/facets: http://api.gbif.org/v1/occurrence/search?facet=COUNTRY&COUNTRY.facetLimit=3… 3. A new repatriation filter allows to search for records whose publishing country is different to the country where the record was recorded in. For example, the following link shows the repatriated data of Costa Rica:http://api.gbif.org/v1/occurrence/search?COUNTRY=CR&REPATRIATED=true . Be aware that this is still in a experimental phase and its definition could change in the future. Please refer to the API documentation http://www.gbif.org/developer/occurrence and don't hesitate to contact us if you have further questions! Best regards, GBIFS Informatics.

7 11

birdhouse meets GBIF
by Nils Hempelmann 08 Jun '16

08 Jun '16

Dear all Here is an current problem to solve :-) for the birdhouse WPS and GBIF developers. There is a species distribution process in the birdhouse WPS, processing Climate data based on occurrence coordinates from GBIF. Currently the GBIF data are given to the process via an url of the GBIF csv (which had to be generated manually before). I was implementing a direct occurrence search from birdhouse to GBIF with pygbif. Worked fine BUT: there is a limit of 300 records which is far to less to train the Species distribution model. So here the question: Is there a way to request the species occurrence coordinates somehow directly (e.g. like a WPS request)? (The previous conversation is following) And some link for the background informations: Birdhouse: http://birdhouse.readthedocs.io/en/latest/ GBIF: http://www.gbif.org/ Species distribution process: http://flyingpigeon.readthedocs.io/en/latest/tutorials/sdm.html Birdhouse architecture: https://github.com/bird-house/birdhouse-docs/blob/master/slides/birdhouse-a… Merci Nils -------- Forwarded Message ------- Subject: Re: pygbif for occurrence coordinates Date: Thu, 2 Jun 2016 09:00:49 -0300 From: Mauro Cavalcanti <maurobio(a)gmail.com> To: Nils Hempelmann <info(a)nilshempelmann.de> CC: wps-dev(a)lists.dkrz.de, Carsten Ehbrecht <ehbrecht(a)dkrz.de>, Wolfgang.Falk(a)lwf.bayern.de, Scott Chamberlain <myrmecocystus(a)gmail.com> Nils, No, there is no other way to get at once a larger number of records using the GBIF API (although one could be able to achieve this by sequentially querying the server using "batches" of 200 or 300 records each). As I said, there are good operational reasons for the GBIF developers to have imposed such limit. As of your other question, it should better be put to the GBIF developers themselves (in the discussion list, so that we can all benefit from the answers! :-)) With warmest regards, -- Dr. Mauro J. Cavalcanti E-mail: maurobio(a)gmail.com <mailto:maurobio@gmail.com> Web: http://sites.google.com/site/maurobio Em 02/06/2016 08:49, "Nils Hempelmann" <info(a)nilshempelmann.de <mailto:info@nilshempelmann.de>> escreveu: Hi Mauro Oh ... No way to set it unlimited? The 'manual via browser' option and parsing the returning csv is the current status. or any alternative to pygbif? If I understood it correctly, GBIF database is organized as Web Server, so I gues there should be a way to connect to the birdhouse WPS, am I right? (put the web-dev list in copy) Merci Nils On 02/06/2016 12:40, Mauro Cavalcanti wrote: > > Nils, > > That's a limit imposed (for good operational reasons) by the GBIF > API. If you want a larger number of records, you'll have to > download them "manually" (that is, via browser) and then parse > locally the csv file returned from the GBIF server. > > Hope this helps. > > Best regards, > > -- > Dr. Mauro J. Cavalcanti > E-mail: maurobio(a)gmail.com <mailto:maurobio@gmail.com> > Web: http://sites.google.com/site/maurobio > > Em 02/06/2016 04:48, "Nils Hempelmann" <info(a)nilshempelmann.de > <mailto:info@nilshempelmann.de>> escreveu: > > Hi Scott > > works fine. Thanks a lot. Just have a question for the search > limits: > > The maximal records seems to be limited to 300. Is that on > purpose? > And requesting more than 200000 gives a 'bad request' > > Merci > Nils > > > In [68]: len( occurrences.search(taxonKey=key, > limit=100)['results']) > Out[68]: 100 > > In [69]: len( occurrences.search(taxonKey=key, > limit=300)['results']) > Out[69]: 300 > > In [70]: len( occurrences.search(taxonKey=key, > limit=3000)['results']) > Out[70]: 300 > > In [71]: len( occurrences.search(taxonKey=key, > limit=200000)['results']) > Out[71]: 300 > > In [72]: len( occurrences.search(taxonKey=key, > limit=200001)['results']) > --------------------------------------------------------------------------- > HTTPError Traceback (most > recent call last) > <ipython-input-72-2f7d7b4ccba0> in <module>() > ----> 1 len( occurrences.search(taxonKey=key, > limit=200001)['results']) > > /home/nils/.conda/envs/birdhouse/lib/python2.7/site-packages/pygbif/occurrences/search.pyc > in search(taxonKey, scientificName, country, > publishingCountry, hasCoordinate, typeStatus, recordNumber, > lastInterpreted, continent, geometry, recordedBy, > basisOfRecord, datasetKey, eventDate, catalogNumber, year, > month, decimalLatitude, decimalLongitude, elevation, depth, > institutionCode, collectionCode, hasGeospatialIssue, issue, q, > mediatype, limit, offset, **kwargs) > 251 'collectionCode': collectionCode, > 'hasGeospatialIssue': hasGeospatialIssue, > 252 'issue': issue, 'q': q, 'mediatype': > mediatype, 'limit': limit, > --> 253 'offset': offset}, **kwargs) > 254 return out > > /home/nils/.conda/envs/birdhouse/lib/python2.7/site-packages/pygbif/gbifutils.pyc > in gbif_GET(url, args, **kwargs) > 17 def gbif_GET(url, args, **kwargs): > 18 out = requests.get(url, params=args, > headers=make_ua(), **kwargs) > ---> 19 out.raise_for_status() > 20 stopifnot(out.headers['content-type']) > 21 return out.json() > > /home/nils/.conda/envs/birdhouse/lib/python2.7/site-packages/requests/models.pyc > in raise_for_status(self) > 842 > 843 if http_error_msg: > --> 844 raise HTTPError(http_error_msg, response=self) > 845 > 846 def close(self): > > HTTPError: 400 Client Error: Bad Request for url: > http://api.gbif.org/v1/occurrence/search?taxonKey=2882316&limit=200001&offs… > > In [73]: > > > On 02/06/2016 02:22, Scott Chamberlain wrote: >> Fixes to docs and fix to download_get are up on Github now. >> Will push new version to pypi soon. Let me know if some thing >> still don't work for you after reinstalling from github >> >> S >> >> On Wed, Jun 1, 2016 at 4:51 PM Nils Hempelmann >> <info(a)nilshempelmann.de <mailto:info@nilshempelmann.de>> wrote: >> >> Hi Scott and Mauro >> >> Mauro sended me a snipped of code which worked fine: >> https://github.com/bird-house/flyingpigeon/blob/develop/scripts/pygbif_occu… >> >> I found the example here: >> https://github.com/maurobio/pygbif#occurrence-data >> >> and here: >> https://github.com/sckott/pygbif#occurrences-module >> >> Thanks a lot very great. That enables a lot ;-) >> I ll keep you posted >> >> merci >> Nils >> >> >> >> >> >> On 02/06/2016 01:45, Scott Chamberlain wrote: >>> And where are those example from exactly? I don't see >>> those examples searching the repo (which includes all >>> docs). >>> >>> `pygbif.name_suggest` wouldn't work because >>> `name_suggest` is a method in the `species` module, so >>> you'd have to do pygbif.species.name_suggest, or `from >>> pygbif import species`, then `species.name_suggest` >>> >>> Looks like occ.get(taxonKey = 252408386) is a >>> documentation bug, that should be `key` instead of >>> `taxonKey`, a copy paste error. will fix that. >>> >>> The `occ.download_get` call has a small bug, will fix that >>> >>> All other calls work for me >>> >>> S >>> >>> On Wed, Jun 1, 2016 at 4:34 PM Scott Chamberlain >>> <myrmecocystus(a)gmail.com >>> <mailto:myrmecocystus@gmail.com>> wrote: >>> >>> Hi, >>> >>> What version of pygbif are you on? And what version >>> of Python> >>> >>> Best, Scott >>> >>> On Wed, Jun 1, 2016 at 4:02 PM Nils Hempelmann >>> <info(a)nilshempelmann.de >>> <mailto:info@nilshempelmann.de>> wrote: >>> >>> Hi Mauro and Scott >>> >>> Was checking out pygbif. seems to be a very >>> useful tool. >>> >>> Can you help me with the syntax (or forward me >>> to the appropriate person >>> ;-) ) >>> The given snippets of code are outdated. >>> >>> I am basically just looking for the occurrence >>> coordinates: >>> here is my first try : >>> >>> import pygbif >>> occ = >>> pygbif.occurrences.search(scientificName='Fagus >>> sylvatica') >>> occ['count'] >>> >>> ... and further? ;-) >>> >>> the examples in the docu are throwing errors: >>> >>> key= pygbif.name_suggest(q='Helianthus >>> annuus',rank='species')['key'] >>> pygbif.search(taxonKey=key[0]['key'],limit=2) >>> >>> from pygbif import occurrences as occ >>> occ.search(taxonKey = 3329049) >>> occ.get(taxonKey = 252408386) >>> occ.count(isGeoreferenced = True) >>> occ.download_list(user = "sckott", limit = 5) >>> occ.download_meta(key = "0000099-140929101555934") >>> occ.download_get("0000099-140929101555934") >>> >>> >>> Thanks >>> Nils >>> >> >

4 8

Re: [API-users] Is there any NEO4J or graph-based driver for this API ?
by Jan Legind 31 May '16

31 May '16

Dear Juan, Unfortunately we have no tool for creating these kind of SQL like queries to the portal. I am sure you are aware that the filters in the occurrence search pages can be applied in combination in numerous ways. The API can go even further in this regard[1], but it not well suited for retrieving occurrence records since there is a 200.000 records ceiling making it unfit for species exceeding this number. There is going be updates to the pygbif package[2] in the near future that will enable you to launch user downloads programmatically where a whole list of different species can be used as a query parameter as well as adding polygons.[3] In the meantime, Mauro’s suggestion is excellent. If you can narrow your search down until it returns a manageable download (say less than 100 million records), importing this into a database should be doable. From there, you can refine using SQL queries. Best, Jan K. Legind, GBIF Data manager [1] http://www.gbif.org/developer/occurrence#search [2] https://github.com/sckott/pygbif [3] https://github.com/jlegind/GBIF-downloads From: API-users [mailto:api-users-bounces@lists.gbif.org] On Behalf Of Mauro Cavalcanti Sent: 30. maj 2016 14:06 To: Juan M. Escamilla Molgora Cc: api-users(a)lists.gbif.org Subject: Re: [API-users] Is there any NEO4J or graph-based driver for this API ? Hi, One solution I have successfully adopted for this is to download the records (either "manually" via browser or, yet better, using a Python script using the fine pygbif library), storing them into a MySQL or SQLite database and then perform the relational queries. I can provide examples if you are interested. Best regards, 2016-05-30 8:59 GMT-03:00 Juan M. Escamilla Molgora <j.escamillamolgora(a)lancaster.ac.uk<mailto:j.escamillamolgora@lancaster.ac.uk>>: Hola, Is there any API for making relational queries like taxonomy, location or timestamp? Thank you and best wishes Juan _______________________________________________ API-users mailing list API-users(a)lists.gbif.org<mailto:API-users@lists.gbif.org> http://lists.gbif.org/mailman/listinfo/api-users -- Dr. Mauro J. Cavalcanti E-mail: maurobio(a)gmail.com<mailto:maurobio@gmail.com> Web: http://sites.google.com/site/maurobio

3 2

Is there any NEO4J or graph-based driver for this API ?
by Juan M. Escamilla Molgora 31 May '16

31 May '16

Hola, Is there any API for making relational queries like taxonomy, location or timestamp? Thank you and best wishes Juan

2 1

timeframe on implementation of dataset "subtype" parameter?
by Stoner, Dan 28 May '16

28 May '16

Hi, Just wondering if dataset subtype will become available in the API anytime soon. This would be very helpful to me as I am looking for SPECIMEN occurrence datasets. The doc that mentions "subtype": http://www.gbif.org/developer/registry#datasetSearch Description: "Not yet implemented, but will eventually allow filtering of datasets by their dataset subtype as given in our DatasetSubtype enum." DatasetSubtype: http://gbif.github.io/gbif-api/apidocs/org/gbif/api/vocabulary/DatasetSubty… Thanks, Dan Stoner iDigBio / ACIS Laboratory University of Florida

2 1