- API-users - lists.gbif.org

Improvements to GBIF data downloads
by Matthew Blissett 13 Aug '19

13 Aug '19

Hi all, Some improvements to data downloads from GBIF.org are new ready for use: • The "simple CSV" format download includes four additional columns: verbatimScientificName, stateProvince, individualCount and occurrenceStatus. The first makes it clear when GBIF processing has replaced a scientific name, such as for occurrences using recently published names. The last two help with filtering absence data. Scripts accessing columns by number (rather than header) will probably need to be updated. • Downloading zip files now supports HTTP Range requests / partial requests. This allows resuming an interrupted download (using CLI tools like curl or wget, or some browser extensions). It may also help when using Google or Amazon's cloud tools. Due to limitations in the length of a URL, these two changes are only available through API requests, not through the GBIF.org website: • When using the API, it's now possible for a download query predicate to include up to 101,000 terms — for example, 100,000 taxon keys, catalogue numbers, collection codes or similar. It's best to use the "in" predicate to make these requests: "predicate": { "type":"in", "key":"CATALOG_NUMBER", "values":["cat1","cat2","cat3"] } • It's also possible to query using more complex polygons, containing up to 10,000 points. Some polygons are slow to process, so if we regularly reach the capacity of our systems we may review this limit. If you spot any issues with these changes, please let us know — by mail to this list, through the feedback button on the website, or to the portal-feedback repository on GitHub: https://github.com/gbif/portal-feedback/issues Thanks, Matt Blissett

1 0

FW: [datacite-allusers] Delays in DOI registrations and updates
by Tim Robertson 02 Aug '19

02 Aug '19

Dear GBIF community Please be aware of this communication from DataCite which will likely cause some disruption on GBIF services with relation to DOI issuing. Many thanks, Tim From: Martin Fenner <martin.fenner(a)datacite.org> Reply-To: "allusers(a)datacite.org" <allusers(a)datacite.org> Date: Friday, 2 August 2019 at 08.22 To: DataCite Allmembers <allmembers(a)datacite.org>, DataCite Allusers <allusers(a)datacite.org> Subject: [datacite-allusers] Delays in DOI registrations and updates Dear DataCite members and users, processing of DOI registrations is currently delayed by up to 12 hours, affecting registration in the handle system and inclusion in the search index. This is caused by an unusually very large number of maintenance jobs running in our system, and we expect this to be resolved by the end of the day. This is a delay in processing, all DOI registrations and updates are successfully captured by our system. Requesting or searching DOI metadata is not affected. Please check https://status.datacite.org for updated status information, or subscribe to email updates at this site. The reason for the very large number of maintenance jobs is two important updates to better handle malformed dates in DOI metadata and to prepare for affiliation identifier support in the upcoming schema 4.3. The load on the system for these two tasks is much higher than expected, causing the delays described above. I sincerely apologize for the inconvenience this is causing, and going forward we will be better in giving advance notice with scheduled maintenances for situations like this. What is happening today is unrelated to the service incident we reported on July 24 (https://status.datacite.org/incidents/d00gnm05ps3k). That incident was caused by a very high number (> 1 million in 24 hours) of DOI registration requests by a single client, temporarily overloading the system. To prevent this from happening in the future, we have this Monday implemented rate-limiting of 3,000 requests per IP address within 5 min, which translates to 10 requests per second or 36,000 requests per hour per IP address. Please reach out to support(a)datacite.org<mailto:support@datacite.org> if you need higher rate-limits temporarily or generally, and we can adjust the rate-limits for your IP addresses. This is the first time we implemented rate-limiting, but we had to take this step to make sure DOI registrations for all users are not affected by high numbers of requests by a single user. We have also increased the number of servers and improved monitoring to better handle these kinds of situations in the future. Please reach out to me via support(a)datacite.org<mailto:support@datacite.org> if you have any comments or feedback. Kind regards, Martin Fenner -- Martin Fenner DataCite Technical Director http://orcid.org/0000-0003-1419-2405 -- You received this message because you are subscribed to the Google Groups "DataCite Allusers" group. To unsubscribe from this group and stop receiving emails from it, send an email to allusers+unsubscribe(a)datacite.org<mailto:allusers+unsubscribe@datacite.org>. To view this discussion on the web visit https://groups.google.com/a/datacite.org/d/msgid/allusers/CAG9q6AbF31-QENqc…<https://groups.google.com/a/datacite.org/d/msgid/allusers/CAG9q6AbF31-QENqc…>.

1 0

projections and occurrence density data
by Daniel McGlinn 23 Jul '19

23 Jul '19

Dear list, My group has been developing some tools that depend on the occurrence density data that GBIF provides through its API: https://www.gbif.org/en/developer/maps We have been using the R function rgbif::map_fetch to query the api. The data is returned faithfully but we are experiencing some issues with projections of the raster data. I have posted a reproducible example in R here: https://github.com/ropensci/rgbif/issues/366 . But briefly the issue is that EPSG:3857 that is downloaded from the API has an extent from -180 to 180 in the x-axis and -86 to 86 in the y-axis but when converting longitude latitude coordinates into the EPSG:3857 coordinate reference system the spatial domain is much larger (I think because it is defined in meters rather than degrees). What am I missing here? I am still a neophyte when it comes to projections so I apologize if this an obvious question. One other quick point is that the GBIF maps API does not seem to serve up the same spatial domain for each projection that is possible. Specifically compare the following two queries to the api: https://api.gbif.org/v2/map/occurrence/density/0/0/0@1x.png?srs=EPSG:3857 https://api.gbif.org/v2/map/occurrence/density/0/0/0@1x.png?srs=EPSG:4326 Note that EGSG:4326 is on a smaller spatial domain that does not include the whole Earth. Maybe this is by design but this webpage led me to believe differently: https://tile.gbif.org/ui/ ' Thanks in advance for your help! Dan -- Daniel J. McGlinn, PhD Assistant Professor mailing address: Department of Biology College of Charleston 66 George Street Charleston, SC 29424 https://www.mcglinnlab.org physical address: Rita Hollings Science Center (office: 239, lab: 292) 58 Coming Street, Suite 245-255 Charleston, SC 29401 office: 843-953-0190

1 0

Re: [API-users] rgbif - species match with higher rank
by Tim Robertson 09 Jul '19

09 Jul '19

Thanks Kathryn This happens when you provide a classification where the terminal taxa is not known to the backbone taxonomy. What it returns is the higher node in the tree that matches, and stating it with the “HIGHER_RANK” flag. Periodically we rebuild the backbone using the best available checklists. If you find cases where this happens the best approach today would be to help register good checklists through which we can build the backbone. The next backbone build is likely to happen within the next 2-3 months. I hope this helps. Best wishes, Tim From: API-users <api-users-bounces(a)lists.gbif.org> on behalf of Kathryn Reynolds <K.Reynolds(a)cabi.org> Date: Tuesday, 9 July 2019 at 10.59 To: "api-users(a)lists.gbif.org" <api-users(a)lists.gbif.org> Subject: [API-users] rgbif - species match with higher rank Hi All, I am using the rgbif package to match up names in a database with the accepted name on gbif (name_lookup). In the output for some instances, the data returned is for the species, and the species exists on GBIF. However, the match type is “higher rank” (see attached file for a sample). I was wondering if anyone could explain to me why this is the case? It seems to be happening most often with bacterial or parasitic species… Thanks in advance, Kathryn Reynolds Junior Data Analyst CABI Head Office Nosworthy Way Wallingford Oxfordshire OX10 8DE United Kingdom Telephone: +44 (0)1491 8293<tel:+44%201491%20829305>58 Email: K.Reynolds<mailto:K.Reynolds@cabi.org>@cabi.org Visit us at www.cabi.org<http://www.cabi.org/> CABI improves people's lives worldwide by providing information and applying scientific expertise to solve problems in agriculture and the environment P Think Green - don't print this email unless you really need to ************************************************************************ The information contained in this e-mail and any files transmitted with it is confidential and is for the exclusive use of the intended recipient. If you are not the intended recipient please note that any distribution, copying or use of this communication or the information in it is prohibited. Whilst CAB International trading as CABI takes steps to prevent the transmission of viruses via e-mail, we cannot guarantee that any e-mail or attachment is free from computer viruses and you are strongly advised to undertake your own anti-virus precautions. If you have received this communication in error, please notify us by e-mail at cabi(a)cabi.org or by telephone on +44 (0)1491 832111 and then delete the e-mail and any copies of it. CABI is an International Organization recognised by the UK Government under Statutory Instrument 1982 No. 1071... **************************************************************************

1 0

rgbif - species match with higher rank
by Kathryn Reynolds 09 Jul '19

09 Jul '19

Hi All, I am using the rgbif package to match up names in a database with the accepted name on gbif (name_lookup). In the output for some instances, the data returned is for the species, and the species exists on GBIF. However, the match type is "higher rank" (see attached file for a sample). I was wondering if anyone could explain to me why this is the case? It seems to be happening most often with bacterial or parasitic species... Thanks in advance, Kathryn Reynolds Junior Data Analyst CABI Head Office Nosworthy Way Wallingford Oxfordshire OX10 8DE United Kingdom Telephone: +44 (0)1491 8293<tel:+44%201491%20829305>58 Email: K.Reynolds<mailto:K.Reynolds@cabi.org>@cabi.org Visit us at www.cabi.org<http://www.cabi.org/> CABI improves people's lives worldwide by providing information and applying scientific expertise to solve problems in agriculture and the environment P Think Green - don't print this email unless you really need to ************************************************************************ The information contained in this e-mail and any files transmitted with it is confidential and is for the exclusive use of the intended recipient. If you are not the intended recipient please note that any distribution, copying or use of this communication or the information in it is prohibited. Whilst CAB International trading as CABI takes steps to prevent the transmission of viruses via e-mail, we cannot guarantee that any e-mail or attachment is free from computer viruses and you are strongly advised to undertake your own anti-virus precautions. If you have received this communication in error, please notify us by e-mail at cabi(a)cabi.org or by telephone on +44 (0)1491 832111 and then delete the e-mail and any copies of it. CABI is an International Organization recognised by the UK Government under Statutory Instrument 1982 No. 1071... **************************************************************************

1 0

Get image from species in Api
by Magdiel Arias 08 Jun '19

08 Jun '19

Hello Gbif staff, I'm making an app for my school and i was wondering if there is another consult in the api besides occurrence that can provide me image, for example: Search: Carcharhiniformes: and get every species in this order with images to see the specie. While delimiting to Mexico only, thanks for the advice, amazing api.

2 1

Some data flaws
by Adriaan van Os 14 May '19

14 May '19

Some data flaws (not sure where to report) 1. Chloroclystis v-ata has no occurrence data, probably due to wrong naming - Chloroclystis v-ata (Haworth, 1809) <https://www.gbif.org/species/4523887> was deleted June 20, 2018, So, the species with the right name was deleted - Chloroclystis vata <https://www.gbif.org/species/1989428> has the wrong name but does exist in the backbone, although without occurrence data 2. Pempelia albariella <https://www.gbif.org/species/4531279> has no GBIF occurrence data with coordinates, even if it occurs <http://www.lepiforum.de/lepiwiki.pl?Pempelia_Albariella> in e.g the South of France, where I photographed it. Well, this could be due to the species being quite rare and/or difficult to identify. Regards, Adriaan van Os

2 1

Requesting Occurrence Data for Large List of Species
by Benjamin Feinsilver 17 Apr '19

17 Apr '19

Hello, If I have a list of around 3,000 species, and I would like to request occurrence data for each species, is it more efficient to use the Search or Download API? If using the Download API, could I include the list of species in an external query file and use the "in" predicate? For example: { "creator":"userName", "notification_address": ["userName(a)example.org"], "predicate": { "type":"in", "key":"SCIENTIFIC_NAME", "values":["cat1","cat2","cat3"] } } Thanks, Ben

3 6

how to get sampling-event data via API
by Otsubo, Takayuki 25 Mar '19

25 Mar '19

Hi, Are there any APIs that provide sampling-event data? We would like to get its details as we can for occurrence data. Best Regards, Tak

1 0

Re: [API-users] [gbif/gbif-api] Any place where api's response structures are explanained? (#41)
by Markus Döring 23 Feb '19

23 Feb '19

Hi Javier, you are right, we are lacking a documentation on our API models on our site. The best and probably only place you can find something to read is the java documentation which is here: https://github.com/gbif/gbif-api/tree/master/src/main/java/org/gbif/api/mod… For example the checklist dataset metrics you have mentioned is this: https://github.com/gbif/gbif-api/blob/master/src/main/java/org/gbif/api/mod… ... but it lacks documentation for the properties you were mentioning. "nub" is an alias for "backbone" in GBIF. Hence nubMatchingCount is the count of name usages that can be matched to the GBIF backbone, i.e. do have a corresponding name in the backbone. Similar colMatchingCount is the number of usages that can be matched to the Catalogue of Life. The xxxCoveragePct properties are the same just expressed as percentages compared to the total of the dataset. Hope that helps a little. If you have any further questions please use our API mailing list which I have copied in here to get help. Regards, Markus > On 22. Feb 2019, at 18:17, Javier Seixas <notifications(a)github.com> wrote: > > Hello! > > I'm looking for GBIF API's responses documentation. If this is not the place to ask, please, address me to the correct one. > > I'm currently working with the GBIF API, and I don't understand some of the responses. For example, when asking for dataset metrics GET /v1/dataset/d7dddbf4-2cf0-4f39-9b2a-bb099caae36c/metrics I get that response: > > { > > "key": 210029 > , > > "datasetKey": "d7dddbf4-2cf0-4f39-9b2a-bb099caae36c" > , > > "usagesCount": 5858142 > , > > "synonymsCount": 2361744 > , > > "distinctNamesCount": 5844412 > , > > "nubMatchingCount": 3805144 > , > > "colMatchingCount": 2845782 > , > > "nubCoveragePct": 64 > , > > "colCoveragePct": 48 > , > > "countByConstituent" > : , > > "countByKingdom" > : { > > "ANIMALIA": 1529953 > , > > "PLANTAE": 523372 > , > > "FUNGI": 201853 > , > > "CHROMISTA": 15357 > , > > "BACTERIA": 10847 > , > > "PROTOZOA": 7063 > , > > "INCERTAE_SEDIS": 787 > , > > "ARCHAEA": 439 > , > > "VIRUSES": 119 > > }, > > ... > > } > > What is the meaning of properties such as nubMatchingCount, colMatchingCount, nubCoveragePct, etc?? In the website I can see the API documentation for making request, not explaining responses though. > > I will appreciate any help Thanks! > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub, or mute the thread. >

1 0