- IPT - lists.gbif.org

GBIF.org supports recordedByID and identifiedByID
by Tim Robertson 06 Apr '20

06 Apr '20

Dear all Last week GBIF.org enabled two new filters to allow search using recordedByID and identifiedByID fields. Datasets such as iNaturalist support the use of ORCID IDs for people. Those users who have linked their account in iNaturalist can now find records in GBIF using their ORCID - e.g.: https://www.gbif.org/occurrence/search?recorded_by_id=https:~2F~2Forcid.org… Similarly, users in the Danish SvampeAtlas can find records they've submitted but were identified by a particular user: https://www.gbif.org/occurrence/taxonomy?recorded_by_id=https:~2F~2Forcid.o… We recommend using well-controlled identifiers for people / agents, such as ORCID and WikiData identifiers, but will index any value filled in. The fields are included in the DwC-A downloads from GBIF.org searches. We now plan to enable this for data publishers using the IPT. I have made a new release of the Occurrence core schema and enabled it for IPTs in TEST mode only [1]. I would appreciate if an IPT user could help test this with me. Using an IPT that is in test mode: 1) In Admin / Core and Extensions you should be able to hit "update" 2) Following update, you should see a note on "Occurrence" stating that the fields are available 3) You can then map occurrence records using this term 4) If you register the test dataset in GBIF it should index on https://www.gbif-uat.org and the searches can be done in the test environment. If someone is able to test this, please can you reply to all so we know it is underway? Many thanks, Tim [1] http://rs.gbif.org/sandbox/core/dwc_occurrence_2020-03-30.xml

2 3

IPT mailing list submissions
by Denis Beina 03 Apr '20

03 Apr '20

1 0

coreid (lowercase "i") vs coreId in meta.xml - schema validation
by Stoner, Dan F 12 Dec '19

12 Dec '19

I found some oddities and I am not exactly sure where to go next. We are noticing the following while processing meta.xml in darwin core archives produced by IPT (and other servers): Schema validation failed, continuing unvalidated XMLSyntaxError: Element '{http://rs.tdwg.org/dwc/text/}coreid': This element is not expected. Expected is ( {http://rs.tdwg.org/dwc/text/}coreId ) It seems like most consumers are not actually validating meta.xml using the schema, and the producers are generating files out of compliance with the schema. Most of the Darwin Core archives I have manually inspected and tried to validate contain meta.xml with lowercase "i" in coreid despite the Standard indicating capital "I" in coreId. I poked at the GBIF Darwin Core Validator 3 code repo and found this: schema.meta=https://raw.githubusercontent.com/tdwg/dwc/master/standard/docu… The first link leads to 404, the second leads to an xsd that contains the proper coreId. So maybe the Validator is not being "strict" about validation against the schema? Digging around I find some "historic" evidence of changes that occurred when tdwg migrated from google code to github and moved their services around, and that GBIF might have "grabbed a copy" from somewhere while waiting for the public site to become available again. The current "published" schema contains coreId: http://rs.tdwg.org/dwc/text/tdwg_dwc_text.xsd The current "master" branch contains coreId: https://github.com/tdwg/dwc/blob/master/docs/text/tdwg_dwc_text.xsd The "gh-pages" branch (one would think this is being used to generate the website, but apparently not) includes the lowercase coreid: https://github.com/tdwg/dwc/blob/gh-pages/text/tdwg_dwc_text.xsd So I am wondering why the GBIF Validator isn't noticing this. And why is IPT generating meta.xml that does not agree with the schema? Reference: http://rs.tdwg.org/dwc/text/ 2.1.2 Elements Element Description <core> An <archive> must contain exactly one <core> element, representing the data entity (the actual file and its column header mappings to Darwin Core terms) upon which records are based. If extensions are being used, each record in the core data must have a unique identifier. The field for this identifier must be specified in an explicit <id> field in order to associate extension records with the core record. <extension> An <archive> may define zero or more <extension> elements, each representing an individual extension entity directly related to the core. In addition to the general file attributes described below, every extension entity must have an explicit <coreId> field to relate the extension record to a row in the core entity. The extension itself does not have to have a unique ID field and many rows can point to the same core record. Thanks! Dan Stoner iDigBio / ACIS Laboratory University of Florida

2 2

FW: [datacite-allusers] Delays in DOI registrations and updates
by Tim Robertson 02 Aug '19

02 Aug '19

Dear GBIF community Please be aware of this communication from DataCite which will likely cause some disruption on GBIF services with relation to DOI issuing. Many thanks, Tim From: Martin Fenner <martin.fenner(a)datacite.org> Reply-To: "allusers(a)datacite.org" <allusers(a)datacite.org> Date: Friday, 2 August 2019 at 08.22 To: DataCite Allmembers <allmembers(a)datacite.org>, DataCite Allusers <allusers(a)datacite.org> Subject: [datacite-allusers] Delays in DOI registrations and updates Dear DataCite members and users, processing of DOI registrations is currently delayed by up to 12 hours, affecting registration in the handle system and inclusion in the search index. This is caused by an unusually very large number of maintenance jobs running in our system, and we expect this to be resolved by the end of the day. This is a delay in processing, all DOI registrations and updates are successfully captured by our system. Requesting or searching DOI metadata is not affected. Please check https://status.datacite.org for updated status information, or subscribe to email updates at this site. The reason for the very large number of maintenance jobs is two important updates to better handle malformed dates in DOI metadata and to prepare for affiliation identifier support in the upcoming schema 4.3. The load on the system for these two tasks is much higher than expected, causing the delays described above. I sincerely apologize for the inconvenience this is causing, and going forward we will be better in giving advance notice with scheduled maintenances for situations like this. What is happening today is unrelated to the service incident we reported on July 24 (https://status.datacite.org/incidents/d00gnm05ps3k) That incident was caused by a very high number (> 1 million in 24 hours) of DOI registration requests by a single client, temporarily overloading the system. To prevent this from happening in the future, we have this Monday implemented rate-limiting of 3,000 requests per IP address within 5 min, which translates to 10 requests per second or 36,000 requests per hour per IP address. Please reach out to support(a)datacite.org<mailto:support@datacite.org> if you need higher rate-limits temporarily or generally, and we can adjust the rate-limits for your IP addresses. This is the first time we implemented rate-limiting, but we had to take this step to make sure DOI registrations for all users are not affected by high numbers of requests by a single user. We have also increased the number of servers and improved monitoring to better handle these kinds of situations in the future. Please reach out to me via support(a)datacite.org<mailto:support@datacite.org> if you have any comments or feedback. Kind regards, Martin Fenner -- Martin Fenner DataCite Technical Director http://orcid.org/0000-0003-1419-2405 -- You received this message because you are subscribed to the Google Groups "DataCite Allusers" group. To unsubscribe from this group and stop receiving emails from it, send an email to allusers+unsubscribe(a)datacite.org<mailto:allusers+unsubscribe@datacite.org>. To view this discussion on the web visit https://groups.google.com/a/datacite.org/d/msgid/allusers/CAG9q6AbF31-QENqc…<https://groups.google.com/a/datacite.org/d/msgid/allusers/CAG9q6AbF31-QENqc…>.

1 0

New IPT version: 2.4.0.
by Matthew Blissett 01 Aug '19

01 Aug '19

Dear IPT users, A new version of the IPT has been released, version 2.4.0. This version includes security updates from Struts and Jackson (code libraries the IPT uses). For this reason, all users should upgrade. It also includes support for DataCite's new API for issuing DOIs, since the old API has been switched off. The new version is available for download herehttps://repository.gbif.org/content/groups/gbif/org/gbif/ipt/2.4.0/ipt-… <http://repository.gbif.org/content/groups/gbif/org/gbif/ipt/2.3.4/ipt-2.3.4…> . New CentOS packages and Docker containers have also been generated, please see https://github.com/gbif/ipt/blob/master/package/rpm/README.md and https://github.com/gbif/docker-ipt. Please read the release notes, particularly if you are not upgrading from the previous release, 2.3.6:https://github.com/gbif/ipt/wiki/IPTReleaseNotes233.wiki Sincerely, Matthew Blissett GBIF Secretariat

2 3

Re: [IPT] EZID, ARK and DOI
by Gamble, Beth 08 Jul '19

08 Jul '19

Hello Tim, The Smithsonian Institution National Museum of Natural History (NMNH) uses EZIDs as our unique identifiers. What EZID capability are you removing from IPT? Beth :-) Beth Gamble Senior Systems Analyst IT - Informatics Office w 202.633.0851 e gambleb(a)si.edu SMITHSONIAN INSTITUTION NATIONAL MUSEUM OF NATURAL HISTORY Facebook | Twitter | Instagram -----Original Message----- Date: Wed, 19 Jun 2019 08:38:01 +0000 From: Tim Robertson <trobertson(a)gbif.org> To: "ipt(a)lists.gbif.org" <ipt(a)lists.gbif.org> Cc: informatics <informatics(a)gbif.org> Subject: [IPT] EZID, ARK and DOI Message-ID: <E8F22A9E-D94A-430E-AE4A-BB7138ADB521(a)gbif.org> Content-Type: text/plain; charset="utf-8" Dear all, This email only affects those of you using DOI or EZID directly with your IPT. The next release of the IPT will be prepared to accommodate recent changes in the DataCite API who have deprecated their public API. In addition we are aware that EZID have reduced their capabilities with respect to DOIs. We are not aware of anyone using EZID as an identity service for ARKs. We therefore propose to remove EZID capabilities from the IPT 2.4 release as we are not aware that it is in use by anyone. If you are using EZID with your IPT can you please let me know asap and elaborate on which identifier scheme you are using so we can reconsider? Many thanks, Tim

2 2

EZID, ARK and DOI
by Tim Robertson 19 Jun '19

19 Jun '19

Dear all, This email only affects those of you using DOI or EZID directly with your IPT. The next release of the IPT will be prepared to accommodate recent changes in the DataCite API who have deprecated their public API. In addition we are aware that EZID have reduced their capabilities with respect to DOIs. We are not aware of anyone using EZID as an identity service for ARKs. We therefore propose to remove EZID capabilities from the IPT 2.4 release as we are not aware that it is in use by anyone. If you are using EZID with your IPT can you please let me know asap and elaborate on which identifier scheme you are using so we can reconsider? Many thanks, Tim

1 0

Re: [IPT] Daily feeds and archive history
by Kennedy, Jonathan 19 Feb '19

19 Feb '19

Thanks very much, this is helpful feedback. On a related note, Harvard-IQSS created a platform called Dataverse (https://dataverse.org/about) around 2007 and one interesting element is that they published a method for hashing datasets. This is done for the purpose of creating a citation element that can be used to verify that you have downloaded the same data. Passing along in case this is of interest to the group: http://best-practices.dataverse.org/data-citation/ Best regards, Jonathan A. Kennedy Director of Biodiversity Informatics Harvard University Herbaria, Department of Organismic and Evolutionary Biology From: Daniel Noesgaard <dnoesgaard(a)gbif.org> Date: Tuesday, February 19, 2019 at 3:22 AM To: Quentin Groom <quentin.groom(a)plantentuinmeise.be>, Tim Robertson <trobertson(a)gbif.org> Cc: "Kennedy, Jonathan" <jonathan_kennedy(a)harvard.edu>, "ipt(a)lists.gbif.org list" <ipt(a)lists.gbif.org>, helpdesk <helpdesk(a)gbif.org> Subject: Re: [IPT] Daily feeds and archive history I might also add that every download from GBIF.org–be it a single dataset or an aggregate–is archived and given a unique, persistent DOI for citation. And that citations of downloads count against all the datasets that contributed to that download. -- Daniel Noesgaard Science Communications Coordinator GBIF | Global Biodiversity Information Facility - Secretariat Universitetsparken 15 DK-2100 Copenhagen, Denmark E: dnoesgaard(a)gbif.org<mailto:dnoesgaard@gbif.org> W: www.gbif.org T: +45 35 32 08 74 From: Quentin Groom <quentin.groom(a)plantentuinmeise.be> Date: Tuesday, 19 February 2019 at 08.38 To: Tim Robertson <trobertson(a)gbif.org> Cc: "Kennedy, Jonathan" <jonathan_kennedy(a)harvard.edu>, "ipt(a)lists.gbif.org list" <ipt(a)lists.gbif.org>, helpdesk <helpdesk(a)gbif.org>, Daniel Noesgaard <dnoesgaard(a)gbif.org> Subject: Re: [IPT] Daily feeds and archive history While it would be great to have versioned datasets I generally create a snapshot of the data used in a paper and archive this in Zenodo. This gives complete reproducibility without putting extra demands on the data providers. I do however need to cite the source and the snapshot. Regards Quentin On Mon, 18 Feb 2019, 17:45 Tim Robertson <trobertson(a)gbif.org<mailto:trobertson@gbif.org> wrote: Hi Jonathan (adding GBIF helpdesk to the CC) This is just a quick answer which I expect will result in follow up questions. In terms of citation, we use a DOI to identify the concept of a dataset, not the specific version. E.g. https://doi.org/10.15468/cup0nk<https://urldefense.proofpoint.com/v2/url?u=https-3A__doi.org_10.15468_cup0n…> If you start deleting copies of data (e.g. a background housekeeping task) what will break are links to the downloads in the IPT pages. https://ipt.huh.harvard.edu/ipt/resource?r=huh_all_records&v=1.3 This may or may not be considered a problem for you. I think others might have contacted you about suggestions for improving the dataset titles being used but if not I would suggest considering correctly formatted titles as they are used in many places (https://www.gbif.org/dataset/4e4f97d2-4670-4b24-b982-261e0a450faf)<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.gbif.org_dataset_4…>. I hope this helps as a start, Tim From: IPT <ipt-bounces(a)lists.gbif.org<mailto:ipt-bounces@lists.gbif.org>> on behalf of "Kennedy, Jonathan" <jonathan_kennedy(a)harvard.edu<mailto:jonathan_kennedy@harvard.edu>> Date: Monday, 18 February 2019 at 18.31 To: "ipt(a)lists.gbif.org<mailto:ipt@lists.gbif.org>" <ipt(a)lists.gbif.org<mailto:ipt@lists.gbif.org>> Subject: [IPT] Daily feeds and archive history Hi All, I am finishing an upgrade to the Harvard University Herbaria IPT instance and have configured our feeds for daily auto-publish. The HUH has invested in a mass digitization workflow and we are currently creating ~20,000 new vascular records per month (with minimal data), so we do have new records on a daily basis. However, our DwC archives are fairly large (100MB+), so we can’t keep the daily archive history. I am looking for guidance on how it will work with GBIF dataset citation if we do not preserve each daily archive. It seems problematic if a version of our dataset is used and cited but cannot be reconstructed. Best regards, Jonathan A. Kennedy Director of Biodiversity Informatics Harvard University Herbaria, Department of Organismic and Evolutionary Biology _______________________________________________ IPT mailing list IPT(a)lists.gbif.org<mailto:IPT@lists.gbif.org> https://lists.gbif.org/mailman/listinfo/ipt<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.gbif.org_mailman…>

1 0

Re: [IPT] Daily feeds and archive history
by Tim Robertson 19 Feb '19

19 Feb '19

Hi Jonathan (adding GBIF helpdesk to the CC) This is just a quick answer which I expect will result in follow up questions. In terms of citation, we use a DOI to identify the concept of a dataset, not the specific version. E.g. https://doi.org/10.15468/cup0nk If you start deleting copies of data (e.g. a background housekeeping task) what will break are links to the downloads in the IPT pages. https://ipt.huh.harvard.edu/ipt/resource?r=huh_all_records&v=1.3 This may or may not be considered a problem for you. I think others might have contacted you about suggestions for improving the dataset titles being used but if not I would suggest considering correctly formatted titles as they are used in many places (https://www.gbif.org/dataset/4e4f97d2-4670-4b24-b982-261e0a450faf) I hope this helps as a start, Tim From: IPT <ipt-bounces(a)lists.gbif.org> on behalf of "Kennedy, Jonathan" <jonathan_kennedy(a)harvard.edu> Date: Monday, 18 February 2019 at 18.31 To: "ipt(a)lists.gbif.org" <ipt(a)lists.gbif.org> Subject: [IPT] Daily feeds and archive history Hi All, I am finishing an upgrade to the Harvard University Herbaria IPT instance and have configured our feeds for daily auto-publish. The HUH has invested in a mass digitization workflow and we are currently creating ~20,000 new vascular records per month (with minimal data), so we do have new records on a daily basis. However, our DwC archives are fairly large (100MB+), so we can’t keep the daily archive history. I am looking for guidance on how it will work with GBIF dataset citation if we do not preserve each daily archive. It seems problematic if a version of our dataset is used and cited but cannot be reconstructed. Best regards, Jonathan A. Kennedy Director of Biodiversity Informatics Harvard University Herbaria, Department of Organismic and Evolutionary Biology

2 1

Daily feeds and archive history
by Kennedy, Jonathan 18 Feb '19

18 Feb '19

Hi All, I am finishing an upgrade to the Harvard University Herbaria IPT instance and have configured our feeds for daily auto-publish. The HUH has invested in a mass digitization workflow and we are currently creating ~20,000 new vascular records per month (with minimal data), so we do have new records on a daily basis. However, our DwC archives are fairly large (100MB+), so we can’t keep the daily archive history. I am looking for guidance on how it will work with GBIF dataset citation if we do not preserve each daily archive. It seems problematic if a version of our dataset is used and cited but cannot be reconstructed. Best regards, Jonathan A. Kennedy Director of Biodiversity Informatics Harvard University Herbaria, Department of Organismic and Evolutionary Biology

1 0