- IPT - lists.gbif.org

Re: [IPT] Publication of big dataset fails, can't find why
by Peter Desmet 03 Mar '16

03 Mar '16

Increased storage, which solved the issue. Thanks! André, I'm not aware of identical records, but if they have different occurrenceIDs, the IPT validation will pass. On Wed, Mar 2, 2016 at 5:06 PM, André Heughebaert <a.heughebaert(a)biodiversity.be> wrote: > Hi Peter, > > If I remember correctly the Florabank contains some identical records, > although having different IDs. > Could that be the reason of rejection by the IPT? > > Best regards, > André > > Le 02/03/16 15:59, Peter Desmet a écrit : > > Hi, > > We're trying to publish version 45.4 of this dataset: > http://data.inbo.be/ipt/resource?r=florabank1-occurrences, but the > validation seems to fail. Here's the publication log: > http://data.inbo.be/ipt/publicationlog.do?r=florabank1-occurrences > > As far as I can tell, the validation fails on "the core ID field > occurrenceID is always present and unique", but we have verified this > in the generated dwca-45.4.zip file, and all records have a unique > occurrenceID. > > Any idea what might be going on? Possible causes: > > 1. The dataset is quite big (3,5 million records) > 2. We've just solved this issue: > https://github.com/LifeWatchINBO/data-publication/issues/104 by > following Kyle Braak's instructions. The latest published version is > now 45.3, the current (to be published) version is 45.4, so everything > seems fine there. > 3. Even though the publication failed, the following files are created > in /resource/florabank1-occurrences: > > eml-45.4.xml > dwca-45.4,zip > florabank1-occurrences-45.4. > rtf > > Will those file be overwritten if I try to republish or might they be > causing the publication to fail? > > Thanks, > > Peter > _______________________________________________ > IPT mailing list > IPT(a)lists.gbif.org > http://lists.gbif.org/mailman/listinfo/ipt > > > -- > Ir Andre Heughebaert > Belgian Biodiversity Platform > +32(0)2238 3796 > Av. Louise 231 Louizalaan > B-1050 Brussels ORCID 0000-0002-7839-5300

2 1

Publication of big dataset fails, can't find why
by Peter Desmet 02 Mar '16

02 Mar '16

Hi, We're trying to publish version 45.4 of this dataset: http://data.inbo.be/ipt/resource?r=florabank1-occurrences, but the validation seems to fail. Here's the publication log: http://data.inbo.be/ipt/publicationlog.do?r=florabank1-occurrences As far as I can tell, the validation fails on "the core ID field occurrenceID is always present and unique", but we have verified this in the generated dwca-45.4.zip file, and all records have a unique occurrenceID. Any idea what might be going on? Possible causes: 1. The dataset is quite big (3,5 million records) 2. We've just solved this issue: https://github.com/LifeWatchINBO/data-publication/issues/104 by following Kyle Braak's instructions. The latest published version is now 45.3, the current (to be published) version is 45.4, so everything seems fine there. 3. Even though the publication failed, the following files are created in /resource/florabank1-occurrences: eml-45.4.xml dwca-45.4,zip florabank1-occurrences-45.4.rtf Will those file be overwritten if I try to republish or might they be causing the publication to fail? Thanks, Peter

4 5

migrate dataset from one IPT to another.
by Xander van der Sar 09 Dec '15

09 Dec '15

Dear GBIF Friends, We want to migrate a couple of datasets to an other IPT. There is a workflow for migrating from older publication tools [1] in the IPT manual. Can I use this workflow as well when the source publication tool is an IPT? Or do I have to follow another workflow. [1] https://code.google.com/p/gbif-providertoolkit/wiki/IPT2ManualNotes#Migrate… <https://code.google.com/p/gbif-providertoolkit/wiki/IPT2ManualNotes#Migrate…> All the best, Xander

3 2

usage of occurrence in basiOfRecord.
by Xander van der Sar 09 Dec '15

09 Dec '15

Dear GBIF Friends, In the IPT v2.3.2 it is possible to select the default value “Occurrence” for the basisOfRecord in the mapping screen. This value is not in the basisOfRecord Enum [1]. Is this a replacement of the “observation” value? If so, does this value has the same purpose als the old Observation value? [1] http://gbif.github.io/gbif-api/apidocs/org/gbif/api/vocabulary/BasisOfRecor… <http://gbif.github.io/gbif-api/apidocs/org/gbif/api/vocabulary/BasisOfRecor…> All the best, Xander van der Sar data manager NLBIF

2 2

OAI-PMH On IPT
by Carlos Andres Beltran 23 Nov '15

23 Nov '15

Good Morning, We have implemented IPT at our institution ( http://i2d.humboldt.org.co/ceiba/) We need to do the harvest this catalog with others who have implemented. According to this URL: http://gbif.blogspot.com.co/2011/04/oia-pmh-harvesting-at-gbif.html It can be done, someone could help us? Thanks, Regards -- *Carlos Andrés Beltran *Investigador I2D * Instituto de Investigación de Recursos Biológicos Alexander von Humboldt Avenida Paseo Bolivar 16-20 PBX: (57) 1 320 2767 Ext. 2103 Bogotá, D.C., Colombia* Antes de imprimir este mensaje, piensa si realmente necesitas hacerlo. ¡Gracias!

4 3

Darwin Core Star Schema
by Quentin Groom 11 Nov '15

11 Nov '15

I'm rather confused how the Darwin Core Star Schema is meant to work for survey data. Darwin Core can have one of two Core files, taxon or occurrence. The most appropriate for a survey would seem to be occurrence. So I imagine that in the star schema you could also have a related event file detailing the date and location of each survey and a non-core taxon file detailing the taxa that are observed. However, this does not seem to be possible. The DWC-A validator ( http://tools.gbif.org/dwca-validator/) assumes only on core id in the core file so you can't link an occurrence both to a taxon and to an event. This is also true in the Darwin Core Archive Assistant ( http://tools.gbif.org/dwca-assistant/) The solution seems to be to put all the information from the taxon core file into the occurrence file, but keep the separate event file linked with the core occurrence id. Is this correct? It seems rather counter intuitive. Regards Quentin Dr. Quentin Groom (Botany and Information Technology) Botanic Garden Meise Domein van Bouchout B-1860 Meise Belgium ORCID: 0000-0002-0596-5376 Landline; +32 (0) 226 009 20 ext. 364 FAX: +32 (0) 226 009 45 E-mail: quentin.groom(a)plantentuinmeise.be Skype name: qgroom Website: www.botanicgarden.be

4 7

Re: [IPT] [tdwg-content] Reverting the process of DwC standardization
by Shorthouse, David 29 Oct '15

29 Oct '15

All, Is part of the issue being expressed here because the raw ecological data sets we're discussing are small-ish matrices rather than occurrences, with site codes as columns, taxa as rows and measures of density/abundance as cells (and similar for environmental variables)? Such structures are often used as input for software that executes eg ordinations, classification & regression trees, species richness estimates. The shortcoming of such a structure is the inherent idiosyncratic nature of "site codes", with variable numbers of them, i.e. an arbitrary number of columns. I doubt it was ever designed for ease of dataset integration, but rather for ease of computation. Representing this structure as Event core requires significant transposition & potential for error if it were manual. Open Refine is one such tool that could permit bi-directional transpositions (DwC -> matrix and then matrix -> DwC), but it is still clunky and accommodation of extensions is virtually non-existent. But, perhaps Open Refine recipes and guides gets us one step closer to finding a balance between the need for standardized representation & efficient transport (DwC) vs. end-users who want matrices for ease of computation. David P. Shorthouse On Tue, Oct 27, 2015 at 7:36 AM, David Valentim Dias <dvdias(a)sibbr.gov.br> wrote: > Hi again, > > I think the problem target both. DwC because is a solution to a problem > creating another problem to researchers less "skilled" in table > manipulation. Ecological data with occurrence is resulting in three tables > and manipulation of these are getting harder with the number of core or > extensions used. > Two possible solutions comes in mind: create a new term describing the > original layout of the columns (so we can use csvjoin like Menashe suggest) > or ipt with option to store the original table associated with resource. > We can always use external links in eml and save the file somewhere but > this means creating another service and managing more login (aka resource > cost and new problems). > > I think any solution will need ipt changes. > > 2015-10-27 9:08 GMT-02:00 Menashe' Eliezer <menashe.eliezer(a)gmail.com>: > >> Hi Tim, >> I believe that the IPT feature I've requested long ago could be helpful >> for David: https://github.com/gbif/ipt/issues/1165 >> Consumers and also the data providers don't have a DwC-A viewer, and they >> need to join the separate csv files for having one table in a worksheet. >> Web applications like the one at OBIS website do let end users download >> one big table. >> >> Best regards, >> Menashè >> >> >> 2015-10-27 9:53 GMT+01:00 Tim Robertson <trobertson(a)gbif.org>: >> >>> Hi David >>> (CC’ing the IPT list as this might be an IPT specific thread - >>> http://lists.gbif.org/mailman/listinfo/ipt) >>> >>> For clarification - is your question specific to the DwC-A standard >>> which is possible as Alex says or is it specific to the IPT tool please? >>> >>> Do you imagine a scenario where you’d effectively map the same extension >>> 2 times - once to interpreted and once to verbatim - or do you envisage a >>> different data schema for each? >>> >>> Thanks, >>> Tim >>> >>> >>> >>> On 23 Oct 2015, at 16:00, Alex Thompson <godfoder(a)acis.ufl.edu> wrote: >>> >>> David, >>> >>> It's certainly possible, within the context of a Darwin Core Archive, to >>> include other files within the ZIP file that lie outside the schema of the >>> archive. Both GBIF and iDigBio do this when generating downloads for >>> various reasons (RIGHTS & LICENSE files, additional EML metadata, etc). >>> However, I do not believe it is possible to do this within IPT. You might >>> submit an issue on the IPT issue tracker ( >>> https://github.com/gbif/ipt/issues) for potential inclusion of this >>> feature in a future version of IPT. >>> >>> There are workarounds you can use to include additional data in Darwin >>> Core archives, but none of them will exactly match your old format. For >>> instance, including an additional Occurrence file with the values as JSON >>> in dynamicProperties or in some other verbatim format in the >>> occurrenceRemarks field. Both of those would at least give some method of >>> single-row access (vs joining multiple measurementOrFacts to a single event >>> id) if that is the primary concern, even if they would require additional >>> parsing steps to be useful. >>> >>> Alex Thompson >>> iDigBio Infrastructure >>> >>> >>> On 10/23/2015 09:40 AM, David Valentim Dias wrote: >>> >>> Dear colleagues, >>> >>> Here on SiBBr we're using the new eventCore and measurementOrFacts and >>> after the process of standardization to DwC and publishing we think some >>> users/researchers will want the "original" table format because of multiple >>> reasons. >>> >>> Is possible to have a vertabimTable or some place where we can store the >>> original table/column format? >>> >>> Regards >>> >>>

2 2

Re: [IPT] [tdwg-content] Reverting the process of DwC standardization
by Tim Robertson 27 Oct '15

27 Oct '15

Hi David (CC’ing the IPT list as this might be an IPT specific thread - http://lists.gbif.org/mailman/listinfo/ipt <http://lists.gbif.org/mailman/listinfo/ipt>) For clarification - is your question specific to the DwC-A standard which is possible as Alex says or is it specific to the IPT tool please? Do you imagine a scenario where you’d effectively map the same extension 2 times - once to interpreted and once to verbatim - or do you envisage a different data schema for each? Thanks, Tim > On 23 Oct 2015, at 16:00, Alex Thompson <godfoder(a)acis.ufl.edu> wrote: > > David, > > It's certainly possible, within the context of a Darwin Core Archive, to include other files within the ZIP file that lie outside the schema of the archive. Both GBIF and iDigBio do this when generating downloads for various reasons (RIGHTS & LICENSE files, additional EML metadata, etc). However, I do not believe it is possible to do this within IPT. You might submit an issue on the IPT issue tracker (https://github.com/gbif/ipt/issues <https://github.com/gbif/ipt/issues>) for potential inclusion of this feature in a future version of IPT. > > There are workarounds you can use to include additional data in Darwin Core archives, but none of them will exactly match your old format. For instance, including an additional Occurrence file with the values as JSON in dynamicProperties or in some other verbatim format in the occurrenceRemarks field. Both of those would at least give some method of single-row access (vs joining multiple measurementOrFacts to a single event id) if that is the primary concern, even if they would require additional parsing steps to be useful. > > Alex Thompson > iDigBio Infrastructure > > On 10/23/2015 09:40 AM, David Valentim Dias wrote: >> Dear colleagues, >> >> Here on SiBBr we're using the new eventCore and measurementOrFacts and after the process of standardization to DwC and publishing we think some users/researchers will want the "original" table format because of multiple reasons. >> >> Is possible to have a vertabimTable or some place where we can store the original table/column format? >> >> Regards >> >> -- >> >> >> >> >> David Valentim Dias >> >> Biodiversity Data Management >> >> SiBBr | MCTI | PNUMA (UNEP) >> >> Phone: +55 61 3329 6045 >> Phone: +55 61 9359 6151 >> Skype: ctenidae >> >> dvdias(a)sibbr.gov.br <mailto:dvdias@sibbr.gov.br> >> www.sibbr.gov.br <http://www.sibbr.gov.br/> >> >> _______________________________________________ >> tdwg-content mailing list >> tdwg-content(a)lists.tdwg.org <mailto:tdwg-content@lists.tdwg.org> >> http://lists.tdwg.org/mailman/listinfo/tdwg-content <http://lists.tdwg.org/mailman/listinfo/tdwg-content> > > _______________________________________________ > tdwg-content mailing list > tdwg-content(a)lists.tdwg.org > http://lists.tdwg.org/mailman/listinfo/tdwg-content

2 1

Fwd: Reverting the process of DwC standardization
by David Valentim Dias 23 Oct '15

23 Oct '15

Dear colleagues, Here on SiBBr we're using the new eventCore and measurementOrFacts and after the process of standardization to DwC and publishing we think some users/researchers will want the "original" table format because of multiple reasons. Is possible to have a vertabimTable or some place where we can store the original table/column format? Regards -- [image: ass_sibbr] *David Valentim Dias* *Biodiversity Data Management* SiBBr | MCTI | PNUMA (UNEP) Phone: +55 61 3329 6045 Phone: +55 61 9359 6151 Skype: ctenidae dvdias(a)sibbr.gov.br www.sibbr.gov.br

1 0

IPT v2.3.2 released
by Kyle Braak 13 Oct '15

13 Oct '15

Dear IPT users, A new minor release of the IPT (v2.3.2) has been released and is available for download here <http://repository.gbif.org/content/groups/gbif/org/gbif/ipt/2.3.2/ipt-2.3.2…> [1]. This version fixes a critical bug <https://github.com/gbif/ipt/issues/1205> [2] introduced in version 2.3 that prevents resources using database connections from publishing successfully. Instructions on how to upgrade your IPT can be found in the IPT’s Release Notes <https://github.com/gbif/ipt/wiki/IPTReleaseNotes23.wiki> [3]. I’m sorry if you were affected by this bug, and I hope you enjoy using the latest version! Best regards, Kyle [1] http://repository.gbif.org/content/groups/gbif/org/gbif/ipt/2.3.2/ipt-2.3.2… <http://repository.gbif.org/content/groups/gbif/org/gbif/ipt/2.3.2/ipt-2.3.2…> [2] https://github.com/gbif/ipt/issues/1205 <https://github.com/gbif/ipt/issues/1205> [3] https://github.com/gbif/ipt/wiki/IPTReleaseNotes23.wiki <https://github.com/gbif/ipt/wiki/IPTReleaseNotes23.wiki>

1 0