- IPT - lists.gbif.org

Darwin Core Star Schema
by Quentin Groom 12 Nov '15

12 Nov '15

I'm rather confused how the Darwin Core Star Schema is meant to work for survey data. Darwin Core can have one of two Core files, taxon or occurrence. The most appropriate for a survey would seem to be occurrence. So I imagine that in the star schema you could also have a related event file detailing the date and location of each survey and a non-core taxon file detailing the taxa that are observed. However, this does not seem to be possible. The DWC-A validator ( http://tools.gbif.org/dwca-validator/) assumes only on core id in the core file so you can't link an occurrence both to a taxon and to an event. This is also true in the Darwin Core Archive Assistant ( http://tools.gbif.org/dwca-assistant/) The solution seems to be to put all the information from the taxon core file into the occurrence file, but keep the separate event file linked with the core occurrence id. Is this correct? It seems rather counter intuitive. Regards Quentin Dr. Quentin Groom (Botany and Information Technology) Botanic Garden Meise Domein van Bouchout B-1860 Meise Belgium ORCID: 0000-0002-0596-5376 Landline; +32 (0) 226 009 20 ext. 364 FAX: +32 (0) 226 009 45 E-mail: quentin.groom(a)plantentuinmeise.be Skype name: qgroom Website: www.botanicgarden.be

4 7

Re: [IPT] [tdwg-content] Reverting the process of DwC standardization
by Shorthouse, David 30 Oct '15

30 Oct '15

All, Is part of the issue being expressed here because the raw ecological data sets we're discussing are small-ish matrices rather than occurrences, with site codes as columns, taxa as rows and measures of density/abundance as cells (and similar for environmental variables)? Such structures are often used as input for software that executes eg ordinations, classification & regression trees, species richness estimates. The shortcoming of such a structure is the inherent idiosyncratic nature of "site codes", with variable numbers of them, i.e. an arbitrary number of columns. I doubt it was ever designed for ease of dataset integration, but rather for ease of computation. Representing this structure as Event core requires significant transposition & potential for error if it were manual. Open Refine is one such tool that could permit bi-directional transpositions (DwC -> matrix and then matrix -> DwC), but it is still clunky and accommodation of extensions is virtually non-existent. But, perhaps Open Refine recipes and guides gets us one step closer to finding a balance between the need for standardized representation & efficient transport (DwC) vs. end-users who want matrices for ease of computation. David P. Shorthouse On Tue, Oct 27, 2015 at 7:36 AM, David Valentim Dias <dvdias(a)sibbr.gov.br> wrote: > Hi again, > > I think the problem target both. DwC because is a solution to a problem > creating another problem to researchers less "skilled" in table > manipulation. Ecological data with occurrence is resulting in three tables > and manipulation of these are getting harder with the number of core or > extensions used. > Two possible solutions comes in mind: create a new term describing the > original layout of the columns (so we can use csvjoin like Menashe suggest) > or ipt with option to store the original table associated with resource. > We can always use external links in eml and save the file somewhere but > this means creating another service and managing more login (aka resource > cost and new problems). > > I think any solution will need ipt changes. > > 2015-10-27 9:08 GMT-02:00 Menashe' Eliezer <menashe.eliezer(a)gmail.com>: > >> Hi Tim, >> I believe that the IPT feature I've requested long ago could be helpful >> for David: https://github.com/gbif/ipt/issues/1165 >> Consumers and also the data providers don't have a DwC-A viewer, and they >> need to join the separate csv files for having one table in a worksheet. >> Web applications like the one at OBIS website do let end users download >> one big table. >> >> Best regards, >> Menashè >> >> >> 2015-10-27 9:53 GMT+01:00 Tim Robertson <trobertson(a)gbif.org>: >> >>> Hi David >>> (CC’ing the IPT list as this might be an IPT specific thread - >>> http://lists.gbif.org/mailman/listinfo/ipt) >>> >>> For clarification - is your question specific to the DwC-A standard >>> which is possible as Alex says or is it specific to the IPT tool please? >>> >>> Do you imagine a scenario where you’d effectively map the same extension >>> 2 times - once to interpreted and once to verbatim - or do you envisage a >>> different data schema for each? >>> >>> Thanks, >>> Tim >>> >>> >>> >>> On 23 Oct 2015, at 16:00, Alex Thompson <godfoder(a)acis.ufl.edu> wrote: >>> >>> David, >>> >>> It's certainly possible, within the context of a Darwin Core Archive, to >>> include other files within the ZIP file that lie outside the schema of the >>> archive. Both GBIF and iDigBio do this when generating downloads for >>> various reasons (RIGHTS & LICENSE files, additional EML metadata, etc). >>> However, I do not believe it is possible to do this within IPT. You might >>> submit an issue on the IPT issue tracker ( >>> https://github.com/gbif/ipt/issues) for potential inclusion of this >>> feature in a future version of IPT. >>> >>> There are workarounds you can use to include additional data in Darwin >>> Core archives, but none of them will exactly match your old format. For >>> instance, including an additional Occurrence file with the values as JSON >>> in dynamicProperties or in some other verbatim format in the >>> occurrenceRemarks field. Both of those would at least give some method of >>> single-row access (vs joining multiple measurementOrFacts to a single event >>> id) if that is the primary concern, even if they would require additional >>> parsing steps to be useful. >>> >>> Alex Thompson >>> iDigBio Infrastructure >>> >>> >>> On 10/23/2015 09:40 AM, David Valentim Dias wrote: >>> >>> Dear colleagues, >>> >>> Here on SiBBr we're using the new eventCore and measurementOrFacts and >>> after the process of standardization to DwC and publishing we think some >>> users/researchers will want the "original" table format because of multiple >>> reasons. >>> >>> Is possible to have a vertabimTable or some place where we can store the >>> original table/column format? >>> >>> Regards >>> >>>

2 2

Re: [IPT] [tdwg-content] Reverting the process of DwC standardization
by Tim Robertson 28 Oct '15

28 Oct '15

Hi David (CC’ing the IPT list as this might be an IPT specific thread - http://lists.gbif.org/mailman/listinfo/ipt <http://lists.gbif.org/mailman/listinfo/ipt>) For clarification - is your question specific to the DwC-A standard which is possible as Alex says or is it specific to the IPT tool please? Do you imagine a scenario where you’d effectively map the same extension 2 times - once to interpreted and once to verbatim - or do you envisage a different data schema for each? Thanks, Tim > On 23 Oct 2015, at 16:00, Alex Thompson <godfoder(a)acis.ufl.edu> wrote: > > David, > > It's certainly possible, within the context of a Darwin Core Archive, to include other files within the ZIP file that lie outside the schema of the archive. Both GBIF and iDigBio do this when generating downloads for various reasons (RIGHTS & LICENSE files, additional EML metadata, etc). However, I do not believe it is possible to do this within IPT. You might submit an issue on the IPT issue tracker (https://github.com/gbif/ipt/issues <https://github.com/gbif/ipt/issues>) for potential inclusion of this feature in a future version of IPT. > > There are workarounds you can use to include additional data in Darwin Core archives, but none of them will exactly match your old format. For instance, including an additional Occurrence file with the values as JSON in dynamicProperties or in some other verbatim format in the occurrenceRemarks field. Both of those would at least give some method of single-row access (vs joining multiple measurementOrFacts to a single event id) if that is the primary concern, even if they would require additional parsing steps to be useful. > > Alex Thompson > iDigBio Infrastructure > > On 10/23/2015 09:40 AM, David Valentim Dias wrote: >> Dear colleagues, >> >> Here on SiBBr we're using the new eventCore and measurementOrFacts and after the process of standardization to DwC and publishing we think some users/researchers will want the "original" table format because of multiple reasons. >> >> Is possible to have a vertabimTable or some place where we can store the original table/column format? >> >> Regards >> >> -- >> >> >> >> >> David Valentim Dias >> >> Biodiversity Data Management >> >> SiBBr | MCTI | PNUMA (UNEP) >> >> Phone: +55 61 3329 6045 >> Phone: +55 61 9359 6151 >> Skype: ctenidae >> >> dvdias(a)sibbr.gov.br <mailto:dvdias@sibbr.gov.br> >> www.sibbr.gov.br <http://www.sibbr.gov.br/> >> >> _______________________________________________ >> tdwg-content mailing list >> tdwg-content(a)lists.tdwg.org <mailto:tdwg-content@lists.tdwg.org> >> http://lists.tdwg.org/mailman/listinfo/tdwg-content <http://lists.tdwg.org/mailman/listinfo/tdwg-content> > > _______________________________________________ > tdwg-content mailing list > tdwg-content(a)lists.tdwg.org > http://lists.tdwg.org/mailman/listinfo/tdwg-content

2 1

Fwd: Reverting the process of DwC standardization
by David Valentim Dias 24 Oct '15

24 Oct '15

Dear colleagues, Here on SiBBr we're using the new eventCore and measurementOrFacts and after the process of standardization to DwC and publishing we think some users/researchers will want the "original" table format because of multiple reasons. Is possible to have a vertabimTable or some place where we can store the original table/column format? Regards -- [image: ass_sibbr] *David Valentim Dias* *Biodiversity Data Management* SiBBr | MCTI | PNUMA (UNEP) Phone: +55 61 3329 6045 Phone: +55 61 9359 6151 Skype: ctenidae dvdias(a)sibbr.gov.br www.sibbr.gov.br

1 0

IPT v2.3.2 released
by Kyle Braak 14 Oct '15

14 Oct '15

Dear IPT users, A new minor release of the IPT (v2.3.2) has been released and is available for download here <http://repository.gbif.org/content/groups/gbif/org/gbif/ipt/2.3.2/ipt-2.3.2…> [1]. This version fixes a critical bug <https://github.com/gbif/ipt/issues/1205> [2] introduced in version 2.3 that prevents resources using database connections from publishing successfully. Instructions on how to upgrade your IPT can be found in the IPT’s Release Notes <https://github.com/gbif/ipt/wiki/IPTReleaseNotes23.wiki> [3]. I’m sorry if you were affected by this bug, and I hope you enjoy using the latest version! Best regards, Kyle [1] http://repository.gbif.org/content/groups/gbif/org/gbif/ipt/2.3.2/ipt-2.3.2… <http://repository.gbif.org/content/groups/gbif/org/gbif/ipt/2.3.2/ipt-2.3.2…> [2] https://github.com/gbif/ipt/issues/1205 <https://github.com/gbif/ipt/issues/1205> [3] https://github.com/gbif/ipt/wiki/IPTReleaseNotes23.wiki <https://github.com/gbif/ipt/wiki/IPTReleaseNotes23.wiki>

1 0

Re: [IPT] IPT Digest, Vol 63, Issue 4
by Dinesh Singh Pundir 17 Sep '15

17 Sep '15

Dear Kyle, I have upgraded the IPT v2.3.1 released but there is one issue. I am not able to see the users which where there in the earlier version. Secondly a new user cannot be created in this version. Kindly look into it. Regards Dinesh Pundir WII, INDIA Node On 9/16/2015 3:30 PM, ipt-request(a)lists.gbif.org wrote: > Send IPT mailing list submissions to > ipt(a)lists.gbif.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://lists.gbif.org/mailman/listinfo/ipt > or, via email, send a message with subject or body 'help' to > ipt-request(a)lists.gbif.org > > You can reach the person managing the list at > ipt-owner(a)lists.gbif.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of IPT digest..." > > > Today's Topics: > > 1. IPT v2.3.1 released (Kyle Braak) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 15 Sep 2015 17:14:36 +0200 > From: Kyle Braak <kbraak(a)gbif.org> > To: ipt(a)lists.gbif.org > Subject: [IPT] IPT v2.3.1 released > Message-ID: <6D664F2A-BEC1-4CD5-9FAA-7F5E209F3587(a)gbif.org> > Content-Type: text/plain; charset="windows-1252" > > Dear IPT users, > > A new minor release of the IPT (v2.3.1) has been released and is available for download here [1]. > > This minor release fixes a bug [2] limited to IPT version 2.3, and affects resources whose metadata: > contains non-English characters > was edited after upgrading to version 2.3 > To fix the problem, please upgrade to 2.3.1 and fix badly encoded characters in your metadata. > > Instructions on how to upgrade your IPT can be found in the IPT?s Release Notes [3]. > > Sorry for the inconvenience this bug may have caused you, > > Kyle > > [1] http://repository.gbif.org/content/groups/gbif/org/gbif/ipt/2.3.1/ipt-2.3.1… > [2] https://github.com/gbif/ipt/issues/1197 > [3] https://github.com/gbif/ipt/wiki/IPTReleaseNotes23.wiki >

2 1

IPT v2.3.1 released
by Kyle Braak 16 Sep '15

16 Sep '15

Dear IPT users, A new minor release of the IPT (v2.3.1) has been released and is available for download here [1]. This minor release fixes a bug [2] limited to IPT version 2.3, and affects resources whose metadata: contains non-English characters was edited after upgrading to version 2.3 To fix the problem, please upgrade to 2.3.1 and fix badly encoded characters in your metadata. Instructions on how to upgrade your IPT can be found in the IPT’s Release Notes [3]. Sorry for the inconvenience this bug may have caused you, Kyle [1] http://repository.gbif.org/content/groups/gbif/org/gbif/ipt/2.3.1/ipt-2.3.1… [2] https://github.com/gbif/ipt/issues/1197 [3] https://github.com/gbif/ipt/wiki/IPTReleaseNotes23.wiki

1 0

IPT v2.3 released
by Kyle Braak 11 Sep '15

11 Sep '15

Dear IPT users, The GBIF Secretariat is pleased to announce that a new release of the Integrated Publishing Toolkit [1] (IPT v2.3) is now available for download here [2]. This version supports publishing sample-based datasets, realising the new features that were in prototype at the EU BON IPT [3]. For information about how to upgrade your installation, as well as enable publishing sample-based datasets, please check out the Release Notes [4]. New users should follow the installation guidelines in the IPT User Manual [5] that has been updated for this version. Thank you once again for using the IPT, and we hope you enjoy the latest version. With kind regards, Kyle, on behalf of the GBIF development team and the GBIF Secretariat PS: Please note that the IPT project site has officially moved to GitHub [6] now. [1] http://www.gbif.org/ipt [2] http://repository.gbif.org/content/groups/gbif/org/gbif/ipt/2.3/ipt-2.3.war [3] http://eubon-ipt.gbif.org/ [4] https://github.com/gbif/ipt/wiki/IPTReleaseNotes23.wiki [5] https://github.com/gbif/ipt/wiki/IPT2ManualNotes.wiki [6] https://github.com/gbif/ipt

1 0

Re: [IPT] How to build IPT from source?
by Francisco Moura 27 Aug '15

27 Aug '15

Hi Kyle, thanks for your replay. The compilation error was not fixed, even entering the command mvn clean -U -Prelease package. I noticed that inside my ~/.m2/repository/com/lowagie/itext/4.2.1 there isn't the itext-4.2.1.jar file, even using the commands mvn clean -U -Prelease package, mvn -U -Prelease package, etc; Visiting the page http://mvnrepository.com/artifact/com.lowagie/itext/4.2.1 we can notice that the artifact was moved to a new group. So, I will download a package from gbif build repository for our purpose. If possible, I would ask you to let me know when the source from github are ready to build. Br, Francisco Moura +55 (31) 8277-8448 (Claro) 2015-08-26 5:45 GMT-03:00 Kyle Braak <kbraak(a)gbif.org>: > Hi Francisco, > > I noticed this error myself when testing an upgrade to a newer version of > the com.lowagie:itext dependency. > > Why don’t you try the command below including -U to tell Maven to use the > latest dependency versions. > > #mvn clean -U -Prelease package > > Below are my Maven, Java, and OS versions I am using in case that helps > you: > > dhcp-12:ipt kbraak$ mvn --version > Apache Maven 3.2.3 (33f8c3e1027c3ddde99d3cdebad2656a31e8fdf4; > 2014-08-11T22:58:10+02:00) > Java version: 1.8.0_45, vendor: Oracle Corporation > OS name: "mac os x", version: "10.9.5", arch: "x86_64", family: “mac" > > Let me know if you have success or not, > > Kyle > > PS1: You can monitor whether the IPT will build successfully from source > here: http://builds.gbif.org/job/ipt/ If the last build was ‘blue’ it > will build successfully :) > > PS2: The IPT mailing list (http://lists.gbif.org/mailman/listinfo/ipt) is > ideal for these types of questions. That way your question reaches a > broader audience around the world, and you have a better chance of getting > a response when the GBIF Secretariat is out of office > > On 26 Aug 2015, at 05:56, Francisco Moura <franciscomoura(a)gmail.com> > wrote: > > > Hi Kyle, I hope you are doing well! > > > > I need to build IPT from source. So, my environment setup is: > > - Oracle JDK 7 and 8 > > - Maven 3.3.3 > > > > When I try to build the application, a lot of errors are showed. > > The output process is inside the attached file. > > > > I had cloned the source from github.com/gbif/ipt, and I issued the mvn > install command and the build failed. > > > > My questions are: > > 1) The source from github are ready to do the IPT build? > > 2) Which is the correct environment setup to build the IPT with success > from source? > > > > Thanks for your attention. > > > > Best regards, > > > > Francisco Moura > > +55 (31) 8277-8448 (Claro) > > <build-error.txt> > >

2 1

IPT v2.3 - release candidate
by Kyle Braak 21 Aug '15

21 Aug '15

Dear IPT users, A release candidate of IPT v2.3 is now available for download [1]. Please note this release is only intended for testing, and should not be installed in production. If you don’t have time to install it, you can always try it out at http://ipt.gbif-uat.org/ [2] (just email me if you want an account). Its most notable enhancements include: Support for publishing sample-based datasets, incorporating all the changes now in the prototype at the EU BON IPT [3] Support for versionable IPT extensions, making it possible to update to newer versions whenever they become available A slightly redesigned mapping page, making it easier to map your source data If no major problems are uncovered during testing, IPT v2.3 will be released next week. In the meantime, volunteer translators are hard at work, and the user manual is being updated. Thank you for your attention and help, Kyle, on behalf of the IPT development team and the GBIF Secretariat [1] https://dl.dropboxusercontent.com/u/4552753/ipt-2.3-rc1.war [2] http://ipt.gbif-uat.org/ [3] http://eubon-ipt.gbif.org/

1 1