- IPT - lists.gbif.org

Migrating IPT and changing Resource Managers
by Dan Lear 23 Sep '14

23 Sep '14

I have migrated a resource to a new instance of IPT on a different server, however I want to change the Resource Manager. Migrating the resource without changing the manager give a "FreeMarker template error". Do I have to recreate the Resource Managers on the new server, or can I make the edits somewhere else? Thanks Dan

2 2

A "simple" question about switch from Test to Production mode
by Michel Kapel 18 Sep '14

18 Sep '14

Hello. We are going to switch our IPT from "test" to "production" mode. The documentation seems clear about it but I'd like to be sure of something. At this point In the doc https://code.google.com/p/gbif-providertoolkit/wiki/IPT2ManualNotes#IPT_set… -------------------------------- To switch from test to production mode or vice versa, you will have to reinstall your IPT and repeat any configurations you made. (see the "Starting Over" section in this Getting Started Guide). -------------------------------- Starts clearly enough REINSTALL but then comes the Starting Over part https://code.google.com/p/gbif-providertoolkit/wiki/IPT2ManualNotes#Startin… ---------------------------------------------- It is relatively easy to re-initiate the IPT and begin again with the first setup page by doing the following: Every user who is logged in to the IPT should first log out. Remove the file called datadir.location from the folder where it was created by the IPT (normally WEB-INF within the IPT base installation directory - not within the IPT data directory configured in the first setup step). The person having the default administrator information and knowledge of the IPT installation should open the IPT in a web browser. This will show the initial IPT setup page again (see the "IPT setup" section in this Getting Started Guide). If the user enters the same absolute path to the data directory as before, the previous configuration will be completely restored, assuming that there has been no incompatible upgrade in IPT versions between the time when the data directory was last used and when it is used to re-initiate the IPT. If the same data directory location is desired, but without the previous IPT configuration, then the contents of that data directory should be moved to a different location or removed entirely before clicking on "Save" in the initial IPT setup page. If an entirely new data directory is used, then the setup will proceed exactly as described in the "IPT setup" section of this Getting Started Guide. ---------------------------------------------- So here comes my question. Could just starting with an entirely new data directory allow switching for "test" to "production"? The choice of IPT mode looks like part of the rest of the procedure in that particular case.

1 0

strange meta.xml file
by "Dröge, Gabriele" 19 Aug '14

19 Aug '14

Hi everyone, I'm using IPT Version 2.1.1-r4640 with several extensions and I am wondering why my meta.xml looks different from others. Usually for the extensions there is just a row <coreid index="0"> But in my meta.xml there is a second one <field term="http://rs.tdwg.org/dwc/terms/occurenceID"> I've attached both meta.xml-files (taken from a GBIF provider). HIT seems to have some problems with it, so do you have any idea why my meta.xml looks like this? Best, Gabi

2 1

IPT 2.2.2 with SQL Server 2008; jtds error
by Blum, Stan 29 May '14

29 May '14

Hi all, Is anyone else using: IPT 2.1.1 on CentOS 6.5 Java 1.6.0_30 Tomcat 6 Connecting to: SQL Server 2008 R2 Windows 2008 Server We had been operating just fine with 2.0.4 since last fall. We recently updated to IPT 2.1.1 about two weeks ago. Now we are unable to connect to our SQL Server instance. We’re getting: java.lang.UnsupportedClassVersionError: net/sourceforge/jtds/jdbc/Driver : Unsupported major.minor version 51.0 (unable to load class net.sourceforge.jtds.jdbc.Driver) (We also upgraded the SQL Server database/server recently, but I think that was working fine for at least two weeks before we upgraded our IPT.) Does anyone know how to fix (or help us further diagnose) our “unsupported major.minor version 51.0” problem? Thanks, -Stan

2 1

Security update
by Kyle Braak [GBIF] 01 May '14

01 May '14

Dear IPT users, This is an important message regarding the IPT. IPT administrators should update their IPTs to the new version (2.1.1) which has a security update and is available for download here. The security update fixes a couple vulnerabilities inherited from the Apache Struts web framework, which the IPT uses. Struts made this announcement describing their latest release and its patched security issues that the IPT has upgraded to. For further information about the Struts vulnerabilities, as well as some helpful mitigation strategies for server administrators, you can refer to this article. Simple instructions on how to update your IPT are below. If you don't have time to update your IPT immediately, I would advise you to take it offline until you can. Please email the IPT list directly for more help upgrading your installation. Sincerely, Kyle, on behalf of the IPT development team and the GBIF Secretariat Instructions how to update IPT in Tomcat: Please download: http://repository.gbif.org/content/groups/gbif/org/gbif/ipt/2.1.1/ipt-2.1.1… Backup IPT data directory somewhere safe Remove ipt.war from $tomcat/webapps/ (some seconds later, the deployed /ipt folder should automatically delete) Once ipt.war and /ipt have been removed from /webapps - stop Tomcat Add new version to /webapps renaming it from ipt-2.1.1.war to ipt.war Start Tomcat In a browser open the application (if it doesn't appear at first, try restarting Tomcat once more). When prompted for IPT data directory, enter same location as existing IPT data directory Press continue, hopefully installation succeeds.

1 0

IPT v2.1 released
by Kyle Braak [GBIF] 23 Apr '14

23 Apr '14

Dear IPT users, The GBIF Secretariat is pleased to announce that a new release of the Integrated Publishing Toolkit (IPT v2.1) is now available for download here. For information about how to upgrade your installation, you can check out the Release Notes. Added to this version, is support for Microsoft Excel files as data sources, and the ability to validate that each record within a dataset has a unique identifier. To find out what else is new, you can check out the latest post on the GBIF Developer Blog. New users should follow the installation guidelines in the IPT User Manual that has been updated for this version. Thank you once again for using the IPT, and we hope you enjoy the latest version. With kind regards, Kyle, on behalf of the GBIF development team and the GBIF Secretariat

1 0

Pre-release testing, IPT v2.1
by Kyle Braak [GBIF] 09 Apr '14

09 Apr '14

Dear IPT users, A pre-release version of IPT v2.1 is online for wider testing at http://ipt-uat.gbif.org I would greatly appreciate your help with testing, if you have a little time to spare. To login, you can use your existing account on http://ipt.gbif.org/. If you need an account setup, or if you need a password reminder, just let me know. Here are a few of the new features we need help testing: Identifier validation: when the occurrenceID/taxonID is mapped, the IPT will validate that each record has an ID, and each ID is unique Excel file support: Excel files can now be uploaded as source data Interface in Japanese: making the interface available in 6 languages now As always, our aim is to ensure that the released version is as sturdy as possible. We are currently putting the release candidate through rigorous testing too. In testing this morning, we successfully published a copy of the EOD-eBird Observation Dataset with 151 million records (check out the attached screenshot). We want to be confident that the identifier validation will scale up, and the initial tests look positive. If no major issues are uncovered during testing, IPT v2.1 will be released next week. Thank you for your assistance, Kyle, on behalf of the IPT development team and the GBIF Secretariat

1 0

Re: [IPT] How to add his own/custom extension
by Julien Husson 09 Apr '14

09 Apr '14

Hi Tim, thanks for your prompt answer. It's not really what i expected, to be more specific > First, i have to harvest lot of data from multiple DB (remote or not) with, of course, different structures/models and format of datas ... sometimes no DB just flat CSV/xsl files... That's why i use an IPT to create data mapping and to standardize the data stream into Dwc standard. (I'm talking about millions of specimen) Secondly, i build specific indexes of harvested data with a custom harvester using canadensys-harvester lib. ( Thanks to Christian). It's at this point that it's begin to be difficult with a denormalized view of the data from IPT Dwc-A. 'Cause i need to transform this denormalized view or raw model into a normalize view that match with my big relational database model which become the new repository. That's why i thought that the custom extensions could be make my life easier. On Fri, Mar 14, 2014 at 5:12 PM, Julien Husson <biology.info(a)gmail.com>wrote: > Hi Tim, thanks for your prompt answer. > > It's not really what i expected, to be more specific > > > First, i have to harvest lot of data from multiple DB (remote or not) > with, of course, different structures/models and format of datas ... > sometimes no DB just flat CSV/xsl files... > That's why i use an IPT to create data mapping and to standardize the data > stream into Dwc standard. > (I'm talking about millions of specimen) > > Secondly, i build specific indexes of harvested data with a custom > harvester using canadensys-harvester lib. ( Thanks to Christian). It's at > this point that it's begin to be difficult with a denormalized view of the > data from IPT Dwc-A. > 'Cause i need to transform this denormalized view or raw model into a > normalize view that match with my big relational database model which > become the new repository. > > That's why i thought that the custom extensions could be make my life > easier. > > > > > On Fri, Mar 14, 2014 at 4:32 PM, Tim Robertson [GBIF] <trobertson(a)gbif.org > > wrote: > >> Hi Julien, >> >> Thanks for your question. It really depends on what you are trying to >> publish. We can add extensions of course, but without knowing the >> specifics it is difficult to comment. >> >> However, a "specimen, event, location" DB model would typically map to an >> Occurrence core with no extensions required - this is the most common use >> case of Darwin Core and the IPT. An Occurrence core is basically a >> denormalized view of the data. >> >> If I were the data manager, I would probably consider that I was >> publishing a "DwC Occurrence view" of my more complex model and as such >> would keep a view in the database along the lines of: >> >> CREATE VIEW view_dwc_occurrence AS >> SELECT >> specimen.bar_code AS occurrenceID, >> specimen.name AS scientificName, >> location.latitude AS decimalLatitude, >> location.longitude AS decimalLongitude, >> event.year AS year >> FROM >> specimen >> JOIN event ON ... >> JOIN location ON ... >> WHERE >> <insert any conditions for record inclusion, such as non endangered etc> >> >> Then in my IPT I would simply do "SELECT * FROM view_dwc_occurrence". >> Here you are flattening the normalised model into a denormalized DwC view >> of the data. >> >> Maintaining a view in the database layer as opposed to a custom mapping >> in the IPT benefits you by: >> i) catching issues early with database schema changes since the DB will >> likely stop you with an error >> ii) offering an easy mapping of DB table field names, to DwC terms in a >> language I find very familiar (SQL) >> iii) a super simple IPT mapping, as all columns will map automatically >> in the IPT since they are DwC recognised terms already >> >> Does that help in any way? If not, could you please elaborate on the >> model and what you are trying to achieve and we'll do all we can. >> >> Thanks, >> Tim >> >> >> >> On 14 Mar 2014, at 16:06, Julien Husson <biology.info(a)gmail.com> wrote: >> >> Hi, >> >> I use Dwc-A to feed my BD. >> >> We know the limits of the Dwc star schema to represent a relationnal >> database. >> >> For example in the case of 1-n cardinality : specimen --- event/record >> --- location. >> If i understand the concept, I need to use the Darwin *Core* *Occurrence* >> *extension*, denormalize my relational model in a big raw model and >> transform / re-normalize this model to match with my DB model. >> >> In order to reduce cost*, *dev and optimize sql statement, it will be >> grandly appreciate to add custom extension. In this case, i can to be >> very close of my relational database model and avoid multiple step of dev. >> >> I discovered this link but explanantion is now deprecated >> >> http://dag-endresen.blogspot.fr/2009/06/adding-extension-for-germplasm-to-g… >> >> Thanks, >> >> J. >> _______________________________________________ >> IPT mailing list >> IPT(a)lists.gbif.org >> http://lists.gbif.org/mailman/listinfo/ipt >> >> >> >

2 2

How to add his own/custom extension
by Julien Husson 14 Mar '14

14 Mar '14

Hi, I use Dwc-A to feed my BD. We know the limits of the Dwc star schema to represent a relationnal database. For example in the case of 1-n cardinality : specimen --- event/record --- location. If i understand the concept, I need to use the Darwin *Core* *Occurrence* *extension*, denormalize my relational model in a big raw model and transform / re-normalize this model to match with my DB model. In order to reduce cost*, *dev and optimize sql statement, it will be grandly appreciate to add custom extension. In this case, i can to be very close of my relational database model and avoid multiple step of dev. I discovered this link but explanantion is now deprecated http://dag-endresen.blogspot.fr/2009/06/adding-extension-for-germplasm-to-g… Thanks, J.

2 1

Happy new year, two questions
by Michel Kapel 03 Jan '14

03 Jan '14

Happy new year everybody. I have two questions here. 1. I am searching for DarwinCore element "verbatim_scientific_name" is this available somewhere? 2. I have a need for a non DarwinCore field namely "WaterBodyType" with values such as (RIVER|LAKE|ESTUARY|SEA|...) I have looked but it does not seem to exist in DarwinCore. WaterBody does exist but this refers to the name of the water body such as "Nile river" or "Ganges river" or just "Amazon" . I want something that just specifies the type. Would it be possible to define such a field and add it to the mapping interface? Thanks in advance for any info. Michel K.

2 1