- IPT - lists.gbif.org

Re: [IPT] Adding an extension for germplasm to IPT
by Dag Terje Filip Endresen 27 May '09

27 May '09

Dear Eli, The instructions I received from Markus is actually already included here below as included in your email. I have also added the instructions from Tim, as also posted to the IPT mailing list. Basically what (I think) the IPT does is to contact the registry as defined in "ipt.properties". By default the GBIF registry in Copenhagen is defined here, but you may modify this to point to your own registry. I did not mimic the complete features of the registry - ONLY the list of extensions "extensions.json" to add the new draft germplasm extension I made. You may want to add your own thesaurus "thesauri.json" of controlled vocabulary or more features in the similar way. After loading my new extension to my IPT at NordGen, I modified the "properties.ipt" back to point to the GBIF registry in Copenhagen again. To mimic the registry feature for adding the extension, I made a copy of the file from "http://gbrds.gbif.org/registry/ipt/extensions.json" and added my application schema from the extension as you will see at "http://wwwdev.ngb.se/portal/rs/ipt/extensions.json". The application schema (as defined in "extensions.json") and online from: "http://wwwdev.ngb.se/portal/rs/ipt_germplasm.xml", creates a namespace (for the use by IPT) as well as inclides the URL for the actual draft germplasm extension I made: "http://ipt.gbif.org/extension/germplasm/ http://wwwdev.ngb.se/portal/rs/dwc_germplasm.xsd" The application schema itself (ipt_germplasm.xml) only lists the terms I want to use in IPT, with the property name to be displayed in the IPT mapping interface. To create this file I started with one of the other extensions "http://gbrds.gbif.org/resources/extensions/vernacularName.xml". I just made a copy of this file and replaced the names and terms for vernacular name extension with the terms I wanted from the draft germplasm extension. I hope that my description of what I think it was I did is more or less correct. I copy the IPT mailing list so that Markus and Tim may have a chance to correct me or to add more information to the procedure. Cheers Dag Terje _____ From: Eli Agbayani [mailto:eagbayani@eol.org] To: Dag Terje Filip Endresen [mailto:dag.endresen@nordgen.org] Sent: Tue, 26 May 2009 22:52:35 +0200 Subject: Re: Adding an extension for germplasm to IPT Dear Tag, Thanks for your email. Can you please forward to me the instructions you said you followed in testing your own extension. That is the instructions the GBIF guys gave you. Thanks. Eli ----- Original Message ----- From: "Dag Terje Filip Endresen" <dag.endresen(a)nordgen.org> To: "\"Markus Döring (GBIF)\"" <mdoering(a)gbif.org>, "Eli Agbayani" <eagbayani(a)eol.org> Cc: ipt(a)lists.gbif.org Sent: Friday, May 15, 2009 9:07:29 AM (GMT-0500) America/New_York Subject: Adding an extension for germplasm to IPT Hi Markus and Tim, Thanks a lot for your help! I followed your instructions and successfully added my DRAFT version of a DarwinCore extension for germplasm to the IPT. I have listed a brief summary of each of the steps I followed below. If you are interested, you may visit the IPT from NordGen with a DEMO dataset mapped to terms from this DRAFT Darwin Core extension for germplasm at: http://atlas.nordgen.org/ipt/occResource.html?resource_id=77 1. Schema I built a suggestion to a DRAFT Darwin Core extansion for germplasm http://code.google.com/p/darwincore/source/browse/trunk/xsd/profiles/germpl… http://wwwdev.ngb.se/portal/rs/dwc_germplasm.xsd 2. XML definition for IPT Listing the terms to use from the suggested DRAFT DarwinCore extension http://wwwdev.ngb.se/portal/rs/ipt_germplasm.xml 3. Add this XML definition to a copy of extensions.json Starting with " http://gbrds.gbif.org/registry/ipt/extensions.json " Add text {"title":"Germplasm","url":" http://wwwdev.ngb.se/portal/rs/ipt_germplasm.xml ","id":9} Saved as: http://wwwdev.ngb.se/portal/rs/ipt/extensions.json 4. Modify ipt.properties PATH_TO_TOMCAT_WEBAPPS/ipt/WEB-INF/classes/ipt.properties Line 7: "registryUrl= http://gbrds.gbif.org/registry " modified to "registryUrl= http://wwwdev.ngb.se/portal/rs " 5. Restart Tomcat (reload ipt) The new ipt.properties settings does not seem to be loaded without restart of tomcat 6. Add/load the new extension to IPT Login with admin permissions http://your_server_name/ipt/admin/extensions.html Click "Check for all available extensions" Select the name of the new extension Select "install" and "Done" 7. Mapping of terms to your data Select the new extension under "Add New Mapping" Click "Add" button to the right Select and map terms I selected the "publish" button and hope that this will not cause this dataset to be registered at the GBIF registry...? Without selecting the "publish" button the dataset is not listed from the menu item "explore". I was missing the option to publish a dataset under this "explore", but not not to register with the GBIF registry...? Should there not be one more step to this procedure? Cheers Dag Terje PS. I have noticed that the right side context menu is very sensitive to missing data (Navigate Taxonomy, Navigate Geography). If some of the higher taxonomic levels is missing I get an error message instead of the context menu. For example I did not have phylum for the genebank dataset and the error message on "Navigate Taxonomy" untill I added data for this term. Similar I did not have continent causing the "Navigate Geography" context menu to start at country. When continent and/or country is missing, the "Navigate Geography" context menu creates a top node for the lowest level available - sometimes even the coordinate is displayed as the top node here. Would be cute if the IPT could default to "unknown continent", "unknown country"; "unknown phylum", ... or something similar. PPS. NB! This DRAFT extension for germplasm is still under development and I am sure that I still duplicate some terms from DarwinCore or other extensions! I did not yet work much with the terms for trait observations on germplasm, but added most of the terms from the new proposed draft standard for these data for the European genebank community. From: "Markus Döring (GBIF)" [mailto: mdoering(a)gbif.org ] To: Dag Terje Filip Endresen [mailto: dag.endresen(a)nordgen.org ], Eli Agbayani [mailto: eagbayani(a)eol.org ] Cc: ipt(a)lists.gbif.org Sent: Wed, 13 May 2009 00:05:33 +0200 Subject: Re: [IPT] IPT connection problem Dag, Eli, I've checked the IPT code again for the extension registry URL and I was actually wrong. The current version *does* make use of the ipt.properties settings. The standard GBIF URL that is used in the ipt.properties should be: registryUrl= http://gbrds.gbif.org/registry The registry then returns a JSON list of all known extensions located at: http://gbrds.gbif.org/registry /ipt/extensions.json So to test your own extensions you can modify this json list to include entries for your own (pointing to your extension xml definitions hosted on some URL), upload that json file to some http server and modify the ipt.properties setting to point to your registry, e.g.: registryUrl= http://dev.eol.org/ipt that will finally point to the file http://dev.eol.org/ipt/ipt/extensions.json To get everything working this registry base URL should also deal with other things, e.g. the registry of resources, organisations, services, etc. But in order to test the extensions this should still work. If you give it a try, please report your findings back to this list. cheers, Markus On May 12, 2009 , at 6:05 PM, Markus Döring (GBIF) wrote: PS. I agree with Jörg that a convenient way to keep the old data directory when upgrading the IPT would be cute. This is indeed the case. We will aim wherever possible to make upgrading as simple as it can be. For bug fix releases, such as this snapshot, with no structural change the data directory can be migrated directly. In future releases, we might require a "one time" migration task be run. I anticipate the first one being required when the DwC standard is ratified. if you look into your ipt.properties file you can modify the following settings: dataDir= ${ipt.datadir} registryUrl= ${ipt.registry.url} the dataDir should be an absolute path pointing to your data dir (defaults to the ipt webapp data subfolder). the registry URL can point to a development registry - unfortunately the extensions and vocabs currently dont use this setting, only the UDDI style registration of resources and organisations. We will make sure that the IPT uses this setting also for the extension registration, so you can easily point it to an alternative extension site for testing. markus --------------------------------------------------------------------- -------- Some more useful instructions from Tim ----------- --------------------------------------------------------------------- _____ From: Tim Robertson [mailto:trobertson@gbif.org] To: Dag Terje Filip Endresen [mailto:dag.endresen@nordgen.org] Cc: ipt(a)lists.gbif.org Sent: Tue, 12 May 2009 15:11:10 +0200 Subject: Re: [IPT] IPT connection problem Hi all, PS. I agree with Jörg that a convenient way to keep the old data directory when upgrading the IPT would be cute. This is indeed the case. We will aim wherever possible to make upgrading as simple as it can be. For bug fix releases, such as this snapshot, with no structural change the data directory can be migrated directly. In future releases, we might require a "one time" migration task be run. I anticipate the first one being required when the DwC standard is ratified. PPS. And a convinient way to add a thematic extension to the DwC (for testing and demonstration) would also be very useful. I am thinking of the draft DwC germplasm extension - as well as a new draft for germplasm trait observation data I have started to work on. I assume this perhaps could be done as an addition in the IPT code somewhere around "WEB-INF/classes/org/gbif/provider/model/" ...? A good suggestion. We should look at setting up some sandbox repository for exactly this purpose. Dag, for testing and demonstration you could do the following, but please be very careful not to register these datasets with GBIF. - The IPT will call (http://gbrds.gbif.org/registry/ipt/extensions.json and http://gbrds.gbif.org/registry/ipt/thesauri.json) to get the extensions and the thesauri. - The response is fairly self explanatory, and for a test, you could override gbrds.gbif.org in you host file, and put up a different version locally for example referencing your test extensions. - Then simply update your extensions through the IPT. - Sorry, not the easiest work around, but might get you able to demonstrate how you can start to build your community without waiting for the sandbox repository. Best wishes, Tim

1 0

Re: [IPT] IPT connection problem
by Tim Robertson 18 May '09

18 May '09

Hi Mickael, Please always post questions to the list, to help us keep an FAQ up to date and allow for others to search etc. I have just tested it myself with the following: Database connection URL: jdbc:mysql://localhost:3306/portal Database User: tim Database Password: password and it worked... so, to get started debugging your issue may I please ask some basic questions: - You wrote "jdbc:mysql://localhost:3306/databasename" - is your database name correct? - You are running the IPT on the same machine as the database? - You are using mysql? Cheers, Tim On 11 May 2009, at 10:26, Mickael Graf wrote: > Hi Tim, > > I'm trying to create a resource in IPT but I encounter a problem while > creating the link to the database: in "Source Data" it is written > under > "SQL Data Sources" I have the message "No Database connected. Please > adjust connection parameters to use SQL sources" while it keeps my > settings correctly. I hope I wrote the jdbc string well: > jdbc:mysql://localhost:3306/databasename. > > I surprised because I am using the same parameters as with TAPIRLink > in > which I don't have any problem. I don't think I should make any > changes > in my grants on my database. > > Cheers, > Mickaël > > -- > --------------------------------- > Mickaël Graf > Database developper, GBIF Sweden - http://www.gbif.se > tel: +46 8 5195 4045 > mailto:mickael.graf@nrm.se > > Swedish Museum of Natural History - http://www.nrm.se > Box 50007 > SE-104 05 Stockholm > Sweden > >

5 23

star schema functionality request
by Eli Agbayani 11 May '09

11 May '09

Markus, Tim, It is a concern for EOL usage of the IPT that the star schema is missing the functionality to add extension to another extension and not just always to the main DWC. The denormalized approach of doing actually this doesn't look good and providers will not like it. Another concern is the size of the recordset if we will (denormalize) repeat the main data object many times to accommodate many references, agents, audience type associated with it. We will continue testing IPT's usage for EOL while trying to minimize the overhead of the denormalized approach. But it would really be helpful to add the functionality making the star schema more relational. Thanks. Eli

3 3

star schema functionality request
by Eli Agbayani 08 May '09

08 May '09

Markus, Tim, It is a concern for EOL usage of the IPT that the star schema is missing the functionality to add extension to another extension and not just always to the main DWC. The denormalized approach of doing actually this doesn't look good and providers will not like it. Another concern is the size of the recordset if we will (denormalize) repeat the main data object many times to accommodate many references, agents, audience type associated with it. We will continue testing IPT's usage for EOL while trying to minimize the overhead of the denormalized approach. But it would really be helpful to add the functionality making the star schema more relational. Thanks. Eli

1 0

IPT test installation
by Michael Denslow 04 May '09

04 May '09

Hello Everyone, I have been attempting to run a demo of the IPT to show some colleagues how the toolkit works. I saw Tim Robertson give a run down of the tool in Virginia a few weeks ago, but I can't seem to recreate this. I am sure that I am missing something obvious here. I go to the Demo site at http://ipt.gbif.org/index.html and create a login and password. Login Then I only have the 'Home' and 'Explore' tabs showing. I am not sure how to start uploading the my example data. Thanks for any help you can provide, Michael -- Michael Denslow Graduate Student I.W. Carpenter Jr. Herbarium [BOON] Department of Biology Appalachian State University Boone, North Carolina U.S.A. -- AND -- Communications Manager Southeast Regional Network of Expertise and Collections sernec.org

2 1

Questions
by Mickael Graf 29 Apr '09

29 Apr '09

Hi again, I have now a working IPT on my server. But when I register the organisation name (under Admin->Settings->Organisation) I can pick up GBIF-Sweden as Title, but I never get a password (and since yesterday I didn't get any in my mailbox). Then I cannot register my IPT (Admin->Settings->IPT instance) because I need this password (which isn't the same as the UDDI one, isn't it ?). Cheers, Mickaël -- --------------------------------- Mickaël Graf Database developper, GBIF Sweden - http://www.gbif.se tel: +46 8 5195 4045 mailto:mickael.graf@nrm.se Swedish Museum of Natural History - http://www.nrm.se Box 50007 SE-104 05 Stockholm Sweden

2 3

Troubles with IPT
by Mickael Graf 27 Apr '09

27 Apr '09

Hi all, I downloaded and installed IPT and geoserver, and I got into troubles. I'm new to Tomcat, and this doesn't help me solving this issue. First I couldn't start geoserver. After searching the solution was to change owner (unzipping and installing defaults to root.root) and to make geoserver/data/service.xml writeable. Now I'm trying to start IPT but this doesn't work. After changing memory values, I finally get the following error message: FAIL - Application at context path /ipt could not be started The message from the logs is the following: INFO: HTMLManager: start: Starting web application at '/ipt' Resolved relative path / with servlet context : /usr/share/tomcat5/webapps/ipt/ Resolved relative path /data with servlet context : /usr/share/tomcat5/webapps/ipt/data Resolved relative path / with servlet context : /usr/share/tomcat5/webapps/ipt/ Resolved relative path /data with servlet context : /usr/share/tomcat5/webapps/ipt/data ERROR http-8080-Processor25-[/ipt].filterStart(3639) | Exception starting filter struts javax.xml.transform.TransformerFactoryConfigurationError: Provider org.apache.xalan.processor.TransformerFactoryImpl not found at javax.xml.transform.TransformerFactory.newInstance(Unknown Source) at com.opensymphony.xwork2.util.DomHelper $DOMBuilder.<clinit>(DomHelper.java:167) at com.opensymphony.xwork2.util.DomHelper.parse(DomHelper.java:115) at com.opensymphony.xwork2.config.providers.XmlConfigurationProvider.loadConfigurationFiles(XmlConfigurationProvider.java:830) at com.opensymphony.xwork2.config.providers.XmlConfigurationProvider.loadDocuments(XmlConfigurationProvider.java:131) at com.opensymphony.xwork2.config.providers.XmlConfigurationProvider.init(XmlConfigurationProvider.java:100) at com.opensymphony.xwork2.config.impl.DefaultConfiguration.reload(DefaultConfiguration.java:130) at com.opensymphony.xwork2.config.ConfigurationManager.getConfiguration(ConfigurationManager.java:52) at org.apache.struts2.dispatcher.Dispatcher.init_PreloadConfiguration(Dispatcher.java:395) at org.apache.struts2.dispatcher.Dispatcher.init(Dispatcher.java:452) at org.apache.struts2.dispatcher.FilterDispatcher.init(FilterDispatcher.java:201) at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:221) at org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:302) at org.apache.catalina.core.ApplicationFilterConfig.<init>(ApplicationFilterConfig.java:78) at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:3635) at org.apache.catalina.core.StandardContext.start(StandardContext.java:4222) at org.apache.catalina.manager.ManagerServlet.start(ManagerServlet.java:1176) at org.apache.catalina.manager.HTMLManagerServlet.start(HTMLManagerServlet.java:545) at org.apache.catalina.manager.HTMLManagerServlet.doGet(HTMLManagerServlet.java:105) at javax.servlet.http.HttpServlet.service(HttpServlet.java:690) at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:210) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:870) at org.apache.coyote.http11.Http11BaseProtocol $Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665) at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528) at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81) at org.apache.tomcat.util.threads.ThreadPool $ControlRunnable.run(ThreadPool.java:685) at java.lang.Thread.run(Thread.java:636) Apr 24, 2009 10:08:26 AM org.apache.catalina.core.StandardContext start SEVERE: Error filterStart Apr 24, 2009 10:08:26 AM org.apache.catalina.core.StandardContext start SEVERE: Context [/ipt] startup failed due to previous errors Apr 24, 2009 10:08:27 AM org.apache.catalina.core.ApplicationContext log INFO: HTMLManager: list: Listing contexts for virtual host 'localhost' I don't know if that means my system is missing some class/package or if it's a configuration issue. My system is Centos 5.3 Cheers, Mickaël -- --------------------------------- Mickaël Graf Database developper, GBIF Sweden - http://www.gbif.se tel: +46 8 5195 4045 mailto:mickael.graf@nrm.se Swedish Museum of Natural History - http://www.nrm.se Box 50007 SE-104 05 Stockholm Sweden

2 1

IPT RC1 released
by "Markus Döring (GBIF)" 03 Apr '09

03 Apr '09

Dear all, we are happy to announce the first release candidate of the IPT. As usual we have an online demo, a war archive & geoserver zip for your local installation and a mac standalone version to play with on your desktop without the need to install anything more: * online demo http://ipt.gbif.org * a war archive & geoserver.zip for your own tomcat installation * a mac standalone version incl geoserver http://code.google.com/p/gbif-providertoolkit/downloads/list?can=3 There you will also find the initial version of the user manual which Arnaud Reveillon is currently working on. He will be glad about any feedback you provide to this list and suggestions on what needs more explanations. If you encounter problems, please report them as usual in the issue tracker: http://code.google.com/p/gbif-providertoolkit/issues/list We are collecting bug fixes for the next release, but will provide snapshot versions for the interested on GBIFs ftp server: ftp://ftp.gbif.org/projects/ipt/downloads/ A Version history and list of fixed bugs for the snapshot is available too: http://code.google.com/p/gbif-providertoolkit/wiki/VersionHistory best wishes, Markus

1 0

IPT Documentation
by Arnaud Réveillon 01 Apr '09

01 Apr '09

Dear all, My name is Arnaud Réveillon and I have been appointed to write the documentation for the IPT. I am not new to the GBIF and biodiversity-related software as I worked during four years (2004-2008) for the Belgian focal point for the Convention on Biological Diversity, acting as tutor (mostly on databases, web applications, web pages editors, etc.) and as webmaster. During this period, I have been regularly in contact with members of several national GBIF nodes and am generally well aware of their work. I am not yet fully familiar with the IPT but hope to get up to speed in the next few days, with the help of the list. Please do not hesitate to let me know about your needs, the difficulties you are having with the IPT or anything that may be of interest regarding the documentation, which is scheduled to be ready on 16 April. Thank you, Arnaud Réveillon-

3 3

IPT 1.0b.8 online
by Markus Döring (GBIF) 22 Mar '09

22 Mar '09

Dear all, a new pre relase of the IPT is available for testing. You can try it here as usual: * online demo http://ipt.gbif.org * a war archive for your own tomcat installation http://gbif-providertoolkit.googlecode.com/files/ipt-1.0b.8.war * a mac standalone version without any further dependecies ftp://ftp.gbif.org/projects/ipt/downloads/ipt-1.0b.8.dmg Many, many thanks for the valuable feedback we have received in the past weeks. There are still outstanding bugs, but the new release 1.0b. 8 fixes many of them. Work has been done to fix SQL source problems, to provide a better property mapping interface, to connect to the GBIF registry (a test registry is currently configured) and to update darwin core extensions and vocabularies for controlled terms from central sites. If you havent tried the new Flash based maps provided by Javier de la Torre, give it a go. If you encounter problems, please report them as usual in the issue tracker: http://code.google.com/p/gbif-providertoolkit/issues/list best wishes, Markus

4 6