- IPT - lists.gbif.org

Open call for testing of the IPT 2.0RC1
by Tim Robertson 13 Dec '10

13 Dec '10

Dear all, The IPT development team are pleased to invite you to participate in the testing of the re-engineered and simplified IPT. The revised IPT is the result of a re-engineering effort of the 1.0 version of the IPT, specifically to: - Reduce the server requirements significantly - Increase the data import performance by removing the embedded database - Remove the dependency on heavy libraries and tools such as Geoserver - Remove all data interfaces that are not necessary for data publication through GBIF - Improve the robustness of the tool The result is an IPT that: - is less than 12MB - starts in only a few seconds - supports fast import (1 million records in 15 seconds or so) - includes an expanded metadata profile - supports the importing of DwC Archives - allows multiple organisation support (an IPT may be installed by an institution, but the resource associated with a different institution) We would like to thank those early testers who have helped us, over the last 5 weeks, identify and address some 81 issues and enhancements already (http://code.google.com/p/gbif-providertoolkit/issues/list?can=1&q=Milestone… ). All reported defects have been addressed, and therefore we have indicated this is a version that is a candidate for being a formal release and therefore labelled IPT 2.0RC1. Issues are actively being tracked, monitored and will be prioritised and addressed, and I ask that you use this utility to record any issues that you come across: http://code.google.com/p/gbif-providertoolkit/issues/list The user manual and engineering documentation drafting are under way and expected to be finished by the end of 2010 prior to the general availability (IPT 2.0GA) release. The IPT 2.0GA release is targeted by the end of the 2010 but will only be made available following a successful community test phase. The IPT 2.0RC1 for this test can be downloaded: http://gbif-providertoolkit.googlecode.com/files/ipt-2.0RC1.war The IPT 2.0RC1 is restricted for use to the GBIF development registry at this stage and therefore can not be used for publication onto the GBIF network during this testing phase. Please find attached some basic guideline information for testers in the absence of the user manual. Thank you once again to all IPT contributers. Best wishes, Tim

5 4

v2.0 release timeline?
by Bruce Wilson 18 Nov '10

18 Nov '10

Hi All. Does an timeline exist for deployment of the v2.0 IPT package? I've been moving slowly on publishing via the v1.0-RC3 package anticipating the release of v2.0. Bruce Wilson - UVSC Department of Chemistry | Mail Stop 179 | UVU | 800 W University Parkway, Orem UT 84058 (801)863-7138 | bruce.wilson(a)uvu.edu | http://science.uvu.edu/wilson

2 1

Notes on configuring RHEL to run IPT (ipt-2.0-SNAPSHOT-r2482)
by Matt 17 Nov '10

17 Nov '10

Hi all, Some notes on my recent effort to get IPT running on RHEL- at minimum they'll help me remember howto repeat this for real in the future. Special thanks to Tim for the very speedy help. One quick question now- if I configure everything is it easy to save the configuration file, then wipe the install, then toss the config back in? Cheers, Matt --- Running RHEL (Realm Linux flavour), see http://www.linux.ncsu.edu/realm_linux/. Did (all) this as a su. Yum installed packages of note (I'm not sure if these were all required or not, it's just the overall configuration): java-1.4.2-gcj-compat.x86_64 java-1.4.2-gcj-compat-devel.x86_64 java-1.6.0-openjdk.x86_64 java-1.6.0-openjdk-devel.x86_64 java_cup.x86_64 xalan-j2.x86_64 xalan-j2-xsltc.x86_64 xerces-j2.x86_64 xerces-j2-scripts.x86_64 tomcat5.x86_64 tomcat5-common-lib.x86_64 tomcat5-jasper.x86_64 tomcat5-jsp-2.0-api.x86_64 tomcat5-server-lib.x86_64 tomcat5-servlet-2.4-api.x86_64 tomcat5-webapps.x86_64 I answered y to all the dependency questions on yum installs. It took me a bit to figure out that the the webapps folder linked at /usr/share/tomcat5/webapps or /var/lib/tomcat5/webapps After cding there I grabbed the files wget http://gbif-providertoolkit.googlecode.com/files/ipt-2.0-SNAPSHOT-r2482.war wget http://gbif-providertoolkit.googlecode.com/files/ipt-geoserver-1.0-RC3.zip Then I chowned/chgrped them to user 'tomcat' and did 'service tomcat5 restart'. After you hit http://your.site.com:8080/ipt-2.0-SNAPSHOT-r2482 you'll have a ipt-2.0-SNAPSHOT-r2482 folder back in webapps. Went there, and edited webapps/ipt/WEB-INF/classes/log4j.xml changing the log lines to: <param name="file" value="/tmp/debug.log "/> <param name="file" value="/tmp/admin.log "/> Note that the Issue 451 fix describing this problem doesn't include the slash preceding tmp in the steps body but the attached file does (the latter is correct). The last trick was to: cd /var/lib/tomcat5/common/endorsed and do a number of "rebuild-jar-repository" rebuild-jar-repository /var/lib/tomcat5/common/endorsed xalan-j2-serializer.jar rebuild-jar-repository /var/lib/tomcat5/common/endorsed xalan-j2.jar rebuild-jar-repository /var/lib/tomcat5/common/endorsed xerces-j2.jar ... and some others The end product of the endorsed folder looks like this: [jaxp_parser_impl].jar -> /usr/share/java/jaxp_parser_impl.jar [xalan-j2-2.7.0].jar -> /usr/share/java/xalan-j2-2.7.0.jar [xalan-j2].jar -> /usr/share/java/xalan-j2.jar [xalan-j2-serializer].jar -> /usr/share/java/xalan-j2-serializer.jar [xerces-j2-2.7.1].jar -> /usr/share/java/xerces-j2-2.7.1.jar [xerces-j2].jar -> /usr/share/java/xerces-j2.jar [xml-commons-apis].jar -> /usr/share/java/xml-commons-apis.jar The xalan-j2-serializer.jar was definitely needed as including this link ultimately resolved the problem. I'm not sure which others were required. After all this (and at various points in the debugging) a "service tomcat5 restart" and the site was live.

2 1

Configuring IPT for RHEL/Centos?
by Matt 17 Nov '10

17 Nov '10

Hi All, Trying to get IPT running. I assume it's possible to do on linux? Is it possible to configure remotely (i.e. not going the localhost route as per the docs)? I think I'm configured thus far correctly since I can see /geoserver fine. I tried both the 1.0 RC3 and the 2.0 invite versions. Present error is a 404, resource () not found. Running Apache Tomcat/5.5.23 installed via yum. Any thoughts? I'd be happy to document the process if I can get something running. Thanks in advance, Matt

3 3

Re: [IPT] IPT v2 Snapshot frequency
by "Markus Döring (GBIF)" 15 Oct '10

15 Oct '10

when setting up a data directory for the IPT a small file called "datadir.location" is created in the WEB-INF folder that points to the actual data dir. If you simply remove this file you force the IPT to the setup page. You can enter an existing data dir if you want and all its configuration is loaded - or create a new, empty one. For jetty when running "mvn" this file is kept with the source files. To remove it simply do: $ cd gbif-ipt $ rm src/main/webapp/WEB-INF/datadir.location Markus On Oct 15, 2010, at 14:12, Nicolas Noé wrote: > One last question, when playing with it (without building the war, letting mvn launch jetty), how can I restore it to its original state (to lose all configurations and send me back on the first (select datat dir) page ? > > Thanks, > > Nicolas > > On 10/15/2010 12:07 PM, "Markus Döring (GBIF)" wrote: >> Nicolas, >> that is indeed a good idea. We will consider some regular build of the war, maybe even on a daily basis. I will upload a new version later today once the serious bugs are fixed that Ive introduced when refactoring the http/registry communication. >> >> If you ever want to try the latest trunk from the svn its rather simple if you have java, maven and svn installed (OSX comes with all of these): >> >> $ svn checkout http://gbif-providertoolkit.googlecode.com/svn/trunk/gbif-ipt >> $ cd gbif-ipt >> $ mvn >> >> The last "mvn" command should start up the IPT with a local jetty instance on port 7001 and you can go into your browser of choice with this url: >> http://localhost:7001/ipt >> >> If you want to build a war instead you need to issue a different maven command: >> >> $ mvn install >> >> and then you should find the war in this folder (example revision 2396): >> >> $ cp target/ipt-2.0-RC1-SNAPSHOT-r2396.war >> >> >> When you do the maven command for the very first time it will download quite a few libraries, so you need some patience at that point. But when done once it will be fairly fast afterwards, even if you update to the latest trunk again. That can be done like this by the way: >> >> $ cd gbif-ipt >> $ svn up >> >> Markus >> >> >> On Oct 15, 2010, at 11:48, Nicolas Noé wrote: >> >>> Hi, >>> >>> With the current move to the v2 of the IPT and the very fast development of the last few weeks, I think it could be great to provide development snapshots (similar to the one Tim uploaded a few days ago) on a regular and very frequent basis. >>> >>> For example, I've reported a bug 2 days ago that is supposed to be fixed, and I'd like to re-test this and investigate deeper the IPT (hat bug make me stuck at the beginning), so I could follow the development day after day and provide immediate feedback to the development team. I think this transparency and rapid feedback mechanism is important if we want a very fast development rate and at the same time keep a good stability. It could also lead to a better communication between IPT developers and end users (for example to keep them aware of which features/bugs should be on the top-priority list). >>> >>> I see 3 ways to make these builds for "faster feedback" possible : >>> >>> 1) The GBIF manually creates "ipt-snapshot.war" on a regular basis (i.e. weekly) >>> 2) Putting an automatic mechanism to create these builds, the frequence could so be increased (nightly builds). >>> 3) Update (and detail) the documentation of the build process on the wiki (available for v1). The java build process/ build tools and the quirks that can be encountered at that step can be, to say the least, quite intimidating for someone who isn't a Java developer. >>> >>> What do you think of this ? >>> >>> Thanks, Nicolas. >>> <niconoe.vcf>_______________________________________________ >>> IPT mailing list >>> IPT(a)lists.gbif.org >>> http://lists.gbif.org/mailman/listinfo/ipt > > <niconoe.vcf>

1 0

IPT v2 Snapshot frequency
by Nicolas Noé 15 Oct '10

15 Oct '10

Hi, With the current move to the v2 of the IPT and the very fast development of the last few weeks, I think it could be great to provide development snapshots (similar to the one Tim uploaded a few days ago) on a regular and very frequent basis. For example, I've reported a bug 2 days ago that is supposed to be fixed, and I'd like to re-test this and investigate deeper the IPT (hat bug make me stuck at the beginning), so I could follow the development day after day and provide immediate feedback to the development team. I think this transparency and rapid feedback mechanism is important if we want a very fast development rate and at the same time keep a good stability. It could also lead to a better communication between IPT developers and end users (for example to keep them aware of which features/bugs should be on the top-priority list). I see 3 ways to make these builds for "faster feedback" possible : 1) The GBIF manually creates "ipt-snapshot.war" on a regular basis (i.e. weekly) 2) Putting an automatic mechanism to create these builds, the frequence could so be increased (nightly builds). 3) Update (and detail) the documentation of the build process on the wiki (available for v1). The java build process/ build tools and the quirks that can be encountered at that step can be, to say the least, quite intimidating for someone who isn't a Java developer. What do you think of this ? Thanks, Nicolas.

2 1

data base wont publish
by Fabian Haas 01 Oct '10

01 Oct '10

Dear All, would need help with some IPT issue (1.0-RC3-SNAPSHOT). I submitted two data bases to GBIF and they work fine, no I added another one with just about 155 data sets. checking on the contents with the explore tools the data have read in properly. Now it say "modified, not publish", clicking on "publish" gets at best a time out error, but nothing happens. neither on my side, changing to published, nor gets Andrea Hahn a message. I have tried several times on different days to exclude Internet issues to no effect. One more thing, which does not bother me, but might be a hint. Some of the meta data like Geographic Coverage, do nort save but get lost, when I leave the toolkit http://kupe.icipe.org:8080/ipt/index.html Any ideas what it could be? best Fabian -- ********************************************************** fhaas(a)icipe.org, Extension -2052 The African Insect Taxonomy Toolkit AITT http://taxonomy.icipe.org Dr. Fabian Haas Insect Taxonomist and ABS Specialist ICIPE - African Insect Science for Food and Health Duduville Campus, Kasarani P.O. Box 30772 - 00100 N A I R O B I Kenya Telephone No. +254 (0)20 8632000 Fax No. +254 (0)20 8632001 Cell Phone +254 (0)728 132868 This position is supported by CIM, the Centre for International Migration www.cimonline.de

1 0

Re: [IPT] GBIF Integrated Publishing Toolkit
by Tim Robertson (GBIF) 20 Sep '10

20 Sep '10

Dear Jonas, Thank you for your questions. Answers given inline: > Dear Tim > > Thanks for the information, and the great work on the IPT. I think > it will be a good move, just seen from my perspective, to > concentrate on the core functionality for better performance. Thank you for this feedback - this is important to help steer the IPT process. > After having done some promotion and held some training and > installation sessions of the IPT under the GBIF - NordGen evaluation > project I see that the general interest in the genebank > documentation community is quite strong and the installation and > configuration is very user-friendly. We just this week had a > workshop in Estonia introducing the IPT and the GBIF network to > people from genebanks in Estonia, Latvia and Lithuania and the > response was very good. > > To be able to make the most out of these training sessions, there > will be a few more during the remainder of this year, I have a > couple of questions about the coming version: > > if the GUI for mapping imported dataset fields to the DwC / DwC > extension fields will be significantly different? It will be slightly different, but familiar enough for people who understand the older version to pick it up easily. We received feedback during the IPT experts training event last year, and have implemented their recommendations. In short, the terms are simply provided in a long list. I attach a screenshot (dwc-mapping.png) showing the mapping page for occurrences. Please remember that we will initiate a testing phase in October, and there will be the ability to comment on this and have it revised if there are problems. > > if the mapping through SQL / direct database connection will work > significantly different? No changes here. You create the SQL statement and then do a mapping the same, regardless of whether it is a SQL source or text file. > if it will be possible to host multiple resources registered under > different GBIF nodes on the same IPT webapp or if all resources on > one IPT webapp must be under the same GBIF node? The revised version will handle this: The IPT instance is registered with the Institution/Organisation/Node to which it is associated. The ADMIN user can then associate further Institution/Organisation/ Nodes from which data MANAGERs can choose when registering a data RESOURCE. Therefore the following could be handled: - ADMIN in Node1 installs the IPT - ADMIN enables Node2 and Institution3 as available entities for MANAGERs to use - ADMIN creates accounts for MANAGERs - MANAGERs can log in and create resources - when a MANAGER creates a resource they see a choice of Node1, Node2 or Institution3 to associate the resource with In addition to this, the ADMIN can provide an alias to each Organisation/Node/Institution, so that MANAGERs see something familiar. For example the GBIF Registry might have "The Botanical holdings of the Harvard Herbarium Museum" but internally the ADMIN and MANAGERs all call this "HUH". The ADMIN will be able to declare this and MANAGERs will simply see "HUH" in the choice of Institutions to associate the resource with. I hope this helps, and thanks again for the feedback, Tim > > Best regards > > Jonas Nordling, NordGen > From: trobertson(a)gbif.org > To: jonas.nordling(a)nordgen.org > Sent: Mon, 13 Sep 2010 16:10:36 +0200 > Subject: GBIF Integrated Publishing Toolkit > > [FOR INFORMATION] > > Dear Node Managers and Data Publishers, > > The objective of this communication is to update you on the current > situation of the GBIF Integrated Publishing Toolkit (IPT). The IPT > is a tool providing publishing capabilities for primary biodiversity > occurrence data, taxonomic checklists and the associated metadata > for these resource types. > > Since its first introduction in March 2009, the IPT has been used > with some success by a limited number of users to publish primary > biodiversity data into the GBIF network, along with descriptive > metadata at the dataset level. > > The GBIF Secretariat (GBIFS) has solicited and received a vast > amount of feedback during the first year of the IPT use, and would > like to thank all those who provided this. The feedback received > overwhelmingly confirms that the concept of the IPT is sound. > However, feedback has also made it clear that it is over-specified > for the majority of user needs, is not yet sufficiently robust, is > too slow in operation, and (due to the excessive functionality) has > high server requirements creating a barrier for many to adopt and > use it. In addition, the feedback has made it clear that our > communications surrounding the status of the IPT releases have been > too infrequent and unclear concerning which release candidate we > were offering and what functionalities this release candidate would > include or not include, as per user needs/expectations. > > Based upon the extensive feedback GBIFS has revised our planned > development roadmap for the IPT and are currently performing a major > refactoring effort to: > • Reduce the server requirements significantly > • Increase the data import performance by removing the embedded > database > • Remove the dependency on heavy libraries and tools such as Geoserver > • Remove all data interfaces that are not necessary for data > publication through GBIF > • Improve the robustness of the tool > > When complete, the refactored IPT will offer through an intuitive > interface: > • Authoring of metadata according to the GBIF metadata profile > • Import and mapping of checklist and occurrence data either from > file upload or by connection of a database > • Registration to and publication onto the GBIF network > • Importing and mapping of data to the DarwinCore, and DarwinCore > extensions > • Output formats of EML (2.1.0) metadata and DarwinCore Archive > • Improved customisation options for the “About this IPT” > • Improved GBIF Registry integration > • Improved DarwinCore extension and vocabulary organization > • Improved management of the organisations to which the IPT and > resources are related, to enable co-hosting capabilities > • Enhanced dataset metadata authoring > > Whilst this is a reduction in existing functionality this has been > deemed necessary to ensure the IPT is meeting the core requirements > as expressed by the user community. The following features will be > removed and only reintroduced if necessary through consultation with > the IPT community: > • TAPIR interface > • TCS output format > • Search and browse web interface > • OGC web services > > All IPT development efforts are now focused on this revised roadmap, > and it is anticipated that early testing of the new codebase will > commence in October 2010 with the involvement of willing Nodes. The > release of a fully stable IPT is targeted for the end of 2010. > However, this will only occur if the testing community is satisfied > that the product is ‘release-ready’ together with the required user > manual and technical documentation. Should that not be the case, as > it may take more time to implement and test the improvements > outlined above, you will be informed as soon as possible. > > Further developments of the IPT beyond this release will only be > implemented after a further scoping with the GBIF community. An area > for further IPT discussion will be set up on the GBIF community > website (http://community.gbif.org) > > If you have any questions or would like to know more about the GBIF > IPT and/or revised process, feel free to contact: > > Tim Robertson > GBIF Information Systems Architect > trobertson(a)gbif.org > > With best regards, > > Tim Robertson > GBIF Secretariat > > >

1 0

Functionality request: ADMIN checking data before GBIF registration
by Tim Robertson (GBIF) 16 Sep '10

16 Sep '10

Hi all, Hannu has raised a request for the following to be satisfied by the IPT: "- Publishing a resource must be accepted by the owner of the provider. It has happened that a test user publishes something odd which goes all the way to the data portal without nobody controlling it." This is a contradiction to the requests of others, and specifically those wishing to promote basic "data hosting centers", who request that a data MANAGER should be able to work autonomously. After discussion with the developers the proposal is to implement the following, which we hope satisfies both requirements: In the Administration section, an ADMIN can choose to enable or disable the ability for MANAGERS to register resources with GBIF. By default MANAGERS can register a resource, but an ADMIN can disable this through this check box. If anyone has any concerns or comments on this approach, please can you raise them on this list? Many thanks, Tim

4 5

Re: [IPT] Functionality request: ADMIN checking data before GBIFregistration
by Tim Robertson (GBIF) 15 Sep '10

15 Sep '10

Thanks all for the comments, which I will try and collate here with some resolutions: Propose resolved: a) The preference would be to allow the ADMIN to provide Registration privileges to individual MANAGERS Outstanding issues: b) Visualisations are important, and will need discussion and potentially further IPT modules developed, or deployment of external services. I know of one group considering the development of DwC-A visualisations already, and technologies like Google Fusion tables makes this kind of thing trivial. c) record resolution is something that has been indicated as important. This can be achieved in one of 2 ways: > implementation in the IPT, and requires research into technologies to perform satisfactorily Potential technologies might be - Berkeley DB - A relational database, such as Mysql or H2 - Lucene indexing > reliance on a "stable cache" for record serving The first release (for test purposes) of the revised IPT software will not have record level serving, but while this is being developed, I would like to ask people to start discussing what kind of record level serving is truly a requirement in the IPT, as opposed to a "nice to have". We support DwC-A in the GBIF portal, and the intention is to simply reserve the record that came from the DwC-A directly, unless the record indicates there is further information on a URL (e.g. if the record identifier is an LSID). Would this strategy not be suitable for the likes of the BioCASe portals as often there is no further information to redirect to? in the case of the IPT, there is no extra information, and I propose should the source be a DwC-A, that individual records be cached in the harvesting portal. With this approach, there would be no individual record serving needs in the IPT. Ultimately, we might consider aiming for data owners offering single records on a resolvable URL, and conforming to Linked/Open Data requirements, along with a DwC-A effectively providing a single "index" view of the dataset. The DwC-A records would reference the originals by resolvable ID so any search system would always be able to point back to the authoritative source. This would effectively be distributed indexing, and not dissimilar to the sharing of sitemaps, but with extra information to enable better discovery. Thank you all for this feedback, and please correct any misunderstandings on my part Tim On Sep 15, 2010, at 12:03 PM, Mihail-Constantin Carausu wrote: > Dear Tim > > I think the development team's mentioned approach is a workable > solution > to cover both requirements at this stage. > However, I think Hannu (and me) had in mind a kind of "Basket of > approvals"-alike functionality in the Admin's (owner of the provider) > administration section side: When a Manager has been published a > dataset > through the IPT, this will automatically trigger a request for > approval > or submits an yes/no event in the Admin's administration section. The > Admin must finally active interfere and approve the dataset > publication > (e.g. by checking an "Approved" check box in the basket of approvals > list with events/datasets in the administration section) at the > absolute > latest stage (e.g. when GBIF just needs to start to index it, or > something like that). Without this final approval the dataset will > still > be published and visible through the IPT but not visible/searchable on > the GBIF data portal. This approach is not necessarily in > contradiction > with the Manager's ability to autonomously publish datasets within the > IPT, only it puts this ability always under control from the central > administration section when the dataset has to go to the GBIF data > portal. > I think both solutions/approaches have obvious advantages and > disadvantages while none of them provides a 100% protection against > publishing something odd by a (test) user. > I have a little question regarding the development team's proposed > solutions: is it not possible for the central Admin to enable the > publishing ability for some "trusted" managers and disable this for > others inside the same instance of the IPT. > > Now I saw Hannu's new message just arrived, sorry for eventually > unsynchronized double-crossing messages, but I will send this anyhow. > > Best regards, > Mihail > > ---------------------------- > Mihail Carausu > MSc.Eng., Informatics Manager > Danish Biodiversity Information Facility (DanBIF) > -------------------------------------------- > > > -----Original Message----- > From: ipt-bounces(a)lists.gbif.org [mailto:ipt-bounces@lists.gbif.org] > On > Behalf Of Tim Robertson (GBIF) > Sent: 15. september 2010 09:43 > To: ipt(a)lists.gbif.org > Subject: [IPT] Functionality request: ADMIN checking data before > GBIFregistration > > Hi all, > > Hannu has raised a request for the following to be satisfied by the > IPT: > "- Publishing a resource must be accepted by the owner of the > provider. It has happened that a test user publishes something odd > which goes all the way to the data portal without nobody controlling > it." > > This is a contradiction to the requests of others, and specifically > those wishing to promote basic "data hosting centers", who request > that a data MANAGER should be able to work autonomously. > > After discussion with the developers the proposal is to implement the > following, which we hope satisfies both requirements: > In the Administration section, an ADMIN can choose to enable or > disable the ability for MANAGERS to register resources with GBIF. By > default MANAGERS can register a resource, but an ADMIN can disable > this through this check box. > > If anyone has any concerns or comments on this approach, please can > you raise them on this list? > > Many thanks, > Tim > > > > > > > _______________________________________________ > IPT mailing list > IPT(a)lists.gbif.org > http://lists.gbif.org/mailman/listinfo/ipt >

2 2