- IPT - lists.gbif.org

IPT v2 Snapshot frequency
by Nicolas Noé 15 Oct '10

15 Oct '10

Hi, With the current move to the v2 of the IPT and the very fast development of the last few weeks, I think it could be great to provide development snapshots (similar to the one Tim uploaded a few days ago) on a regular and very frequent basis. For example, I've reported a bug 2 days ago that is supposed to be fixed, and I'd like to re-test this and investigate deeper the IPT (hat bug make me stuck at the beginning), so I could follow the development day after day and provide immediate feedback to the development team. I think this transparency and rapid feedback mechanism is important if we want a very fast development rate and at the same time keep a good stability. It could also lead to a better communication between IPT developers and end users (for example to keep them aware of which features/bugs should be on the top-priority list). I see 3 ways to make these builds for "faster feedback" possible : 1) The GBIF manually creates "ipt-snapshot.war" on a regular basis (i.e. weekly) 2) Putting an automatic mechanism to create these builds, the frequence could so be increased (nightly builds). 3) Update (and detail) the documentation of the build process on the wiki (available for v1). The java build process/ build tools and the quirks that can be encountered at that step can be, to say the least, quite intimidating for someone who isn't a Java developer. What do you think of this ? Thanks, Nicolas.

2 1

data base wont publish
by Fabian Haas 02 Oct '10

02 Oct '10

Dear All, would need help with some IPT issue (1.0-RC3-SNAPSHOT). I submitted two data bases to GBIF and they work fine, no I added another one with just about 155 data sets. checking on the contents with the explore tools the data have read in properly. Now it say "modified, not publish", clicking on "publish" gets at best a time out error, but nothing happens. neither on my side, changing to published, nor gets Andrea Hahn a message. I have tried several times on different days to exclude Internet issues to no effect. One more thing, which does not bother me, but might be a hint. Some of the meta data like Geographic Coverage, do nort save but get lost, when I leave the toolkit http://kupe.icipe.org:8080/ipt/index.html Any ideas what it could be? best Fabian -- ********************************************************** fhaas(a)icipe.org, Extension -2052 The African Insect Taxonomy Toolkit AITT http://taxonomy.icipe.org Dr. Fabian Haas Insect Taxonomist and ABS Specialist ICIPE - African Insect Science for Food and Health Duduville Campus, Kasarani P.O. Box 30772 - 00100 N A I R O B I Kenya Telephone No. +254 (0)20 8632000 Fax No. +254 (0)20 8632001 Cell Phone +254 (0)728 132868 This position is supported by CIM, the Centre for International Migration www.cimonline.de

1 0

Re: [IPT] GBIF Integrated Publishing Toolkit
by Tim Robertson (GBIF) 21 Sep '10

21 Sep '10

Dear Jonas, Thank you for your questions. Answers given inline: > Dear Tim > > Thanks for the information, and the great work on the IPT. I think > it will be a good move, just seen from my perspective, to > concentrate on the core functionality for better performance. Thank you for this feedback - this is important to help steer the IPT process. > After having done some promotion and held some training and > installation sessions of the IPT under the GBIF - NordGen evaluation > project I see that the general interest in the genebank > documentation community is quite strong and the installation and > configuration is very user-friendly. We just this week had a > workshop in Estonia introducing the IPT and the GBIF network to > people from genebanks in Estonia, Latvia and Lithuania and the > response was very good. > > To be able to make the most out of these training sessions, there > will be a few more during the remainder of this year, I have a > couple of questions about the coming version: > > if the GUI for mapping imported dataset fields to the DwC / DwC > extension fields will be significantly different? It will be slightly different, but familiar enough for people who understand the older version to pick it up easily. We received feedback during the IPT experts training event last year, and have implemented their recommendations. In short, the terms are simply provided in a long list. I attach a screenshot (dwc-mapping.png) showing the mapping page for occurrences. Please remember that we will initiate a testing phase in October, and there will be the ability to comment on this and have it revised if there are problems. > > if the mapping through SQL / direct database connection will work > significantly different? No changes here. You create the SQL statement and then do a mapping the same, regardless of whether it is a SQL source or text file. > if it will be possible to host multiple resources registered under > different GBIF nodes on the same IPT webapp or if all resources on > one IPT webapp must be under the same GBIF node? The revised version will handle this: The IPT instance is registered with the Institution/Organisation/Node to which it is associated. The ADMIN user can then associate further Institution/Organisation/ Nodes from which data MANAGERs can choose when registering a data RESOURCE. Therefore the following could be handled: - ADMIN in Node1 installs the IPT - ADMIN enables Node2 and Institution3 as available entities for MANAGERs to use - ADMIN creates accounts for MANAGERs - MANAGERs can log in and create resources - when a MANAGER creates a resource they see a choice of Node1, Node2 or Institution3 to associate the resource with In addition to this, the ADMIN can provide an alias to each Organisation/Node/Institution, so that MANAGERs see something familiar. For example the GBIF Registry might have "The Botanical holdings of the Harvard Herbarium Museum" but internally the ADMIN and MANAGERs all call this "HUH". The ADMIN will be able to declare this and MANAGERs will simply see "HUH" in the choice of Institutions to associate the resource with. I hope this helps, and thanks again for the feedback, Tim > > Best regards > > Jonas Nordling, NordGen > From: trobertson(a)gbif.org > To: jonas.nordling(a)nordgen.org > Sent: Mon, 13 Sep 2010 16:10:36 +0200 > Subject: GBIF Integrated Publishing Toolkit > > [FOR INFORMATION] > > Dear Node Managers and Data Publishers, > > The objective of this communication is to update you on the current > situation of the GBIF Integrated Publishing Toolkit (IPT). The IPT > is a tool providing publishing capabilities for primary biodiversity > occurrence data, taxonomic checklists and the associated metadata > for these resource types. > > Since its first introduction in March 2009, the IPT has been used > with some success by a limited number of users to publish primary > biodiversity data into the GBIF network, along with descriptive > metadata at the dataset level. > > The GBIF Secretariat (GBIFS) has solicited and received a vast > amount of feedback during the first year of the IPT use, and would > like to thank all those who provided this. The feedback received > overwhelmingly confirms that the concept of the IPT is sound. > However, feedback has also made it clear that it is over-specified > for the majority of user needs, is not yet sufficiently robust, is > too slow in operation, and (due to the excessive functionality) has > high server requirements creating a barrier for many to adopt and > use it. In addition, the feedback has made it clear that our > communications surrounding the status of the IPT releases have been > too infrequent and unclear concerning which release candidate we > were offering and what functionalities this release candidate would > include or not include, as per user needs/expectations. > > Based upon the extensive feedback GBIFS has revised our planned > development roadmap for the IPT and are currently performing a major > refactoring effort to: > • Reduce the server requirements significantly > • Increase the data import performance by removing the embedded > database > • Remove the dependency on heavy libraries and tools such as Geoserver > • Remove all data interfaces that are not necessary for data > publication through GBIF > • Improve the robustness of the tool > > When complete, the refactored IPT will offer through an intuitive > interface: > • Authoring of metadata according to the GBIF metadata profile > • Import and mapping of checklist and occurrence data either from > file upload or by connection of a database > • Registration to and publication onto the GBIF network > • Importing and mapping of data to the DarwinCore, and DarwinCore > extensions > • Output formats of EML (2.1.0) metadata and DarwinCore Archive > • Improved customisation options for the “About this IPT” > • Improved GBIF Registry integration > • Improved DarwinCore extension and vocabulary organization > • Improved management of the organisations to which the IPT and > resources are related, to enable co-hosting capabilities > • Enhanced dataset metadata authoring > > Whilst this is a reduction in existing functionality this has been > deemed necessary to ensure the IPT is meeting the core requirements > as expressed by the user community. The following features will be > removed and only reintroduced if necessary through consultation with > the IPT community: > • TAPIR interface > • TCS output format > • Search and browse web interface > • OGC web services > > All IPT development efforts are now focused on this revised roadmap, > and it is anticipated that early testing of the new codebase will > commence in October 2010 with the involvement of willing Nodes. The > release of a fully stable IPT is targeted for the end of 2010. > However, this will only occur if the testing community is satisfied > that the product is ‘release-ready’ together with the required user > manual and technical documentation. Should that not be the case, as > it may take more time to implement and test the improvements > outlined above, you will be informed as soon as possible. > > Further developments of the IPT beyond this release will only be > implemented after a further scoping with the GBIF community. An area > for further IPT discussion will be set up on the GBIF community > website (http://community.gbif.org) > > If you have any questions or would like to know more about the GBIF > IPT and/or revised process, feel free to contact: > > Tim Robertson > GBIF Information Systems Architect > trobertson(a)gbif.org > > With best regards, > > Tim Robertson > GBIF Secretariat > > >

1 0

Functionality request: ADMIN checking data before GBIF registration
by Tim Robertson (GBIF) 16 Sep '10

16 Sep '10

Hi all, Hannu has raised a request for the following to be satisfied by the IPT: "- Publishing a resource must be accepted by the owner of the provider. It has happened that a test user publishes something odd which goes all the way to the data portal without nobody controlling it." This is a contradiction to the requests of others, and specifically those wishing to promote basic "data hosting centers", who request that a data MANAGER should be able to work autonomously. After discussion with the developers the proposal is to implement the following, which we hope satisfies both requirements: In the Administration section, an ADMIN can choose to enable or disable the ability for MANAGERS to register resources with GBIF. By default MANAGERS can register a resource, but an ADMIN can disable this through this check box. If anyone has any concerns or comments on this approach, please can you raise them on this list? Many thanks, Tim

4 5

Re: [IPT] Functionality request: ADMIN checking data before GBIFregistration
by Tim Robertson (GBIF) 16 Sep '10

16 Sep '10

Thanks all for the comments, which I will try and collate here with some resolutions: Propose resolved: a) The preference would be to allow the ADMIN to provide Registration privileges to individual MANAGERS Outstanding issues: b) Visualisations are important, and will need discussion and potentially further IPT modules developed, or deployment of external services. I know of one group considering the development of DwC-A visualisations already, and technologies like Google Fusion tables makes this kind of thing trivial. c) record resolution is something that has been indicated as important. This can be achieved in one of 2 ways: > implementation in the IPT, and requires research into technologies to perform satisfactorily Potential technologies might be - Berkeley DB - A relational database, such as Mysql or H2 - Lucene indexing > reliance on a "stable cache" for record serving The first release (for test purposes) of the revised IPT software will not have record level serving, but while this is being developed, I would like to ask people to start discussing what kind of record level serving is truly a requirement in the IPT, as opposed to a "nice to have". We support DwC-A in the GBIF portal, and the intention is to simply reserve the record that came from the DwC-A directly, unless the record indicates there is further information on a URL (e.g. if the record identifier is an LSID). Would this strategy not be suitable for the likes of the BioCASe portals as often there is no further information to redirect to? in the case of the IPT, there is no extra information, and I propose should the source be a DwC-A, that individual records be cached in the harvesting portal. With this approach, there would be no individual record serving needs in the IPT. Ultimately, we might consider aiming for data owners offering single records on a resolvable URL, and conforming to Linked/Open Data requirements, along with a DwC-A effectively providing a single "index" view of the dataset. The DwC-A records would reference the originals by resolvable ID so any search system would always be able to point back to the authoritative source. This would effectively be distributed indexing, and not dissimilar to the sharing of sitemaps, but with extra information to enable better discovery. Thank you all for this feedback, and please correct any misunderstandings on my part Tim On Sep 15, 2010, at 12:03 PM, Mihail-Constantin Carausu wrote: > Dear Tim > > I think the development team's mentioned approach is a workable > solution > to cover both requirements at this stage. > However, I think Hannu (and me) had in mind a kind of "Basket of > approvals"-alike functionality in the Admin's (owner of the provider) > administration section side: When a Manager has been published a > dataset > through the IPT, this will automatically trigger a request for > approval > or submits an yes/no event in the Admin's administration section. The > Admin must finally active interfere and approve the dataset > publication > (e.g. by checking an "Approved" check box in the basket of approvals > list with events/datasets in the administration section) at the > absolute > latest stage (e.g. when GBIF just needs to start to index it, or > something like that). Without this final approval the dataset will > still > be published and visible through the IPT but not visible/searchable on > the GBIF data portal. This approach is not necessarily in > contradiction > with the Manager's ability to autonomously publish datasets within the > IPT, only it puts this ability always under control from the central > administration section when the dataset has to go to the GBIF data > portal. > I think both solutions/approaches have obvious advantages and > disadvantages while none of them provides a 100% protection against > publishing something odd by a (test) user. > I have a little question regarding the development team's proposed > solutions: is it not possible for the central Admin to enable the > publishing ability for some "trusted" managers and disable this for > others inside the same instance of the IPT. > > Now I saw Hannu's new message just arrived, sorry for eventually > unsynchronized double-crossing messages, but I will send this anyhow. > > Best regards, > Mihail > > ---------------------------- > Mihail Carausu > MSc.Eng., Informatics Manager > Danish Biodiversity Information Facility (DanBIF) > -------------------------------------------- > > > -----Original Message----- > From: ipt-bounces(a)lists.gbif.org [mailto:ipt-bounces@lists.gbif.org] > On > Behalf Of Tim Robertson (GBIF) > Sent: 15. september 2010 09:43 > To: ipt(a)lists.gbif.org > Subject: [IPT] Functionality request: ADMIN checking data before > GBIFregistration > > Hi all, > > Hannu has raised a request for the following to be satisfied by the > IPT: > "- Publishing a resource must be accepted by the owner of the > provider. It has happened that a test user publishes something odd > which goes all the way to the data portal without nobody controlling > it." > > This is a contradiction to the requests of others, and specifically > those wishing to promote basic "data hosting centers", who request > that a data MANAGER should be able to work autonomously. > > After discussion with the developers the proposal is to implement the > following, which we hope satisfies both requirements: > In the Administration section, an ADMIN can choose to enable or > disable the ability for MANAGERS to register resources with GBIF. By > default MANAGERS can register a resource, but an ADMIN can disable > this through this check box. > > If anyone has any concerns or comments on this approach, please can > you raise them on this list? > > Many thanks, > Tim > > > > > > > _______________________________________________ > IPT mailing list > IPT(a)lists.gbif.org > http://lists.gbif.org/mailman/listinfo/ipt >

2 2

Fwd: Auto-discard notification
by "Markus Döring (GBIF)" 15 Sep '10

15 Sep '10

[forwarded to list] Begin forwarded message: > The attached message has been automatically discarded. > From: "Mihail-Constantin Carausu" <MCCarausu(a)snm.ku.dk> > Date: September 15, 2010 12:03:50 GMT+02:00 > To: "Tim Robertson \(GBIF\)" <trobertson(a)gbif.org>, <ipt(a)lists.gbif.org> > Subject: RE: [IPT] Functionality request: ADMIN checking data before GBIFregistration > > > Dear Tim > > I think the development team's mentioned approach is a workable solution > to cover both requirements at this stage. > However, I think Hannu (and me) had in mind a kind of "Basket of > approvals"-alike functionality in the Admin's (owner of the provider) > administration section side: When a Manager has been published a dataset > through the IPT, this will automatically trigger a request for approval > or submits an yes/no event in the Admin's administration section. The > Admin must finally active interfere and approve the dataset publication > (e.g. by checking an "Approved" check box in the basket of approvals > list with events/datasets in the administration section) at the absolute > latest stage (e.g. when GBIF just needs to start to index it, or > something like that). Without this final approval the dataset will still > be published and visible through the IPT but not visible/searchable on > the GBIF data portal. This approach is not necessarily in contradiction > with the Manager's ability to autonomously publish datasets within the > IPT, only it puts this ability always under control from the central > administration section when the dataset has to go to the GBIF data > portal. > I think both solutions/approaches have obvious advantages and > disadvantages while none of them provides a 100% protection against > publishing something odd by a (test) user. > I have a little question regarding the development team's proposed > solutions: is it not possible for the central Admin to enable the > publishing ability for some "trusted" managers and disable this for > others inside the same instance of the IPT. > > Now I saw Hannu's new message just arrived, sorry for eventually > unsynchronized double-crossing messages, but I will send this anyhow. > > Best regards, > Mihail > > ---------------------------- > Mihail Carausu > MSc.Eng., Informatics Manager > Danish Biodiversity Information Facility (DanBIF) > -------------------------------------------- > > > -----Original Message----- > From: ipt-bounces(a)lists.gbif.org [mailto:ipt-bounces@lists.gbif.org] On > Behalf Of Tim Robertson (GBIF) > Sent: 15. september 2010 09:43 > To: ipt(a)lists.gbif.org > Subject: [IPT] Functionality request: ADMIN checking data before > GBIFregistration > > Hi all, > > Hannu has raised a request for the following to be satisfied by the IPT: > "- Publishing a resource must be accepted by the owner of the > provider. It has happened that a test user publishes something odd > which goes all the way to the data portal without nobody controlling > it." > > This is a contradiction to the requests of others, and specifically > those wishing to promote basic "data hosting centers", who request > that a data MANAGER should be able to work autonomously. > > After discussion with the developers the proposal is to implement the > following, which we hope satisfies both requirements: > In the Administration section, an ADMIN can choose to enable or > disable the ability for MANAGERS to register resources with GBIF. By > default MANAGERS can register a resource, but an ADMIN can disable > this through this check box. > > If anyone has any concerns or comments on this approach, please can > you raise them on this list? > > Many thanks, > Tim > > > > > > > _______________________________________________ > IPT mailing list > IPT(a)lists.gbif.org > http://lists.gbif.org/mailman/listinfo/ipt > >

1 0

Re: [IPT] GBIF Integrated Publishing Toolkit
by "Markus Döring (GBIF)" 15 Sep '10

15 Sep '10

[forwarding to list again as the message was rejected] retrieving a full record from the network is a basic often needed feature. But there might be scenarios where people do not want to have a running server at all and prefer to simply publish their dataset to somewhere else. In that case "somewhere else" could be responsible for exposing the records. An IPT used for (national) hosting might be such a solution (given the record resolution is implemented). But one could also think to serve the full, original record by GBIF centrally for those cases. That would require GBIF to cache a full record as it came in of course, e.g. the full ABCD or a DWC with extensions potentially unknown to us. We have that on our wishlist for quite some time already. Is it correct that both solutions, IPT record resolution or central GBIF resolution, would work for the BioCASe portals? Markus On Sep 15, 2010, at 10:48, Guentsch, Anton wrote: > Hi Markus, > > I am not so familiar with the IPT but may be this as a clarification: Some of the "special interest networks" using GBIF/BioCASE technology heavily rely on the ability of provider software to return a full data record (e.g. ABCD and its extensions) for a given ID. This is in particular essential for the DNA-Bank-Network, which uses ABCD plus the DNA-extension. A second example is the emerging GBIF-D soil zoology network. Personally, I think that these networks have a great potential for demonstrating the capabilities of the GBIF infrastructures being based on high-quality data subsets and interfaces which directly serve a certain user community. > > For the IPT I think that at the very least we would need some kind of interface which makes every record addressable. Preferably, the record should be a full record in the sense that DNA-extensions etc. are somehow represented. > > Best regards, > Anton > >> -----Ursprüngliche Nachricht----- >> Von: "Markus Döring (GBIF)" [mailto:mdoering@gbif.org] >> Gesendet: Mittwoch, 15. September 2010 10:24 >> An: =?iso-8859-1?Q?=22Holetschek(a)ns.gbif.org; Holetschek, Jörg >> Cc: Tim Robertson (trobertson(a)gbif.org) ipt(a)lists.gbif.org; Guentsch, >> Anton; Berendsohn, Walter G. >> Betreff: Re: [IPT] GBIF Integrated Publishing Toolkit >> >> Jörg, >> currently it is not planned to provide single record access of any kind >> as we removed any record storage, e.g. the previous H2 database, to >> increase performance and reduce resource requirements. >> But if that is indeed needed we could add it back in of course. A >> simple link to a record in xml or rdf based on its ID would be >> sufficient? >> >> Markus >> >> >> On Sep 15, 2010, at 10:11, Holetschek, Jörg wrote: >> >>> Hi Tim, >>> >>> the removal of the TAPIR interface will cause problems for all of our >> BioCASe portals - for showing the complete record details, they do a >> DiGIR/BioCASe/TAPIR request, which would then fail. >>> >>> Will there be an alternative option to get the DwC XML document, for >> example via a deep link? >>> >>> Cheers from Berlin, >>> Jörg >>> >>> -----Original Message----- >>> From: trobertson(a)gbif.org [mailto:trobertson@gbif.org] >>> Sent: Monday, 13 September, 2010 16:11 >>> To: Berendsohn, Walter G. >>> Subject: GBIF Integrated Publishing Toolkit >>> >>> [FOR INFORMATION] >>> >>> Dear Node Managers and Data Publishers, >>> >>> The objective of this communication is to update you on the current >> situation of the GBIF Integrated Publishing Toolkit (IPT). The IPT is >> a tool providing publishing capabilities for primary biodiversity >> occurrence data, taxonomic checklists and the associated metadata for >> these resource types. >>> >>> Since its first introduction in March 2009, the IPT has been used >> with some success by a limited number of users to publish primary >> biodiversity data into the GBIF network, along with descriptive >> metadata at the dataset level. >>> >>> The GBIF Secretariat (GBIFS) has solicited and received a vast amount >> of feedback during the first year of the IPT use, and would like to >> thank all those who provided this. The feedback received >> overwhelmingly confirms that the concept of the IPT is sound. However, >> feedback has also made it clear that it is over-specified for the >> majority of user needs, is not yet sufficiently robust, is too slow in >> operation, and (due to the excessive functionality) has high server >> requirements creating a barrier for many to adopt and use it. In >> addition, the feedback has made it clear that our communications >> surrounding the status of the IPT releases have been too infrequent and >> unclear concerning which release candidate we were offering and what >> functionalities this release candidate would include or not include, as >> per user needs/expectations. >>> >>> Based upon the extensive feedback GBIFS has revised our planned >> development roadmap for the IPT and are currently performing a major >> refactoring effort to: >>> . Reduce the server requirements significantly >>> . Increase the data import performance by removing the embedded >> database >>> . Remove the dependency on heavy libraries and tools such as >> Geoserver >>> . Remove all data interfaces that are not necessary for data >> publication through GBIF >>> . Improve the robustness of the tool >>> >>> When complete, the refactored IPT will offer through an intuitive >> interface: >>> . Authoring of metadata according to the GBIF metadata profile >>> . Import and mapping of checklist and occurrence data either from >> file upload or by connection of a database >>> . Registration to and publication onto the GBIF network >>> . Importing and mapping of data to the DarwinCore, and DarwinCore >> extensions >>> . Output formats of EML (2.1.0) metadata and DarwinCore Archive >>> . Improved customisation options for the "About this IPT" >>> . Improved GBIF Registry integration >>> . Improved DarwinCore extension and vocabulary organization >>> . Improved management of the organisations to which the IPT and >> resources are related, to enable co-hosting capabilities >>> . Enhanced dataset metadata authoring >>> >>> Whilst this is a reduction in existing functionality this has been >> deemed necessary to ensure the IPT is meeting the core requirements as >> expressed by the user community. The following features will be >> removed and only reintroduced if necessary through consultation with >> the IPT community: >>> . TAPIR interface >>> . TCS output format >>> . Search and browse web interface >>> . OGC web services >>> >>> All IPT development efforts are now focused on this revised roadmap, >> and it is anticipated that early testing of the new codebase will >> commence in October 2010 with the involvement of willing Nodes. The >> release of a fully stable IPT is targeted for the end of 2010. However, >> this will only occur if the testing community is satisfied that the >> product is 'release-ready' together with the required user manual and >> technical documentation. Should that not be the case, as it may take >> more time to implement and test the improvements outlined above, you >> will be informed as soon as possible. >>> >>> Further developments of the IPT beyond this release will only be >> implemented after a further scoping with the GBIF community. An area >> for further IPT discussion will be set up on the GBIF community website >> (http://community.gbif.org) >>> >>> If you have any questions or would like to know more about the GBIF >> IPT and/or revised process, feel free to contact: >>> >>> Tim Robertson >>> GBIF Information Systems Architect >>> trobertson(a)gbif.org >>> >>> With best regards, >>> >>> Tim Robertson >>> GBIF Secretariat >>> >>> >>> >>> _______________________________________________ >>> IPT mailing list >>> IPT(a)lists.gbif.org >>> http://lists.gbif.org/mailman/listinfo/ipt >>> > >

2 1

Re: [IPT] GBIF Integrated Publishing Toolkit
by "Holetschek, Jörg" 15 Sep '10

15 Sep '10

Hi Tim, the removal of the TAPIR interface will cause problems for all of our BioCASe portals - for showing the complete record details, they do a DiGIR/BioCASe/TAPIR request, which would then fail. Will there be an alternative option to get the DwC XML document, for example via a deep link? Cheers from Berlin, Jörg -----Original Message----- From: trobertson(a)gbif.org [mailto:trobertson@gbif.org] Sent: Monday, 13 September, 2010 16:11 To: Berendsohn, Walter G. Subject: GBIF Integrated Publishing Toolkit [FOR INFORMATION] Dear Node Managers and Data Publishers, The objective of this communication is to update you on the current situation of the GBIF Integrated Publishing Toolkit (IPT). The IPT is a tool providing publishing capabilities for primary biodiversity occurrence data, taxonomic checklists and the associated metadata for these resource types. Since its first introduction in March 2009, the IPT has been used with some success by a limited number of users to publish primary biodiversity data into the GBIF network, along with descriptive metadata at the dataset level. The GBIF Secretariat (GBIFS) has solicited and received a vast amount of feedback during the first year of the IPT use, and would like to thank all those who provided this. The feedback received overwhelmingly confirms that the concept of the IPT is sound. However, feedback has also made it clear that it is over-specified for the majority of user needs, is not yet sufficiently robust, is too slow in operation, and (due to the excessive functionality) has high server requirements creating a barrier for many to adopt and use it. In addition, the feedback has made it clear that our communications surrounding the status of the IPT releases have been too infrequent and unclear concerning which release candidate we were offering and what functionalities this release candidate would include or not include, as per user needs/expectations. Based upon the extensive feedback GBIFS has revised our planned development roadmap for the IPT and are currently performing a major refactoring effort to: . Reduce the server requirements significantly . Increase the data import performance by removing the embedded database . Remove the dependency on heavy libraries and tools such as Geoserver . Remove all data interfaces that are not necessary for data publication through GBIF . Improve the robustness of the tool When complete, the refactored IPT will offer through an intuitive interface: . Authoring of metadata according to the GBIF metadata profile . Import and mapping of checklist and occurrence data either from file upload or by connection of a database . Registration to and publication onto the GBIF network . Importing and mapping of data to the DarwinCore, and DarwinCore extensions . Output formats of EML (2.1.0) metadata and DarwinCore Archive . Improved customisation options for the "About this IPT" . Improved GBIF Registry integration . Improved DarwinCore extension and vocabulary organization . Improved management of the organisations to which the IPT and resources are related, to enable co-hosting capabilities . Enhanced dataset metadata authoring Whilst this is a reduction in existing functionality this has been deemed necessary to ensure the IPT is meeting the core requirements as expressed by the user community. The following features will be removed and only reintroduced if necessary through consultation with the IPT community: . TAPIR interface . TCS output format . Search and browse web interface . OGC web services All IPT development efforts are now focused on this revised roadmap, and it is anticipated that early testing of the new codebase will commence in October 2010 with the involvement of willing Nodes. The release of a fully stable IPT is targeted for the end of 2010. However, this will only occur if the testing community is satisfied that the product is 'release-ready' together with the required user manual and technical documentation. Should that not be the case, as it may take more time to implement and test the improvements outlined above, you will be informed as soon as possible. Further developments of the IPT beyond this release will only be implemented after a further scoping with the GBIF community. An area for further IPT discussion will be set up on the GBIF community website (http://community.gbif.org) If you have any questions or would like to know more about the GBIF IPT and/or revised process, feel free to contact: Tim Robertson GBIF Information Systems Architect trobertson(a)gbif.org With best regards, Tim Robertson GBIF Secretariat

2 1

IPT Roadmap
by Tim Robertson (GBIF) 14 Sep '10

14 Sep '10

Dear all, Please accept my apologies for cross posting this, but in case you were not on the previous mailing list, I would like to draw your attention to the communication sent below. We will shortly be revising all the issues in the code site to reflect the status of the revised version. An IPT user group is being set up today on http://community.gbif.org to discuss the IPT, in addition to this mailing list. Please do contact me for any clarifications you need. Many thanks, Tim [FOR INFORMATION] Dear Node Managers and Data Publishers, The objective of this communication is to update you on the current situation of the GBIF Integrated Publishing Toolkit (IPT). The IPT is a tool providing publishing capabilities for primary biodiversity occurrence data, taxonomic checklists and the associated metadata for these resource types. Since its first introduction in March 2009, the IPT has been used with some success by a limited number of users to publish primary biodiversity data into the GBIF network, along with descriptive metadata at the dataset level. The GBIF Secretariat (GBIFS) has solicited and received a vast amount of feedback during the first year of the IPT use, and would like to thank all those who provided this. The feedback received overwhelmingly confirms that the concept of the IPT is sound. However, feedback has also made it clear that it is over-specified for the majority of user needs, is not yet sufficiently robust, is too slow in operation, and (due to the excessive functionality) has high server requirements creating a barrier for many to adopt and use it. In addition, the feedback has made it clear that our communications surrounding the status of the IPT releases have been too infrequent and unclear concerning which release candidate we were offering and what functionalities this release candidate would include or not include, as per user needs/expectations. Based upon the extensive feedback GBIFS has revised our planned development roadmap for the IPT and are currently performing a major refactoring effort to: * Reduce the server requirements significantly * Increase the data import performance by removing the embedded database * Remove the dependency on heavy libraries and tools such as Geoserver * Remove all data interfaces that are not necessary for data publication through GBIF * Improve the robustness of the tool When complete, the refactored IPT will offer through an intuitive interface: * Authoring of metadata according to the GBIF metadata profile * Import and mapping of checklist and occurrence data either from file upload or by connection of a database * Registration to and publication onto the GBIF network * Importing and mapping of data to the DarwinCore, and DarwinCore extensions * Output formats of EML (2.1.0) metadata and DarwinCore Archive * Improved customisation options for the "About this IPT" * Improved GBIF Registry integration * Improved DarwinCore extension and vocabulary organization * Improved management of the organisations to which the IPT and resources are related, to enable co-hosting capabilities * Enhanced dataset metadata authoring Whilst this is a reduction in existing functionality this has been deemed necessary to ensure the IPT is meeting the core requirements as expressed by the user community. The following features will be removed and only reintroduced if necessary through consultation with the IPT community: * TAPIR interface * TCS output format * Search and browse web interface * OGC web services All IPT development efforts are now focused on this revised roadmap, and it is anticipated that early testing of the new codebase will commence in October 2010 with the involvement of willing Nodes. The release of a fully stable IPT is targeted for the end of 2010. However, this will only occur if the testing community is satisfied that the product is 'release-ready' together with the required user manual and technical documentation. Should that not be the case, as it may take more time to implement and test the improvements outlined above, you will be informed as soon as possible. Further developments of the IPT beyond this release will only be implemented after a further scoping with the GBIF community. An area for further IPT discussion will be set up on the GBIF community website (http://community.gbif.org) If you have any questions or would like to know more about the GBIF IPT and/or revised process, feel free to contact: Tim Robertson GBIF Information Systems Architect trobertson(a)gbif.org With best regards, Tim Robertson GBIF Secretariat

1 0

Mapping/publishing problem?
by Jens Rathsman 27 Aug '10

27 Aug '10

Dear GBIF helpdesk, I am publishing some datasets on an IPT server (1.0-RC3-SNAPSHOT) for the Swedish GBIF node. There are some problems with the mapping or publishing the following fields: VerbatimCollectingDate --- EarliestDateCollected --- StartTimeOfDay --- YearSampled --- MonthOfYear --- DayOfMonth --- Collector --- For example catalogNumber 38922 ( puiblished at: http://gbif.se:8080/ipt/occDetail.html?guid=e4ec28b2-128a-49ee-b299-00ced41…) has values for all of the above fields in the datasource (a MySql database) but none of them are presenteded by IPT. When I click "Annotations" for the resource there are no problems shown, and I get no clues from the tomcat log file (catalina.out). This is the data from the data source: catalogNumber collector earliestDateCollected verbatimCollectingDate startDayOfYear endDayOfYear eventTime yearSampled monthOfYear dayOfYear 38922 Åhlander, E et al. 1990-04-04 4 and 12 Apr 1990 94 102 10:00:00 1990 4 4 What could be causing this problem? Best regards, --------------------------------- Jens Rathsman, System developper GBIF Sweden - http://www.gbif.se<http://www.gbif.se/> mailto:jens.rathsman@nrm.se tel: +46 8 5195 5116 Swedish Museum of Natural History - http://www.nrm.se<http://www.nrm.se/> Box 50007 SE-104 05 Stockholm Sweden

3 5