April 2011 - IPT - lists.gbif.org

How to register external usesr for locally installed IPT nodes
by info＠pensoft.net 26 Apr '11

26 Apr '11

Dear Tim, In our Data Publishing Guidelines we intend to point out http://ipt.pensoft.net as a venue for our authors to upload their datasets. Will there be a possibility to register/login in each locally installed IPT individually? Or, the user should register/login to the GBIF IPT and then use that account? The background of my question is: what kind of instruction in this respect should we address to our authors who want to publish data through IPT? Best regards, Lyubo -- Dr Lyubomir Penev Managing Director Pensoft Publishers 13a Geo Milev Street 1111 Sofia, Bulgaria Tel +359-2-8704281 Fax +359-2-8704282 www.pensoft.net www.pensoft.net/journals/zookeys www.pensoft.net/journals/phytokeys info(a)pensoft.net

2 2

Security Exposures
by Bruce Wilson 20 Apr '11

20 Apr '11

I have installed the IPT on a Windows 2003 server (at http://science.uvu.edu:8080/ipt/). Recently, a security evaluation was made, and here are the medium-severity weaknesses found, all of which I think are attributable to Tomcat/Apache. What should I be doing to resolve these? I typically use automatic updates to keep things current, and don't normally install software that requires hands-on maintenance, so I'm unsure if an update of the Apache or Tomcat software might break the ITP app. Or even if an update will fix the holes. I think the security certificate errors are Tomcat also, because I didn't install any in Windows, but I'm not certain. # PLUGIN NAME 2 SSL Certificate signed with an unknown Certificate Authority 2 SSL Certificate with Wrong Hostname 2 SSL Certificate Expiry 1 Apache Tomcat < 6.0.32 / 7.0.8 NIO Connector Denial of Service 1 Apache Tomcat 6.0.x < 6.0.30 Multiple Vulnerabilities 1 Apache Tomcat 6.x < 6.0.30 / 7.x < 7.0.5 Multiple XSS Bruce Wilson Department of Chemistry | Mail Stop 179 | UVU | 800 W University Parkway, Orem UT 84058 (801)863-7138 | bruce.wilson(a)uvu.edu | http://science.uvu.edu/wilson

3 2

permission settings for IPT data directory
by Burke Chih-Jen Ko (GBIF) 20 Apr '11

20 Apr '11

Dear IPT list members, For people who sets up Tomcat & IPT on UNIX servers (Linux, BSDs), when there is something not working, and it seems you've followed the manual properly, sometimes it could be the permission that plagues. But, what permission settings are right for IPT data directory? It depends on the user that is designated to run Tomcat. In your shell, type $ ps waux | grep tomcat should reveal who is running Tomcat. So assuming the user is "tomcatuser," and user "tomcatuser" belongs to group "tomcatgroup," and the ownership of the IPT data directory (and it's child folders and files) should be tomcatuser:tomcatgroup Then, with mod 0775 of the IPT data directory, both of the tomcatuser and tomcatgroup have write permission, some issues about IPT settings or using the existing IPT directory will be resolved. As to how the system assigns which user to run Tomcat, it is actually depended on whether Tomcat is set up as a daemon and used a non-root user, or, it's actually you firing up the ./startup.sh so Tomcat is running as the user you used to log in. Either way the IPT data directory should be owned by the user which runs Tomcat, so settings and modifications can be saved. Hope this helps people who have troubles getting IPT behaves properly. Cheers, Burke Informatics Liaison GBIF Secretariat Tel: +45 35 32 15 11 Fax: +45 35 32 14 80 E-mail: bko(a)gbif.org Skype: burkeker http://community.gbif.org/pg/profile/burkeker

1 0

Can you get Metadata For resources that are alredy being served live with Tapir.
by Khwezi Flatela 14 Apr '11

14 Apr '11

HI I Would like to find out if I can get my metadata information on a tapir instance that we have running to feed into the IPT instance and also include the resources listed on that tapir instance. Thank you Khwezi Flatela South African Biodiversity Information Facility (SABIF) SABIF Developer Biodiversity Information Management (BIM) South African National Biodiversity Institute (SANBI) ph: +27 21 799 8699 fax: +27 86 520 6990 cell: +27 82 224 5905 http://www.sanbi.org http://www.sabif.ac.za k.flatela(a)sanbi.org.za<mailto:flatela@sanbi.org> ________________________________ VISIT OUR NEW WEBSITE ----BIODIVERSITY ADVISOR---- ON HTTP://BIODIVERSITYADVISOR.SANBI.ORG/ Think before you print. Please consider the environment before printing this email. NOTE: This e-mail message and any attachments are intended for the addressee only, and contain confidential information that may be legally privileged and/or the subject of copyright that is protected by law. Any unauthorised usage, disclosure, alteration or dissemination is prohibited. SANBI accepts no responsibility for loss, data corruption or mail that fails to reach its intended destination. Furthermore, SANBI cannot assure the integrity of this communication nor guarantee that it is free of errors, viruses, interception or interference. No liability, whether direct or indirect, is accepted by SANBI or the sender. Any view or opinion expressed in this message may not necessarily be that of SANBI or SANBI Management. SANBI reserves the right to monitor all e-mail communication. The disclaimer is located at http://www.sanbi.org/index.php?option=com_content&view=article&id=916:-sanb…

2 1

IPT page title
by Deependra Tandukar 13 Apr '11

13 Apr '11

Hello, I have configured IPT in Linux and running. I want to customize the page and also change the title of the page - right now it is IPT, how do I do this? Best regards, Deependra

2 1

Re: [IPT] IPT feedback notes
by Tim Robertson (GBIF) 12 Apr '11

12 Apr '11

Dear David, Thank you very much for this thorough feedback, which will be logged as issues and addressed to improve the IPT. We are in a testing phase for a 2.0.2 release which has minor enhancements, so I would anticipate the majority of these being fixed for the 2.0.3 release. It is reassuring to hear that you feel the IPT was easy to use and you don't consider these serious issues. Thanks again, Tim On Apr 12, 2011, at 8:52 AM, David Remsen (GBIF) wrote: > Guys - I received this message from one of the checklist award > recipients. Fairly detailed set of IPT reviews. Some of these sure > worthy of new issues. > > DR > > Begin forwarded message: > >> From: David Eades <dceades(a)illinois.edu> >> Date: April 11, 2011 9:21:38 PM GMT+02:00 >> To: "'David Remsen $GBIF$'" <dremsen(a)gbif.org> >> Subject: IPT feedback notes >> Reply-To: <dceades(a)illinois.edu> >> >> Dear David, >> >> This message is sent in response to the request for feedback about >> our use >> of IPT. >> >> Overall, the IPT 2.0.1-r3048 is an easy to use and intuitive >> interface, it >> does a good job at processing our tarball datasets and comes with a >> handy >> interface to fill out the EML metadata. The installation was done >> independently by two persons in two countries. The following is a >> list of >> the issues mentioned. None of the comments indicate serious >> problems with a >> good product. This information is about three weeks old. Some >> issues may >> have already been fixed. >> >> 1. Updating resources from archives: We couldn't find a way in >> which an >> updated archive dataset can be used to replace the existing resource >> data. The only available method using the IPT web interfaces seems to >> be by uploading updated versions of the TXT files one by one (using >> exactly the same file names that were used when creating the resource >> for the first time), and then press "publish". One alternative is >> deleting the resource and re-create it from the updated archive, but >> this option isn't convenient for registered resources. The other >> alternative is to just replace the TXT files by writing to the file >> system directly ([data_directory]/resources/[resource_name]/sources/ >> *.txt) >> and then >> press "publish", but is this a supported method? >> >> 2. Unstable "main" administrator: When the IPT is set up for the >> first >> time, there is an administrator user created and it is the one who >> appears in messages like "If you don't have an account yet, please >> ask >> your IPT administrator to create one for you.: admin_name >> <mail_address>". However, if later a new user is added with (or an >> existing user promoted to) an administration role and its mail >> address >> alphabetically precedes the current "main" administrator, then this >> new administrator starts to appear in the aforementioned messages >> instead of the original one. >> >> 3. Failed processing archives containing some empty TXT files: When >> an >> archive is uploaded at the resource creation stage, the IPT >> immediately >> begins processing it. However, as soon as an empty file is found, the >> process stops without finish processing the remaining TXT files (but >> creating the resource with the files it was able to process), and >> suggesting to run the archive through the validator (which process >> such files without complaint.) The workaround for this is to have >> such files contain at least a newline character, but when the >> resource >> is published, those files are generated completely empty once more, >> and when the generated archive is submitted back to the IPT for >> resource creation, it fails again (despite its generation by the IPT >> itself.) We think such files should be accepted by the IPT, as them >> should be considered empty (0 rows) tables. The reason we have some >> of >> our datasets with empty extension files is because all of them are >> generated from equally-capable databases all maintained by the same >> software (SpeciesFile), but some of them don't have records for all >> kinds of information yet (like common names for instance), however in >> future updates such records may begin to appear. >> >> 4. Sometimes the file sizes are incorrectly reported: When managing a >> resource ( /manage/resource.do?r=resource_name ), some file sizes are >> incorrectly reported. Example: "vernacular [file] 0 bytes, 1 rows, 3 >> columns." The file contains this one line only, terminated by a >> newline character: "1";"webspinners";"English". This doesn't seem to >> be really a problem, but we make the comment just in case it's >> actually more than a "cosmetic" issue. (The published archive the IPT >> generates still contains the example line.) >> >> 5. Validator (http://tools.gbif.org/dwca-validator/ ) and IPT out of >> sync: This issue occurred when migrating from the release candidate 3 >> to the 2.0.1-r3048 version. Previously, the types and specimens >> extension was of rowtype "http://rs.gbif.org/terms/1.0/Specimen", >> however now it is no longer recognized by the IPT and >> "http://rs.gbif.org/terms/1.0/TypesAndSpecimen" must be used instead >> (which in turn is not recognized by the validator and this new >> rowtype doesn't have the identificationRemarks term.) Also, with the >> species profile extension we are using the livingPeriod term, but >> this >> one is not recognized by the validator while it is accepted by the >> IPT. >> >> 6. Explicitly set vocabularies in meta.xml are not preserved by the >> IPT: For several of our columns we are using the vocabularies the IPT >> comes with and we explicitly advertise that fact in our source >> meta.xml (for example: <field index="6" >> term="http://rs.tdwg.org/dwc/terms/taxonRank" >> vocabulary="http://rs.gbif.org/vocabulary/gbif/rank.xml"/>.) However, >> when the resource is published, the archive generated by the IPT >> removes this information. We think that perhaps the IPT should not >> remove the vocabularies from meta.xml and maybe for those >> vocabularies >> the IPT is aware of it should either complain when a row violates the >> vocabulary or else set the column value to NULL (like the automap >> option >> of the value translation page does?) >> >> 7. When the id and a term in the core file are set to the same column >> index the IPT generates a separate column with a duplicated value: >> Our >> meta.xml defines the id and taxonID in the same way as in the example >> at http://rs.tdwg.org/dwc/terms/guides/text/index.htm#implement (id >> and taxonID both mapped to column 0.) However, when we publish the >> resource, the IPT keeps the id at column 0, but also creates a new >> column for taxonID containing the same value column 0 has. This is >> not >> much of a problem, but in doing so it takes more space than >> necessary. >> >> 8. The documentation is heavily biased toward Linux >> (IPTServerPreparation). >> >> No support of Windows Server environments was apparent. We elected >> to >> implement >> a standalone Linux server, rather than try to use Microsoft >> interoperability >> tools >> under Windows Server. >> >> It seems that the presumption is that the installer knows the >> names and locations of the relevant files, which are often not >> explicitly >> stated. >> >> No step-by-step instruction is available. The section related >> to Tomcat comprises about 9 sentences. The time it required to >> research and implement the actual steps was several hours, shortened >> by liberal use of a Linux-familiar collaborator. >> >> 9. No criteria for selection of the server infrastructure are >> provided. This appears to be oriented toward making a given >> pre-existing web server installation adapt to the IPT. Since in our >> case, we created a server from the ground up, we selected an >> Ubuntu/Tomcat approach for no better reason than we had some minimal >> familiarity with them, and access to greater expertise. >> >> 10. Maintenance/version upgrade process (Starting Over) is >> similarly sparsely detailed, and described in general terms. Since >> in-house expertise is oriented toward WS2008 and IIS 7, the >> management >> and maintenance of this web application in the selected environment >> is >> by no means obvious. The initial setup eventually worked, but the >> structure of the web server and relationships among the programs and >> files was essentially mysterious, without reference material like: >> how >> web applications are managed under Tomcat. >> >> 11. Pictorial information is limited to post-installation data >> management user operations. Installation-related graphics would have >> been useful. A disclaimer emphasizing the value of Linux expertise >> would prevent the mistaken assumption that this is a cookbook >> installation. This section of the documentation appears to be where >> the effort was concentrated, but this was not the area of my activity >> (setting up the server and site). >> >> Hope this information is useful. >> >> >

1 0

structured text files
by Wolfgang Lorenz 11 Apr '11

11 Apr '11

Dear All, what everyone can do easily (no matter how data are originally entered in excel or whatsoever format) is export datasets as simple structured text files. Then we just need a central server with IPT installed (e.g., at country nodes?) where the 'little data holders' can submit text files for publication via GBIF. Are there plans to offer such things? Just thinking of those who cannot afford or don't know (like myself) how to set up the IPtoolkit on a server. Cheers, Wolfgang -------------------------------- Wolfgang Lorenz, Tutzing, Germany

2 1

can IPT2 handle big datasets?
by Kyle Braak (GBIF) 10 Apr '11

10 Apr '11

Dear IPT mailing list, This afternoon we conducted a little test to see whether the IPT2 can handle publishing a big dataset from a database. In the test we used a MySQL database, and successfully generated an archive with 24.000.000 records in about 50 minutes! This was run on a Tomcat server with 256MB memory. http://ipt.gbif.org/resource.do?r=bigdbtest Previously IPT1 had serious problems with such large datasets, but during IPT2 development special care was taken to be ensure that they could be handled gracefully. The way it is done now, is that the result sets from the database are streamed to the file system where they are written (about a 1000 records per result set) so there is no memory burden at all. This is one of the reasons why the IPT2 is not as feature rich as the IPT1 was. Best wishes, Kyle Braak Programmer Global Biodiversity Information Facility Secretariat Universitetsparken 15, DK-2100 Copenhagen, Denmark Tel: +45-35321479 Fax: +45-35321480 http://community.gbif.org/pg/profile/kbraak URL: http://www.gbif.org

4 3