Hi Markus

I am considering a local installation. Can you help me with instructions to get it done (today)?

Kind regards, Mihail

---------------------------- Mihail Carausu MSc.Eng., Informatics Manager Danish Biodiversity Information Facility (DanBIF) c/o Zoological Museum University of Copenhagen Universitetsparken 15 DK-2100 Copenhagen Ø

Tel.: (+45) 3532 1063 Mobil:(+45) 2618 3636 email:mccarausu@zmuc.ku.dk web: http://www.DanBIF.dk ....oooO.............. ....(....)....Oooo.... .....)../. ...(....)... .....(_/.......)../...... .............. (_/....... --------------------------------------------

-----Original Message----- From: "Markus Döring (GBIF)" [mailto:mdoering@gbif.org] Sent: 27. februar 2009 10:36 To: Tim Robertson (GBIF) Cc: Mihail-Constantin Carausu; ipt@lists.gbif.org Subject: Re: [IPT] Beta update

it would also be nice if someone could test the SQL based upload, which requires a local installation. If there is help needed with the installation, just post to this list please.

Markus

On Feb 26, 2009, at 8:59 PM, Tim Robertson (GBIF) wrote:

Thank you very much.

I suggest starting with 0.5million, and then 1 million and then 3 million. I will be surprised if this works without some bug fixing.

You may just use http://ipt.gbif.org rather than installation.

Thanks Mihail, help with tests like this is greatly appreciated.

Tim

...

Dear Tim

I can volunteer for uploading up to 3 million records if you want. The only problem is I have not installed yet the IPT software on our servers (sorry) because I didn't get the time, yet. But I will definitely try to do the installation tomorrow and the upload too.

Best wishes Mihail

---

Mihail Carausu MSc.Eng., Informatics Manager Danish Biodiversity Information Facility (DanBIF) c/o Zoological Museum University of Copenhagen Universitetsparken 15 DK-2100 Copenhagen Ø

Tel.: (+45) 3532 1063 Mobil:(+45) 2618 3636 email:mccarausu@zmuc.ku.dk web: http://www.DanBIF.dk ....oooO.............. ....(....)....Oooo.... .....)../. ...(....)... .....(_/.......)../...... .............. (_/.......

---

-----Original Message----- From: ipt-bounces@lists.gbif.org [mailto:ipt- bounces@lists.gbif.org] On Behalf Of Tim Robertson Sent: 26. februar 2009 16:26 To: ipt@lists.gbif.org Subject: [IPT] Beta update

Dear All,

A new version has just been deployed, and all user accounts have been kept intact, but you will notice significant changes, and we hope improvements. Please use the http://code.google.com/p/gbif-providertoolkit/issues/ list to track and input issues.

If there is a volunteer wishing to test an upload of a large dataset (500,000+ records) it would be very much appreciated if you could report back on the experience.

Thanks for all your feedback,

Best wishes,

Tim

---

IPT mailing list IPT@lists.gbif.org http://lists.gbif.org/mailman/listinfo/ipt

---

Tim Robertson Systems Architect Global Biodiversity Information Facility Secretariat (GBIF) Universitetsparken 15, 2100 Copenhagen Ø, Denmark http://www.gbif.org trobertson@gbif.org Phone: +45 35 32 14 87 (Office) Fax: +45 35 32 14 80 ____________________________________________________________

---

IPT mailing list IPT@lists.gbif.org http://lists.gbif.org/mailman/listinfo/ipt

Hi Markus,

Saving the SQL settings from the IPT GUI, I only get the Yikes error message when trying to (both ipt-1.0b.3 and ipt.1.0b.5). But the settings seems to be saved ok when I revisit the Source data settings! Next I may successfully use the Resource Configuration to give the SQL statement and the property mappings. In the property mappings I can see genuine data from my database with the "view source" button. Next step is Cache - and does not work. I get relatively fast to the message "Source ready to be used.", but no data is actually to be seen from Explore - and the Validate button is still gray.

Automapping works for uploaded tab-delimited dataset, but not for the SQL statement (using SELECT column AS dwc_term).

I have tried with both ipt-1.ob.3 and ipt-1.0b.5 (built with Maven from the SVN version at Google Code). My system is: a MacBook OS X version 10.5.6 (Leopard), Tomcat version 6.0.18 (and version 5.5.27 briefly tested), JVM version 1.5.0, Apache 2.2.11 with mod_jk version 1.2.27 (briefly tested, works fine).

When adding dataset from tab-delimited file, I leave empty values empty (just going ahead with the next tab (double tab). It seems to me like the IPT treats these as if one tab shifting the coulmns to the left...?

I also have severe problems with memory! My poor MacBook have only 2 GB and the IPT very often flips very very slow and takes a lot of time before it comes back to responding again. I do not at all manage to load the Geoserver application due to memory. I know I have too little memory and have been stretching the few 2 GB the MacBook came with too long already.

I often see the OutOfMemoryError in a Yikes page.

Caused by: java.lang.OutOfMemoryError: Java heap space Cheers Dag Terje

PS. Great application - if oly it could be a bit less memory hungry...?

_____

From: "Markus Döring (GBIF)" [mailto:mdoering@gbif.org] To: Tim Robertson (GBIF) [mailto:trobertson@gbif.org] Cc: Mihail-Constantin Carausu [mailto:MCCarausu@snm.ku.dk], ipt@lists.gbif.org Sent: Fri, 27 Feb 2009 10:36:11 +0100 Subject: Re: [IPT] Beta update

it would also be nice if someone could test the SQL based upload, which requires a local installation. If there is help needed with the installation, just post to this list please.

Markus

Hi Markus,

I already tried your suggestion to add the JAVA_OPTS setting according to your previous suggestion to Peter Desmet on February 5. I also tried to add this JAVA_OPTS to the setenv.sh to my tomcat bin folder as suggested by Tim.

Digging a bit deeper I found that my Tomcat configured to start at boot time from a plist instruction (following http://www.malisphoto.com/tips/tomcatonosx.html). This plist instruction does not start Tomcat from the catalina.sh script, nor does this startup include the setenv.sh script (the plist script was using Tomcat5.sh). I tried to start Tomcat from catalina.sh instead and now both options of adding the JAVA_OPTS directly in the catalina.sh or in the setenv.sh scripts works fine. Another option I found with Google is to set JAVA_OPTS (and CATALINA_OPTS) in the environment (eg. in .bask_profile).

I was getting a maximum memory for tomcat of only 63.56 MB, now I get the 1 GB (1016.12 MB) as you suggests - and both the IPT and the Geoserver now starts and persists very fine! I may still sometimes get the Yikes memory error when I also have the PostgreSQL database running, but everything generally flows excellent now. Thanks!!

Trying the SQL data source Upload again I still get the same problem. "Upload in Progress - Source ready to be used", but no records indexed. I have limited the results to 165 records and I have verified the SQL statement to be valid from the PostgreSQL database promt. But no records gets cached (not visible from IPT menu Explore). I use the very same SQL statement to generate the tab delimited text file as dataset source for the IPT. When using the text file import, automapping works fine (I am using lowercase DarwinCore terms as descriptors), but when using the SQL data source, there is no automapping...?

Looking in the PostgreSQL log I find: LOG: unexpected EOF on client connection

Thanks!! Dag Terje

_____

From: "Markus Döring (GBIF)" [mailto:mdoering@gbif.org] To: Dag Terje Filip Endresen [mailto:dag.endresen@nordgen.org] Cc: ipt@lists.gbif.org Sent: Fri, 27 Feb 2009 20:44:13 +0100 Subject: Re: [IPT] Beta update

Dag, thanks for those reports. The SQL import has been neglected for a while, I will try my best next week. Regarding memory your macbook with 2gb should be more than fine. All tomcats I have used so far were configured to use no more than 1gb. Did you increase tomcats default memory size? It is very little by default, so increasing to 1gb is sometimes needed. If you run jetty via maven you can do the following to allow java to use up to 1gb of memory:

export MAVEN_OPTS='-Xmx1024m'

for tomcat edit the catalina.sh and add:

JAVA_OPTS="-Xms512M -Xmx1024M"

I have just finished implementing file upload for zip archives. So for those of you using the subversion trunk version locally you should be able to upload 100MB of zipped data. And thanks to Javi there are also much nicer maps now instead of openlayers... more soon.

Markus

On Feb 27, 2009, at 1:33 PM, Dag Terje Filip Endresen wrote:

Hi Markus,

Saving the SQL settings from the IPT GUI, I only get the Yikes error message when trying to (both ipt-1.0b.3 and ipt.1.0b.5). But the settings seems to be saved ok when I revisit the Source data settings! Next I may successfully use the Resource Configuration to give the SQL statement and the property mappings. In the property mappings I can see genuine data from my database with the "view source" button. Next step is Cache - and does not work. I get relatively fast to the message "Source ready to be used.", but no data is actually to be seen from Explore - and the Validate button is still gray.

Automapping works for uploaded tab-delimited dataset, but not for the SQL statement (using SELECT column AS dwc_term).

I have tried with both ipt-1.ob.3 and ipt-1.0b.5 (built with Maven from the SVN version at Google Code). My system is: a MacBook OS X version 10.5.6 (Leopard), Tomcat version 6.0.18 (and version 5.5.27 briefly tested), JVM version 1.5.0, Apache 2.2.11 with mod_jk version 1.2.27 (briefly tested, works fine).

When adding dataset from tab-delimited file, I leave empty values empty (just going ahead with the next tab (double tab). It seems to me like the IPT treats these as if one tab shifting the coulmns to the left...?

I also have severe problems with memory! My poor MacBook have only 2 GB and the IPT very often flips very very slow and takes a lot of time before it comes back to responding again. I do not at all manage to load the Geoserver application due to memory. I know I have too little memory and have been stretching the few 2 GB the MacBook came with too long already.

I often see the OutOfMemoryError in a Yikes page.

Caused by: java.lang.OutOfMemoryError: Java heap space Cheers Dag Terje

PS. Great application - if oly it could be a bit less memory hungry...?

_____

From: "Markus Döring (GBIF)" [mailto:mdoering@gbif.org] To: Tim Robertson (GBIF) [mailto:trobertson@gbif.org] Cc: Mihail-Constantin Carausu [mailto:MCCarausu@snm.ku.dk], ipt@lists.gbif.org Sent: Fri, 27 Feb 2009 10:36:11 +0100 Subject: Re: [IPT] Beta update

it would also be nice if someone could test the SQL based upload, which requires a local installation. If there is help needed with the installation, just post to this list please.

Markus