[IPT] Beta update

Sat Feb 28 12:30:31 CET 2009

Hi Markus,

I already tried your suggestion to  add the JAVA_OPTS setting according to your previous suggestion to  Peter Desmet on February 5. I also tried to add this JAVA_OPTS to the  setenv.sh to my tomcat bin folder as suggested by Tim.

Digging a bit deeper I found that my Tomcat configured to start at boot time from a plist instruction (following http://www.malisphoto.com/tips/tomcatonosx.html).  This plist instruction does not start Tomcat from the catalina.sh  script, nor does this startup include the setenv.sh script (the plist  script was using Tomcat5.sh). I tried to start Tomcat from catalina.sh  instead and now both options of adding the JAVA_OPTS directly in the  catalina.sh or in the setenv.sh scripts works fine. Another option I  found with Google is to set JAVA_OPTS (and CATALINA_OPTS) in the  environment (eg. in .bask_profile).

I was getting a maximum  memory for tomcat of only 63.56 MB, now I get the 1 GB (1016.12 MB) as  you suggests - and both the IPT and the Geoserver now starts and  persists very fine! I may still sometimes get the Yikes memory error  when I also have the PostgreSQL database running, but everything  generally flows excellent now. Thanks!!

Trying the SQL data  source Upload again I still get the same problem. "Upload in Progress -  Source ready to be used", but no records indexed. I have limited the  results to 165 records and I have verified the SQL statement to be  valid from the PostgreSQL database promt. But no records gets cached  (not visible from IPT menu Explore). I use the very same SQL statement  to generate the tab delimited text file as dataset source for the IPT.  When using the text file import, automapping works fine (I am using  lowercase DarwinCore terms as descriptors), but when using the SQL data  source, there is no automapping...?

Looking in the PostgreSQL log I find:
   LOG:  unexpected EOF on client connection

Thanks!!
Dag Terje

  _____  

From: "Markus Döring (GBIF)" [mailto:mdoering at gbif.org]
To: Dag Terje Filip Endresen [mailto:dag.endresen at nordgen.org]
Cc: ipt at lists.gbif.org
Sent: Fri, 27 Feb 2009 20:44:13 +0100
Subject: Re: [IPT] Beta update

Dag,
thanks for those reports. The SQL import has been neglected for a while, I will try my best next week.
Regarding memory your macbook with 2gb should be more than fine. All tomcats I have used so far were configured to use no more than 1gb. Did you increase tomcats default memory size? It is very little by default, so increasing to 1gb is sometimes needed. If you run jetty via maven you can do the following to allow java to use up to 1gb of memory:

export MAVEN_OPTS='-Xmx1024m'

for tomcat edit the catalina.sh and add:

JAVA_OPTS="-Xms512M -Xmx1024M"

I have just finished implementing file upload for zip archives. So for those of you using the subversion trunk version locally you should be able to upload 100MB of zipped data.
And thanks to Javi there are also much nicer maps now instead of openlayers... more soon.

Markus

On Feb 27, 2009, at 1:33 PM, Dag Terje Filip Endresen wrote:

Hi Markus,

Saving the SQL settings from the IPT GUI, I only get the Yikes error message when trying to (both ipt-1.0b.3 and ipt.1.0b.5). But the settings seems to be saved ok when I revisit the Source data settings! Next I may successfully use the Resource Configuration to give the SQL statement and the property mappings. In the property mappings I can see genuine data from my database with the "view source" button. Next step is Cache - and does not work. I get relatively fast to the message "Source ready to be used.", but no data is actually to be seen from Explore - and the Validate button is still gray.

Automapping works for uploaded tab-delimited dataset, but not for the SQL statement (using SELECT column AS dwc_term).

I have tried with both ipt-1.ob.3 and ipt-1.0b.5 (built with Maven from the SVN version at Google Code).
My system is: a MacBook OS X version 10.5.6 (Leopard), Tomcat version 6.0.18 (and version 5.5.27 briefly tested), JVM version 1.5.0, Apache 2.2.11 with mod_jk version 1.2.27 (briefly tested, works fine).

When adding dataset from tab-delimited file, I leave empty values empty (just going ahead with the next tab (double tab). It seems to me like the IPT treats these as if one tab shifting the coulmns to the left...?

I also have severe problems with memory! My poor MacBook have only 2 GB and the IPT very often flips very very slow and takes a lot of time before it comes back to responding again. I do not at all manage to load the Geoserver application due to memory. I know I have too little memory and have been stretching the few 2 GB the MacBook came with too long already.

I often see the OutOfMemoryError in a Yikes page.

Caused by: java.lang.OutOfMemoryError: Java heap space
Cheers
Dag Terje

PS. Great application - if oly it could be a bit less memory hungry...?

  _____  

From: "Markus Döring (GBIF)" [mailto:mdoering at gbif.org]
To: Tim Robertson (GBIF) [mailto:trobertson at gbif.org]
Cc: Mihail-Constantin Carausu [mailto:MCCarausu at snm.ku.dk], ipt at lists.gbif.org
Sent: Fri, 27 Feb 2009 10:36:11 +0100
Subject: Re: [IPT] Beta update

it would also be nice if someone could test the SQL based upload, 
which requires a local installation.
If there is help needed with the installation, just post to this list 
please.

Markus

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.gbif.org/pipermail/ipt/attachments/20090228/654c5076/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: IPT 1.0b.5-SQL Source-Upload in Progress-Done.jpg
Type: image/jpeg
Size: 32239 bytes
Desc: not available
Url : http://lists.gbif.org/pipermail/ipt/attachments/20090228/654c5076/attachment-0002.jpg 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Tomcat status 1GB.jpg
Type: image/jpeg
Size: 24862 bytes
Desc: not available
Url : http://lists.gbif.org/pipermail/ipt/attachments/20090228/654c5076/attachment-0003.jpg 