August 2011 - IPT - lists.gbif.org

Re: [IPT] GBIF Case 1773: UTF8
by Burke Chih-Jen Ko (GBIF) 08 Sep '11

08 Sep '11

And have you tried Tim's suggestion? Could you try issuing \s command in the mysql client shell to show it's character set settings? That would show information like this: Connection id: 37141 Current database: Current user: root@localhost SSL: Not in use Current pager: stdout Using outfile: '' Using delimiter: ; Server version: 5.1.56 Source distribution Protocol version: 10 Connection: Localhost via UNIX socket Server characterset: utf8 Db characterset: utf8 Client characterset: utf8 Conn. characterset: utf8 UNIX socket: /var/lib/mysql/mysql.sock Uptime: 41 days 22 hours 43 min 50 sec Threads: 1 Questions: 597738 Slow queries: 20 Opens: 2328 Flush tables: 1 Open tables: 64 Queries per second avg: 0.164 -------------- which contains character set settings. Cheers, Burke On Aug 12, 2011, at 2:30 PM, Mickael Graf wrote: > That's possible. But the very same data is displayed correctly with TapirLink. > > Yes, you can copy to the mailing list later. > > Cheers, > Mickaël > ________________________________________ > From: Burke Chih-Jen Ko (GBIF) [bko(a)gbif.org] > Sent: Friday, August 12, 2011 2:08 PM > To: Mickael Graf > Cc: helpdesk(a)gbif.org > Subject: Re: GBIF Case 1773: UTF8 > > Hi Mickael, > > I can see in the script the accented characters are already wrong. If you generate the script from sql client, perhaps the problem is on the DB side? > > For the script I see now I am sure the IPT won't read it correctly if the source is already "NÃ¤rke". > > Would you mind I copy the thread to the IPT mailing list later? > > Burke > > On Aug 12, 2011, at 1:51 PM, Mickael Graf wrote: > >> Hi Burke, >> >> >> I am using a view. Here come some scripts for checking the data/IPT. >> >> The statement in IPT is then simply 'select * from rcDwCIPT'. >> >> Cheers, >> Mickaël >> >> ________________________________________ >> From: Burke Chih-Jen Ko (GBIF) [bko(a)gbif.org] >> Sent: Thursday, August 11, 2011 9:24 AM >> To: Mickael Graf >> Cc: helpdesk(a)gbif.org >> Subject: GBIF Case 1773: UTF8 >> >> Hi Mickaël, >> >> Do you use SQL view or text file as the source for IPT? May I have some sample records to test and reproduce your issue? >> >> Thanks! >> >> Burke >> >> On Aug 10, 2011, at 4:25 PM, Mickael Graf wrote: >> >>> Hi, >>> >>> >>> I am testing IPT 2 and NRM RingedBirds is the guinea pig. >>> >>> Well, I have some issue with the encoding because, while swedish characters work fine with TapirLink (see http://www.gbif.se/tapir/tapir_client.php, choose NRM-RingedBirds and make an inventory over StateProvince), it's a mess with IPT, both as a preview and as a zipped file. For instance 'Närke' is displayed as 'NÃ¤rke'. This happens regardless of the character encoding chosen under /source.do. The original data is UTF8, but then I don't know if any settings in tomcat need to be changed. >>> >>> Do you have some knowledge about this issue? I am very bad at java, so I don't know where to look (and issue 418 doesn't help). >>> >>> >>> Cheers, >>> Mickaël >> >> <View_RC_DwC_IPT.sql><rc_test.sql> > >

2 13

Re: [IPT] System requirements for IPT
by Tim Robertson (GBIF) 26 Aug '11

26 Aug '11

Dear Massimo > Thank you for the quick reply! You are very welcome. > Yes there will be for the moment a single DBMS with 2 big databases > (1.2 GB and 800 MB respectively). > We would like to create a single hub for the synchronization of > several other databases in future (someone will be really huge). > Which configuration can you suggest us? I don't anticipate any special configuration is needed. Tomcat likes around 128MB of memory and you'll of course need enough disk space. > Would it be possible to create a test environment with you for > testing the sync from our server? Sure. If you get the resources mapped, we can verify that they are indexable. Cheers, Tim > Thank you, > Massimo > > > From: Tim Robertson (GBIF) [mailto:trobertson@gbif.org] > Sent: Friday, August 26, 2011 16:50 > To: Buonaiuto, Massimo (Bioversity) > Cc: ipt(a)lists.gbif.org > Subject: Re: [IPT] System requirements for IPT > > Dear Massimo > > You are likely to be the first person to really use *large* datasets > in the IPT, although we have put several 10s of millions of records > in a test environment. > The IPT has been carefully designed to use a low memory footprint. > To start with, I would recommend 128MB for Tomcat, and enough disk > space to hold your output archives. There are no particular > requirements other than disk space for larger datasets. > > Will the input be sourced from a database? > > Best wishes, > Tim > > > On Aug 26, 2011, at 4:46 PM, Buonaiuto, Massimo (Bioversity) wrote: > > > Dear IPT staff, > > My name is Massimo Buonaiuto, Multimedia Web Specialist at > Bioversity International. > > I am very interested in installing locally the IPT for the > synchronization of some datasets we have with GBIF. > > At the moment we plan to synchronize about 5 GB and I would like to > receive suggestions and best practices for a correct and > successfully installation in-house. > > > Thank you and kind regards, > > Massimo > > _______________________________________________ > IPT mailing list > IPT(a)lists.gbif.org > http://lists.gbif.org/mailman/listinfo/ipt >

1 0

System requirements for IPT
by Buonaiuto, Massimo (Bioversity) 26 Aug '11

26 Aug '11

Dear IPT staff, My name is Massimo Buonaiuto, Multimedia Web Specialist at Bioversity International. I am very interested in installing locally the IPT for the synchronization of some datasets we have with GBIF. At the moment we plan to synchronize about 5 GB and I would like to receive suggestions and best practices for a correct and successfully installation in-house. Thank you and kind regards, Massimo

2 1

IPT XML metadata
by torsten dikow 19 Aug '11

19 Aug '11

Hi IPT users, I have recently uploaded occurrence data from a published article to the Pensoft IPT instance. In order to set-up an XML meta file, I used the Darwin Core Archive assistant and exported that XML file thinking that I would need it when I upload the resource to the IPT. But when I tried to map my columns on the IPT, I couldn't just upload that XML file and needed to select each column from a drop-down menu. This seems to me to be double work and I would rather like to use the XML meta file generated by the Darwin Core Archive assistant. This XML meta file had the columns arranged in a way so that they correspond to the exported occurrence data from my FileMaker Pro database. Is there any way I can use the XML meta file for mapping columns? Thanks, Torsten Dikow

2 1

Re: [IPT] IPT Digest, Vol 30, Issue 5
by Stewart, Aimee Marian 17 Aug '11

17 Aug '11

Sent from 785-331-8952. "ipt-request(a)lists.gbif.org" <ipt-request(a)lists.gbif.org> wrote: Send IPT mailing list submissions to ipt(a)lists.gbif.org To subscribe or unsubscribe via the World Wide Web, visit http://lists.gbif.org/mailman/listinfo/ipt or, via email, send a message with subject or body 'help' to ipt-request(a)lists.gbif.org You can reach the person managing the list at ipt-owner(a)lists.gbif.org When replying, please edit your Subject line so it is more specific than "Re: Contents of IPT digest..." Today's Topics: 1. Re: GBIF Case 1773: UTF8 (Mickael Graf) ---------------------------------------------------------------------- Message: 1 Date: Tue, 16 Aug 2011 14:38:53 +0200 From: Mickael Graf <Mickael.Graf(a)nrm.se> Subject: Re: [IPT] GBIF Case 1773: UTF8 To: "Burke Chih-Jen Ko (GBIF)" <bko(a)gbif.org> Cc: GBIF IPT mailing list <ipt(a)lists.gbif.org>, Johan Dunfalk <Johan.Dunfalk(a)nrm.se>, GBIF Helpdesk <helpdesk(a)gbif.org> Message-ID: <9F4FCAEFF4CBCB4CA1AF9E8BDABF4B8FA3629854B0@saruman> Content-Type: text/plain; charset="iso-8859-1" > 1. Did you use mysqldump to create the script that you sent me earlier? Yes, for the table definition and the data. The script for the view is hand written. > 2. From the script, I can see the database stores data in UTF-8, is it correct? Unfortunately no. Default settings for the server are latin1 so is the database. The data itself is (well, shoud be...) utf8 and the table definition has utf8 as character set. >3. Since the character in the dump sql is already broken, could you try, if you temporarily change the connection charset to utf-8, does the same dump contains the correct character for accented letters? Or please try --default-character-set=latin1 as one of your dump option. I just did a dump with --default-character-set=latin1 and I can read the accented letters in less,emacs and firefox (where I need to specify it's utf8). >4. Since all charset settings on your side appear to be latin1, are all databases hosted on the mysql server using UTF8 as the encoding? Including the one serves TapirLink? Most of my databases use latin1, but I have two of them created using utf8. But for all of them the accented letters are correctly displayed with TapirLink. Cheers, Micka?l I am trying to reproduce your environment here. Cheers, Burke On Aug 16, 2011, at 9:26 AM, Mickael Graf wrote: > Hi Burke, > > I tried both UTF-8, Latin1 and Windows 1252, but the result looks always the same. It looks like this setting has no influence on the final result, at least here. > > /Micka?l > > ________________________________________ > From: Burke Chih-Jen Ko (GBIF) [bko(a)gbif.org] > Sent: Monday, August 15, 2011 4:24 PM > To: Mickael Graf > Cc: Johan Dunfalk; GBIF IPT mailing list > Subject: Re: GBIF Case 1773: UTF8 > > Hi Micha?l, > > Have you tried using Latin 1 as the character encoding in the source data editing page of IPT? > > Burke > > On Aug 15, 2011, at 3:19 PM, Mickael Graf wrote: > >> Hi Burke, >> >> Changing my.cnf breaks everything. So I reversed back. I need to study how to correctly migrate my data to a complete utf8 system. >> MySQL still comes with latin1 as default (I just checked on my Ubuntu 11.04!) >> >> How can I test Tim's suggestion? >> >> Cheers, >> Micka?l >> >> ________________________________________ >> From: Burke Chih-Jen Ko (GBIF) [bko(a)gbif.org] >> Sent: Friday, August 12, 2011 4:30 PM >> To: Mickael Graf >> Cc: Johan Dunfalk; GBIF IPT mailing list >> Subject: Re: GBIF Case 1773: UTF8 >> >> Hi Micka?l, >> >> I can see from your script that the database is created using UTF-8, but it could be the connection characterset that interprets the UTF-8 information as iso-8859-1. Force opening a UTF-8 text file with N?rke using latin1 charset indeed render the text as N??rke. >> >> In the [mysqld] section of /etc/my.cnf, you can instruct the server to start with preferred characterset and collation: >> >> character_set_server=utf8 >> default-character-set=utf8 >> character_set_client=utf8 >> collation_server=utf8_general_ci >> skip-character-set-client-handshake >> >> The last line force the connection charset as the one specified for the server. >> >> So I suggest some steps: >> 1. Add lines above to your my.cnf >> 2. Restart the mysql, >> 3. First see if things still looks the same on TapirLink. >> 4. Try export the same sample script you gave us earlier, if the n?rke shows as it should be, then it should be fine on IPT. >> >> Let me know if this setting works. >> >> But if this change breaks TapirLink and others, you'll need to decide to configure others all or, see if Tim's suggestion works. (jdbc:mysql://localhost:3306/specimen_collections?autoReconnect=true&useUnicode=true&characterEncoding=UTF8&characterSetResults=UTF8) >> >> Cheers, >> >> Burke >> > > ------------------------------ _______________________________________________ IPT mailing list IPT(a)lists.gbif.org http://lists.gbif.org/mailman/listinfo/ipt End of IPT Digest, Vol 30, Issue 5 **********************************

1 0

FW: UTF8
by GBIF Helpdesk 11 Aug '11

11 Aug '11

-----Original Message----- From: Mickael Graf [mailto:Mickael.Graf@nrm.se] Sent: 10 August 2011 16:25 To: helpdesk(a)gbif.org Subject: UTF8 Hi, I am testing IPT 2 and NRM RingedBirds is the guinea pig. Well, I have some issue with the encoding because, while swedish characters work fine with TapirLink (see http://www.gbif.se/tapir/tapir_client.php, choose NRM-RingedBirds and make an inventory over StateProvince), it's a mess with IPT, both as a preview and as a zipped file. For instance 'Närke' is displayed as 'NÃ¤rke'. This happens regardless of the character encoding chosen under /source.do. The original data is UTF8, but then I don't know if any settings in tomcat need to be changed. Do you have some knowledge about this issue? I am very bad at java, so I don't know where to look (and issue 418 doesn't help). Cheers, Mickaël

2 1