June 2016 - ALA-Portal - lists.gbif.org

Solutions for some issues
by Marie Elise Lecoq 21 Jun '16

21 Jun '16

Hello all, Thanks again for your help. The 10 years anniversary of GBIF France went well (I guess), nobody told us that it was bad so ... I sent a lot of questions the last few weeks and I would like to give solutions that I found for my issues. It could help others people :-). INDEXATION : 1. About the wrong indexation, I found two bugs on the checklist used for the name indexing : - some of species don't have the entire classification (e.g. http://www.gbif.org/species/4814179) - some of them send a NullPointerException (see error below) when I ran the searchText command directly on the server. $ sudo nameindexer -testSearch "Canis familiaris Linnaeus, 1758" org.apache.lucene.index.IndexFormatTooNewException: Format version is not supported (resource: MMapIndexInput(path="/data/lucene/namematching/cb/segments.gen")): -3 (needs to be between -2 and -2) at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:722) at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:52) at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:65) at au.org.ala.names.search.ALANameSearcher.<init>(ALANameSearcher.java:117) at au.org.ala.names.search.DwcaNameIndexer.main(DwcaNameIndexer.java:488) 2. I still got issues with the punctuation or space in provider codes. My future work will focus on this. 3. I have successfully uploaded my dataset with more than 20 millions occurrences by following those steps : a. I uploaded a DwCArchive with 15 occurrences in order to create the dataset into the system. I need to do this because the Zip File library using in biocache store can't open a file bigger than 1Go. b. I copied the real DwC-Archive instead of the fake one on the /collectory/upload/ folder c. I asked our system administrator to increase the RAM in our Virtual Machine (from 4Go to 80Go). d. I made some correction into the collectory-pluggin (you can see my email that I sent on June, 1st) and the load, process and indexation works well after this. It took ages but it worked. e. Our data is now visible into our portal ( http://metadonnee.gbif.fr/public/showDataResource/dr179) I'm not sure it's the good way to do it but it works ! SPATIAL : I removed all the tools using environmental layers but I will be really interested by a training about it in order to install it :-)! DATA : For my error with Institution UID instead of name, I just changed "caches.collections.enabled" to true in the configuration file of biocache and it works perfectly. Thanks again! cheers, Marie --

1 0

[Urgent] Institution uid instead of institution name in advance research tab
by Marie Elise Lecoq 09 Jun '16

09 Jun '16

Hi all, This Friday, we will present the new GBIF France's spatial portal and occurrence portal during the 10 years anniversary of GBIF France. It will be a huge day with lot of talks. We are almost ready for this big day but we just found a last bug on the portal. On the advance tab, we can see institution's uid instead of institution's name. I looked on the "filter your result" and that's the same thing for the dataset name. (You can find attached two screenshots showing the problem). As I understand, those informations come from SOLR but I don't know how to fix it. Thanks a lot in advance for your help especially because that is really urgent, Marie -- [image: Inline image 2][image: Inline image 1]

2 1

Problems loading huge DwC Archive
by melecoq 02 Jun '16

02 Jun '16

Dear all, I'm still hold with my dataset with more than 20 millions occurrences. I understood that the issue was due to the large size of the Zipfile. It's too big for the ZipFile Java Api. I did a little trick and I was able to create the data resource. I integrate the DwC Archive with occurrence and verbatim files with just 15 occurrences, then I changed those files with the real ones and it seems to work. Now, the problem is when I try to load the Zip File into the Cassandra using load function from the biocache, I got a Java out of memory heap error because the code use the RAM to download, unzip and read the file. Unfortunately, 4 Go (zip file) and 23 Go (unzip file) is too big for it. Do you know if there is another way to do it ? Can I unzip the file and run the loading after it ? Or can I "manually" integrate data into the Cassandra ? Thanks in advance for your help. Cheers, Marie

4 5