- ALA-Portal - lists.gbif.org

ALA Documentation - Highlight of the month
by Fabien CAVIERE 01 Feb '18

01 Feb '18

Hello everyone, This month, the wiki part I want to present you is: "Contribution to main project". Because of the incoming workshop in Madrid, we hope that new members will increase our community. As so, this chapter of the wiki is more important than never. There is only one page in this section, " [ https://github.com/AtlasOfLivingAustralia/documentation/wiki/Good-practices | Good-practices ] ", and it has three parts : * Architecture of a wiki page : informations about the skeleton of each page of the wiki. This ways, it is easier to create new page or to fulfill an incomplet one * Editing's rules : the wiki's law ; no one should break the law ! :) These rules are here to assure that each page of the wiki follow the same pattern. Like this, new comers won't be disoriented. * Rules of text formatting : tips and advices for the creation or edition of wiki pages The instructions describe in "Contribution to main project" are strong recommandations to complete the other parts of the documentation. If you have any questions, feel free to send me an email (through the mailing list or directly at dev(a)gbif.fr). Have a good days. Cheers Fabien.

1 0

ALA Documentation - Highlight of the month
by Fabien CAVIERE 07 Dec '17

07 Dec '17

Hello everyone, At the CESP project between Canadensys and GBIF France, we have decide to re-shape the wiki of the ALA documentation. To explains why with we did it, every month, I will highligh one section of the documentation to tell you her purpose, the informations you can find in it, and how you can contribute to the fullfill of this section. This month, the chosen one is "Portals in production" because it's one of the big section of the wiki and one of the first place where people will look for informations. Each page have three parts : * In the First part, "Presentation", you will have a lot of general and technical informations, group by sub-parts : - architecture scheme : a scheme of all the virtual machines who compose the architecture. For each virtual machine, you will have the OS, the amount of RAM, the number of CPU, the amount for storage and all the services that inside it. - introduction : a few sentences that explains the genese and the current state of the project (the actors, the administrative process, ...), the actors involved, ... - metadata on the portal development : contains the number of developers, the time allowed to the portal and other informations on the portal project of the node - general informations : all about the structure that host the portal, the explanations and details on the portal architecture, the context of the beginning, the current state and the future of the portal, ... - stats of your portal : data about the daily traffic and the guess of the amount of data in 5 years - specific settings : if the portal needed a specific setting or set up, you can find them here * In the second part, "Diagram", you have a diagram that show all the dependences between the differents services, elements and modules on the portal ; it very handly for beginners for understands the process of the portal * And in the third and last part "Mains local points to this portal", you can get all the informations about the specificity of the portal compared to others : the differences about the modules implemented, the code modifications, the technical constraints of the regional language, the settings of the servers, and others Currently, none of the differents portals in production pages are fully complete : there are a lot of informations missing. If you want to complete the blank, don't forget to follow the instructions using the [ https://github.com/AtlasOfLivingAustralia/documentation/wiki/Good-pratices-… | Good-practices page ] ; you can find the completion scheme on the following pages : - [ https://github.com/AtlasOfLivingAustralia/documentation/wiki/Canadensys-pro… | Canadensys production environment ] - [ https://github.com/AtlasOfLivingAustralia/documentation/wiki/CRBio-producti… | CRBio production environment ] - [ https://github.com/AtlasOfLivingAustralia/documentation/wiki/GBIF-France-pr… | GBIF France production environment ] - [ https://github.com/AtlasOfLivingAustralia/documentation/wiki/GBIF-Sweden-pr… | GBIF Sweden production environment ] As always, if you want to fullfill other part of the documentation, go for it Don't forget to assign yourself on this [ https://docs.google.com/spreadsheets/d/1e1Y9R_vUUXbO5drE11CZlZS0wTOme-CcBGL… | document ] if you want to complete a part of the wiki. If you have any questions, feel free to send me an email. Have a good days. Cheers Fabien.

1 0

ALA Documentation - Highlight of the month
by Fabien CAVIERE 04 Dec '17

04 Dec '17

Hello everyone, At the CESP project between Canadensys and GBIF France, we have decide to re-shape the wiki of the ALA documentation. To explains why with we did it, every month, I will highligh one section of the documentation to tell you her purpose, the informations you can find in it, and how you can contribute to the fullfill of this section. This month, the chosen one is "Portals in production" because it's one of the big section of the wiki and one of the first place where people will look for informations. Each page have three parts : * In the First part, "Presentation", you will have a lot of general and technical informations, group by sub-parts : - architecture scheme : a scheme of all the virtual machines who compose the architecture. For each virtual machine, you will have the OS, the amount of RAM, the number of CPU, the amount for storage and all the services that inside it. - introduction : a few sentences that explains the genese and the current state of the project (the actors, the administrative process, ...), the actors involved, ... - metadata on the portal development : contains the number of developers, the time allowed to the portal and other informations on the portal project of the node - general informations : all about the structure that host the portal, the explanations and details on the portal architecture, the context of the beginning, the current state and the future of the portal, ... - stats of your portal : data about the daily traffic and the guess of the amount of data in 5 years - specific settings : if the portal needed a specific setting or set up, you can find them here * In the second part, "Diagram", you have a diagram that show all the dependences between the differents services, elements and modules on the portal ; it very handly for beginners for understands the process of the portal * And in the third and last part "Mains local points to this portal", you can get all the informations about the specificity of the portal compared to others : the differences about the modules implemented, the code modifications, the technical constraints of the regional language, the settings of the servers, and others Currently, none of the differents portals in production pages are fully complete : there are a lot of informations missing. If you want to complete the blank, don't forget to follow the instructions using the [ https://github.com/AtlasOfLivingAustralia/documentation/wiki/Good-pratices-… | Good-practices page ] ; you can find the completion scheme on the following pages : - [ https://github.com/AtlasOfLivingAustralia/documentation/wiki/Canadensys-pro… | Canadensys production environment ] - [ https://github.com/AtlasOfLivingAustralia/documentation/wiki/CRBio-producti… | CRBio production environment ] - [ https://github.com/AtlasOfLivingAustralia/documentation/wiki/GBIF-France-pr… | GBIF France production environment ] - [ https://github.com/AtlasOfLivingAustralia/documentation/wiki/GBIF-Sweden-pr… | GBIF Sweden production environment ] As always, if you want to fullfill other part of the documentation, go for it ;) Don't forget to assign yourself on this [ https://docs.google.com/spreadsheets/d/1e1Y9R_vUUXbO5drE11CZlZS0wTOme-CcBGL… | document ] if you want to complete a part of the wiki. If you have any questions, feel free to send me an email. Have a good days. Cheers Fabien. --

1 0

Document for the ALA documentation
by Fabien CAVIERE 16 Oct '17

16 Oct '17

Hello everyone, As I talked about it during the TDWG 2017, I made a Google Sheets for helping the community to complete the ALA documentation. You can find it on the following link: https://docs.google.com/spreadsheets/d/1e1Y9R_vUUXbO5drE11CZlZS0wTOme-CcBGL… This document gives you the opportunity to assign yourself to any part of the wiki. To do this, you just have to fill your name in the "In charge" column. There is a color code for helping participants to understand the state of each chapter : - red - empty: the page needs to be entirely written. - yellow - incomplete: the page is partly written but some important points are missing - light green - almost done: the page is almost finished but some additional information need to be added. - green - complete: the page is fully written. If a page is already tagged "Completed" but you have more information you wish to add, go ahead and follow the instructions using the Good-practices <https://github.com/AtlasOfLivingAustralia/documentation/wiki/Good-practices>. Each month, I will highlight a section of the documentation to help its completion. If you have any questions, feel free to send me an email. Have a good day. Cheers, Fabien. --

1 0

[ALA Documentation] More information
by Fabien CAVIERE 12 Sep '17

12 Sep '17

Hello everyone, In my previous message, I forgot two information: - ALA wiki URL: https://github.com/AtlasOfLivingAustralia/documentation/wiki. - If you want to participate, you can write to me directly by sending an email at dev(a)gbif.fr <mailto:dev@gbif.fr>. Have a nice day, Cheers, Fabien --

1 0

Help about the ALA documentation
by Fabien CAVIERE 06 Sep '17

06 Sep '17

Hello everyone, One of the main objectives of the mentoring between Canadensys and GBIF France is to improve the documentation in English and in French. As you may see, we made several modifications on the Documentation wiki especially on the structure. However, there are still missing parts on the documentation; mostly technical so we need your help to fulfill them. In order to organize this, I would like to know if you are interested by working on this ? Do you have any time to allocate on this task ? If so, could you please tell me when you are available ? And which part of the documentation would you wish to complete ? Have a nice day. Cheers, Fabien Caviere --

1 0

Solutions for some issues
by Marie Elise Lecoq 21 Jun '16

21 Jun '16

Hello all, Thanks again for your help. The 10 years anniversary of GBIF France went well (I guess), nobody told us that it was bad so ... I sent a lot of questions the last few weeks and I would like to give solutions that I found for my issues. It could help others people :-). INDEXATION : 1. About the wrong indexation, I found two bugs on the checklist used for the name indexing : - some of species don't have the entire classification (e.g. http://www.gbif.org/species/4814179) - some of them send a NullPointerException (see error below) when I ran the searchText command directly on the server. $ sudo nameindexer -testSearch "Canis familiaris Linnaeus, 1758" org.apache.lucene.index.IndexFormatTooNewException: Format version is not supported (resource: MMapIndexInput(path="/data/lucene/namematching/cb/segments.gen")): -3 (needs to be between -2 and -2) at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:722) at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:52) at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:65) at au.org.ala.names.search.ALANameSearcher.<init>(ALANameSearcher.java:117) at au.org.ala.names.search.DwcaNameIndexer.main(DwcaNameIndexer.java:488) 2. I still got issues with the punctuation or space in provider codes. My future work will focus on this. 3. I have successfully uploaded my dataset with more than 20 millions occurrences by following those steps : a. I uploaded a DwCArchive with 15 occurrences in order to create the dataset into the system. I need to do this because the Zip File library using in biocache store can't open a file bigger than 1Go. b. I copied the real DwC-Archive instead of the fake one on the /collectory/upload/ folder c. I asked our system administrator to increase the RAM in our Virtual Machine (from 4Go to 80Go). d. I made some correction into the collectory-pluggin (you can see my email that I sent on June, 1st) and the load, process and indexation works well after this. It took ages but it worked. e. Our data is now visible into our portal ( http://metadonnee.gbif.fr/public/showDataResource/dr179) I'm not sure it's the good way to do it but it works ! SPATIAL : I removed all the tools using environmental layers but I will be really interested by a training about it in order to install it :-)! DATA : For my error with Institution UID instead of name, I just changed "caches.collections.enabled" to true in the configuration file of biocache and it works perfectly. Thanks again! cheers, Marie --

1 0

[Urgent] Institution uid instead of institution name in advance research tab
by Marie Elise Lecoq 08 Jun '16

08 Jun '16

Hi all, This Friday, we will present the new GBIF France's spatial portal and occurrence portal during the 10 years anniversary of GBIF France. It will be a huge day with lot of talks. We are almost ready for this big day but we just found a last bug on the portal. On the advance tab, we can see institution's uid instead of institution's name. I looked on the "filter your result" and that's the same thing for the dataset name. (You can find attached two screenshots showing the problem). As I understand, those informations come from SOLR but I don't know how to fix it. Thanks a lot in advance for your help especially because that is really urgent, Marie -- [image: Inline image 2][image: Inline image 1]

2 1

Problems loading huge DwC Archive
by melecoq 01 Jun '16

01 Jun '16

Dear all, I'm still hold with my dataset with more than 20 millions occurrences. I understood that the issue was due to the large size of the Zipfile. It's too big for the ZipFile Java Api. I did a little trick and I was able to create the data resource. I integrate the DwC Archive with occurrence and verbatim files with just 15 occurrences, then I changed those files with the real ones and it seems to work. Now, the problem is when I try to load the Zip File into the Cassandra using load function from the biocache, I got a Java out of memory heap error because the code use the RAM to download, unzip and read the file. Unfortunately, 4 Go (zip file) and 23 Go (unzip file) is too big for it. Do you know if there is another way to do it ? Can I unzip the file and run the loading after it ? Or can I "manually" integrate data into the Cassandra ? Thanks in advance for your help. Cheers, Marie

4 5

[Spatial] Error during integration of layer
by Marie Elise Lecoq 27 May '16

27 May '16

Hi all, When I integreted a new layer on the spatial portal, I got this error : org.postgresql.util.PSQLException: ERROR: value too long for type character varying(160) at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2102) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1835) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257) at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388) at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:381) at au.org.ala.layers.ingestion.contextual.ContextualFromShapefileDatabaseLoader.importShapeFile(ContextualFromShapefileDatabaseLoader.java:293) at au.org.ala.layers.ingestion.contextual.ContextualFromShapefileDatabaseLoader.create(ContextualFromShapefileDatabaseLoader.java:99) at au.org.ala.layers.ingestion.contextual.ContextualFromShapefileDatabaseLoader.main(ContextualFromShapefileDatabaseLoader.java:54) It seems that one of the name is too long. I tried to fixed on the data and I tried to re-run the script sh but I was not possible because the shapefile is already here. How can reload it ? Do I have to remove rows from the postgresql and remove files in /data/ala/data/layers/ready/shape/ ? Thanks for your help, Cheers, Marie --

2 2