Thanks to the three of you for your complementary answers, that's really helpful.
Dave, I confirm that we are really interested in an improved version that triggers download in a more gentle way to avoid triggering GBIF's defense mechanism. And Marie-Elise, we are also very interested in your scripts. Don't hesitate if we can be of any help to one of you!
Have a great weekend,
Nico
Le 11/06/15 09:42, Marie Elise Lecoq a écrit :
Hi Nico,
For the point 3, I wrote some scripts in order to add data using the GBIF API. It helps me a lot especially for contacts and institutions. I will clean up and test a little bit more then I can put it on my github if that can help.
Cheers from Madagascar ! Marie
On Thu, Jun 11, 2015 at 10:33 AM, Tim Robertson <trobertson@gbif.org mailto:trobertson@gbif.org> wrote:
1) My understanding is the GBIF API has been bombarded recently and they have now put in place a limit of the number of concurrent downloads to 3. The collectory currently attempts 10 download at a time - hence our problem. We’ll try and push out an update to fix this very soon.I’d like to just explain why this was rushed out. This was a necessary step as GBIF.org <http://GBIF.org> was exposed to DOS attacks (often inadvertently) which was happening. In addition the GBIF R library now makes it trivial to download from GBIF and users can easily write a loop in code and trigger huge numbers of download which would block the system for others. We’ll continue to tune the values by monitoring operations, and probably introduce higher per-user limits so we can e.g. increase the concurrency for known ALA accounts. Please contact me if this causes any issue Nico.On 10 Jun 2015, at 1:59 pm, Nicolas Noé <n.noe@biodiversity.be <mailto:n.noe@biodiversity.be>> wrote: Hi all, I'm trying to set-up an experimental instance of ALA for Belgium. The idea is to present all data published by our belgian institutions, in order to give those institutations / collections / datasets more visibility. What's the recommended way to load this data (everything at GBIF published by Belgian institutions). I used the "Add all GBIF resource for a country" from the Collectory admin interface, but 1) I encountered some errors. All datasets detected correctly, but only 10 are loaded - for the rest I get the message "Failed. Please check your authentication credentials are valid". The authentication credentials are indeed correct, since I only entered once (so, also for the working dataset). When looking at my downloads on the GBIF website, I can see no track those "failed" datasets. Can I get debugging information ? Should I fill a bug report ? In the ala-collectory repository ? Or some other ? 2) Also, the /collectory/manage/gbifLoadCountry pages states that "This is a simple method of bootstrapping an installation with data provided by GBIF web services. This is not intended for long-term production use.". So maybe I should use another method to load all the data for my country. 3) How are the (dataset) metadata populated when using this method ? It seems the basic metadata is there, but not much more. Should we do it by hand ? Should we write a tool to do it ? How about updates? Thanks a lot! _______________________________________________ Ala-portal mailing list Ala-portal@lists.gbif.org <mailto:Ala-portal@lists.gbif.org> http://lists.gbif.org/mailman/listinfo/ala-portal_______________________________________________ Ala-portal mailing list Ala-portal@lists.gbif.org <mailto:Ala-portal@lists.gbif.org> http://lists.gbif.org/mailman/listinfo/ala-portal_______________________________________________ Ala-portal mailing list Ala-portal@lists.gbif.org <mailto:Ala-portal@lists.gbif.org> http://lists.gbif.org/mailman/listinfo/ala-portal--
Ala-portal mailing list Ala-portal@lists.gbif.org http://lists.gbif.org/mailman/listinfo/ala-portal