[API-users] Downloading large numbers of datasets

Coggins, Clint ccoggins at usgs.gov
Wed Dec 7 21:53:08 CET 2016

I'm trying to download all the occurrence data with COUNTRY=US


I've requested a download file, which is here

However, I've been having a lot of trouble getting the download to complete
since it is so large(43.2GB). I've tried various browsers and also wget on
both Linux and Mac. It typically fails with a network error in the browser.
wget displayed strange behavior in that it claimed the download was
successful after downloading 4.1GB. This was on 64 bit linux with ext4, so
I don't think there was a filesystem limitation.

Any ideas on how to download this data? Does it make sense to write a
script to use the API to request the datasets one by one to split it up?

Thanks for any help

Clint Coggins
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gbif.org/pipermail/api-users/attachments/20161207/e36ffc47/attachment.html>

More information about the API-users mailing list