[IPT] How does one upload large datasets to GBIF?

Tue Jul 7 15:10:50 UTC 2020

Hi Annie,

If your data is in a database (MySQL, Oracle, etc), you can make a database connection to the database within the IPT instead uploading a file. https://github.com/gbif/ipt/wiki/IPT2ManualManageResources.wiki#database-as-data-source

Or (and this is a hack) you create a same named file with a just few records that you can upload and map. Then you take the same named file with all the records and copy it over the uploaded file on the server in the ipt data folder. Then you go back to the IPT, edit the data source and click the Analyze button, Save, then publish. Someone with access to the directories on the server would need to do this.

Best,

Laura

Laura Anne Russell
Programme Officer for Participation and Engagement
Global Biodiversity Information Facility (GBIF) Secretariat

larussell at gbif.org (email)
laura.anne.russell (Skype)
@pagodarose (Twitter)
#CiteTheDOI @GBIF

https://www.gbif.org/

+45 35 33 35 51 (office, direct line)

GBIF
Universitetsparken 15
DK-2100 Copenhagen Ø
Denmark

From: IPT <ipt-bounces at lists.gbif.org> on behalf of "Simpson, Annie" <asimpson at usgs.gov>
Date: Tuesday, 7 July 2020 at 16.48
To: "ipt at lists.gbif.org" <ipt at lists.gbif.org>
Subject: [IPT] How does one upload large datasets to GBIF?

Colleagues:

What is the easiest or most popular way to send large datasets to GBIF, ones that are too large for the IPT software (I think that is more than 100MB zipped, 10+million records)? Does one modify their IPT instance? How? Or is there another process that is preferred?

We currently have IPT Version 2.3.6-r3985b6a installed and plan to upgrade to 2.4.0 soon.

A technical answer is what I seek (on behalf of our technical team).

Again my apologies if the answer to my question is easily found and I'm just not finding it.

Annie Simpson, BISON product owner
(she/her/hers)
BioFoundational Data Team
Science Analytics & Synthesis Program
U.S. Geological Survey
12201 Sunrise Valley Dr. Mailstop 302
Reston VA   20192
asimpson at usgs.gov
+1 703-648-4281
https://orcid.org/0000-0001-8338-5134
https://bison.usgs.gov
[Image removed by sender.]<https://bison.usgs.gov/>
Biodiversity Information Serving Our Nation (BISON)<https://bison.usgs.gov/>
USGS Biodiversity Information Serving Our Nation (BISON) is a unique, web-based Federal mapping resource for species occurrence data in the United States and its Territories and Canada, including marine Exclusive Economic Zones (EEZs).
bison.usgs.gov

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.gbif.org/pipermail/ipt/attachments/20200707/72ae96f9/attachment-0001.html>