<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none"><!-- p { margin-top: 0px; margin-bottom: 0px; }--></style>
</head>
<body dir="ltr" style="font-size:12pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
<p>Jorrit pointed out this thread to us at iDigBio. Downloading and importing data into a relational database will work great, especially if as Jan said you can cut the data size down to a reasonable amount.<br>
</p>
<p><span style="font-size: 12pt;"><br>
</span></p>
<p><span style="font-size: 12pt;">Another approach we've been working on in a collaboration called GUODA [1] is to build an Apache Spark environment with pre-formatted data frames with common data sets in them for researchers to use. This approach would offer
a remote service where you could write arbitrary Spark code, probably in Jupyter notebooks, to iterate over data. Spark does a lot of cool stuff including GraphX which might be of interest. This is </span>definitely pre-alpha at this point and if anyone is
interested, I'd like to hear your thoughts.<span style="font-size: 12pt;"> I'll also be at SPNHC talking about this.</span></p>
<p><span style="font-size: 12pt;"><br>
</span></p>
<p>One thing we've found in working on this is that importing data into a structured data format isn't always easy. If you only want a few columns, it'll be fine. But getting the data typing, format standardization, and column name syntax of the whole width
of an iDigBio record right requires some code. <span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 16px; background-color: rgb(255, 255, 255);">
I looked to see if </span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 16px; background-color: rgb(255, 255, 255);">EcoData R</span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 16px; background-color: rgb(255, 255, 255);">etriever
[2] had a GBIF data source and they have an eBird o</span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 16px; background-color: rgb(255, 255, 255);">ne that perhaps you might find useful as a</span> starting point if you wanted
to try to use someone else's code to download and import data.<br>
</p>
<p><br>
</p>
<p>For other data structures like BHL, we're kind of making stuff up since we're packaging a relational structure and not something nearly as flat as GBIF and DWC stuff.
<br>
</p>
<p><br>
</p>
<p>[1] <a href="http://guoda.bio/">http://guoda.bio/</a><br>
</p>
<p>[2] <a href="http://www.ecodataretriever.org/">http://www.ecodataretriever.org/</a><br>
</p>
<p><br>
</p>
<div id="Signature">
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
<div style="font-family:Tahoma; font-size:13px">
<div style="font-family:Tahoma; font-size:13px">Matthew Collins<br>
Technical Operations Manager<br>
Advanced Computing and Information Systems Lab, ECE<br>
University of Florida<br>
<span class="Object" id="OBJ_PREFIX_DWT1049_com_zimbra_phone"><a href="callto:352-392-5414" tabindex="0" id="NoLP">352-392-5414</a></span></div>
</div>
</div>
</div>
<div style="word-wrap:break-word">
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> jorrit poelen <jhpoelen@xs4all.nl><br>
<b>Sent:</b> Monday, May 30, 2016 11:16 AM<br>
<b>To:</b> Collins, Matthew; Thompson, Alexander M; Hammock, Jennifer<br>
<b>Subject:</b> Fwd: [API-users] Is there any NEO4J or graph-based driver for this API ?</font>
<div> </div>
</div>
<div>Hey y’all:
<div class=""><br class="">
</div>
<div class="">Interesting request below on the GBIF mailing list - sounds like a perfect fit for the GUODA use cases. </div>
<div class=""><br class="">
</div>
<div class="">Would it be too early to jump onto this thread and share our efforts/vision?</div>
<div class=""><br class="">
</div>
<div class="">thx,<br>
</div>
<div class="">-jorrit<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">Begin forwarded message:</div>
<br class="Apple-interchange-newline">
<div class="" style="margin-top:0px; margin-right:0px; margin-bottom:0px; margin-left:0px">
<span class="" style=""><b class="">From: </b></span><span class="" style="">Jan Legind <<a href="mailto:jlegind@gbif.org" class="">jlegind@gbif.org</a>><br class="">
</span></div>
<div class="" style="margin-top:0px; margin-right:0px; margin-bottom:0px; margin-left:0px">
<span class="" style=""><b class="">Subject: </b></span><span class="" style=""><b class="">Re: [API-users] Is there any NEO4J or graph-based driver for this API ?</b><br class="">
</span></div>
<div class="" style="margin-top:0px; margin-right:0px; margin-bottom:0px; margin-left:0px">
<span class="" style=""><b class="">Date: </b></span><span class="" style="">May 30, 2016 at 5:48:51 AM PDT<br class="">
</span></div>
<div class="" style="margin-top:0px; margin-right:0px; margin-bottom:0px; margin-left:0px">
<span class="" style=""><b class="">To: </b></span><span class="" style="">Mauro Cavalcanti <<a href="mailto:maurobio@gmail.com" class="">maurobio@gmail.com</a>>, "Juan M. Escamilla Molgora" <<a href="mailto:j.escamillamolgora@lancaster.ac.uk" class="">j.escamillamolgora@lancaster.ac.uk</a>><br class="">
</span></div>
<div class="" style="margin-top:0px; margin-right:0px; margin-bottom:0px; margin-left:0px">
<span class="" style=""><b class="">Cc: </b></span><span class="">"<a href="mailto:api-users@lists.gbif.org" class="">api-users@lists.gbif.org</a>" <<a href="mailto:api-users@lists.gbif.org" class="">api-users@lists.gbif.org</a>><br class="">
</span></div>
<br class="">
<div class="">
<div class="WordSection1" style="font-family:Helvetica; font-size:12px; font-style:normal; font-weight:normal; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px">
<div class="" style="margin:0cm 0cm 0.0001pt; font-size:12pt; font-family:'Times New Roman',serif">
<span class="" style="font-family:Calibri,sans-serif; color:rgb(31,73,125)">Dear Juan,</span></div>
<div class="" style="margin:0cm 0cm 0.0001pt; font-size:12pt; font-family:'Times New Roman',serif">
<span class="" style="font-family:Calibri,sans-serif; color:rgb(31,73,125)"> </span></div>
<div class="" style="margin:0cm 0cm 0.0001pt; font-size:12pt; font-family:'Times New Roman',serif">
<span class="" style="font-family:Calibri,sans-serif; color:rgb(31,73,125)">Unfortunately we have no tool for creating these kind of SQL like queries to the portal. I am sure you are aware that the filters in the occurrence search pages can be applied in combination
in numerous ways. The API can go even further in this regard[1], but it not well suited for retrieving occurrence records since there is a 200.000 records ceiling making it unfit for species exceeding this number.</span></div>
<div class="" style="margin:0cm 0cm 0.0001pt; font-size:12pt; font-family:'Times New Roman',serif">
<span class="" style="font-family:Calibri,sans-serif; color:rgb(31,73,125)"> </span></div>
<div class="" style="margin:0cm 0cm 0.0001pt; font-size:12pt; font-family:'Times New Roman',serif">
<span class="" style="font-family:Calibri,sans-serif; color:rgb(31,73,125)">There is going be updates to the pygbif package[2] in the near future that will enable you to launch user downloads programmatically where a whole list of different species can be used
as a query parameter as well as adding polygons.[3]</span></div>
<div class="" style="margin:0cm 0cm 0.0001pt; font-size:12pt; font-family:'Times New Roman',serif">
<span class="" style="font-family:Calibri,sans-serif; color:rgb(31,73,125)"> </span></div>
<div class="" style="margin:0cm 0cm 0.0001pt; font-size:12pt; font-family:'Times New Roman',serif">
<span class="" style="font-family:Calibri,sans-serif; color:rgb(31,73,125)">In the meantime, Mauro’s suggestion is excellent. If you can narrow your search down until it returns a manageable download (say less than 100 million records), importing this into
a database should be doable. From there, you can refine using SQL queries.</span></div>
<div class="" style="margin:0cm 0cm 0.0001pt; font-size:12pt; font-family:'Times New Roman',serif">
<span class="" style="font-family:Calibri,sans-serif; color:rgb(31,73,125)"> </span></div>
<div class="" style="margin:0cm 0cm 0.0001pt; font-size:12pt; font-family:'Times New Roman',serif">
<span class="" style="font-family:Calibri,sans-serif; color:rgb(31,73,125)">Best,</span></div>
<div class="" style="margin:0cm 0cm 0.0001pt; font-size:12pt; font-family:'Times New Roman',serif">
<span class="" style="font-family:Calibri,sans-serif; color:rgb(31,73,125)">Jan K. Legind, GBIF Data manager </span></div>
<div class="" style="margin:0cm 0cm 0.0001pt; font-size:12pt; font-family:'Times New Roman',serif">
<span class="" style="font-family:Calibri,sans-serif; color:rgb(31,73,125)"> </span></div>
<div class="" style="margin:0cm 0cm 0.0001pt; font-size:12pt; font-family:'Times New Roman',serif">
<span class="" style="font-family:Calibri,sans-serif; color:rgb(31,73,125)">[1]<span class="Apple-converted-space"> </span><a href="http://www.gbif.org/developer/occurrence#search" class="" style="color:purple; text-decoration:underline">http://www.gbif.org/developer/occurrence#search</a></span></div>
<div class="" style="margin:0cm 0cm 0.0001pt; font-size:12pt; font-family:'Times New Roman',serif">
<span class="" style="font-family:Calibri,sans-serif; color:rgb(31,73,125)">[2]<span class="Apple-converted-space"> </span><a href="https://github.com/sckott/pygbif" class="" style="color:purple; text-decoration:underline">https://github.com/sckott/pygbif</a></span></div>
<div class="" style="margin:0cm 0cm 0.0001pt; font-size:12pt; font-family:'Times New Roman',serif">
<span class="" style="font-family:Calibri,sans-serif; color:rgb(31,73,125)">[3]<span class="Apple-converted-space"> </span><a href="https://github.com/jlegind/GBIF-downloads" class="" style="color:purple; text-decoration:underline">https://github.com/jlegind/GBIF-downloads</a></span></div>
<div class="" style="margin:0cm 0cm 0.0001pt; font-size:12pt; font-family:'Times New Roman',serif">
<span class="" style="font-family:Calibri,sans-serif; color:rgb(31,73,125)"> </span></div>
<div class="" style="margin:0cm 0cm 0.0001pt; font-size:12pt; font-family:'Times New Roman',serif">
<b class=""><span class="" style="font-size:10pt; font-family:Tahoma,sans-serif">From:</span></b><span class="" style="font-size:10pt; font-family:Tahoma,sans-serif"><span class="Apple-converted-space"> </span>API-users [<a href="mailto:api-users-bounces@lists.gbif.org" class="" style="color:purple; text-decoration:underline">mailto:api-users-bounces@lists.gbif.org</a>]<span class="Apple-converted-space"> </span><b class="">On
Behalf Of<span class="Apple-converted-space"> </span></b>Mauro Cavalcanti<br class="">
<b class="">Sent:</b><span class="Apple-converted-space"> </span>30. maj 2016 14:06<br class="">
<b class="">To:</b><span class="Apple-converted-space"> </span>Juan M. Escamilla Molgora<br class="">
<b class="">Cc:</b><span class="Apple-converted-space"> </span><a href="mailto:api-users@lists.gbif.org" class="" style="color:purple; text-decoration:underline">api-users@lists.gbif.org</a><br class="">
<b class="">Subject:</b><span class="Apple-converted-space"> </span>Re: [API-users] Is there any NEO4J or graph-based driver for this API ?</span></div>
<div class="" style="margin:0cm 0cm 0.0001pt; font-size:12pt; font-family:'Times New Roman',serif">
</div>
<div class="">
<div class="">
<div class="">
<p class="MsoNormal" style="margin:0cm 0cm 12pt; font-size:12pt; font-family:'Times New Roman',serif">
Hi,</p>
</div>
<p class="MsoNormal" style="margin:0cm 0cm 12pt; font-size:12pt; font-family:'Times New Roman',serif">
One solution I have successfully adopted for this is to download the records (either "manually" via browser or, yet better, using a Python script using the fine pygbif library), storing them into a MySQL or SQLite database and then perform the relational queries.
I can provide examples if you are interested.</p>
</div>
<div class="" style="margin:0cm 0cm 0.0001pt; font-size:12pt; font-family:'Times New Roman',serif">
Best regards,</div>
</div>
<div class="">
<div class="" style="margin:0cm 0cm 0.0001pt; font-size:12pt; font-family:'Times New Roman',serif">
</div>
<div class="">
<div class="" style="margin:0cm 0cm 0.0001pt; font-size:12pt; font-family:'Times New Roman',serif">
2016-05-30 8:59 GMT-03:00 Juan M. Escamilla Molgora <<a href="mailto:j.escamillamolgora@lancaster.ac.uk" target="_blank" class="" style="color:purple; text-decoration:underline">j.escamillamolgora@lancaster.ac.uk</a>>:</div>
<div class="" style="margin:0cm 0cm 0.0001pt; font-size:12pt; font-family:'Times New Roman',serif">
Hola,<br class="">
<br class="">
Is there any API for making relational queries like taxonomy, location or timestamp?<br class="">
<br class="">
Thank you and best wishes<br class="">
<br class="">
Juan<br class="">
_______________________________________________<br class="">
API-users mailing list<br class="">
<a href="mailto:API-users@lists.gbif.org" target="_blank" class="" style="color:purple; text-decoration:underline">API-users@lists.gbif.org</a><br class="">
<a href="http://lists.gbif.org/mailman/listinfo/api-users" target="_blank" class="" style="color:purple; text-decoration:underline">http://lists.gbif.org/mailman/listinfo/api-users</a></div>
</div>
<div class="" style="margin:0cm 0cm 0.0001pt; font-size:12pt; font-family:'Times New Roman',serif">
<br class="">
<br clear="all" class="">
<br class="">
--<span class="Apple-converted-space"> </span></div>
<div class="">
<div class="" style="margin:0cm 0cm 0.0001pt; font-size:12pt; font-family:'Times New Roman',serif">
Dr. Mauro J. Cavalcanti<br class="">
E-mail:<span class="Apple-converted-space"> </span><a href="mailto:maurobio@gmail.com" target="_blank" class="" style="color:purple; text-decoration:underline">maurobio@gmail.com</a><br class="">
Web:<span class="Apple-converted-space"> </span><a href="http://sites.google.com/site/maurobio" target="_blank" class="" style="color:purple; text-decoration:underline">http://sites.google.com/site/maurobio</a></div>
</div>
</div>
</div>
<span class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-weight:normal; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px; float:none; display:inline!important">_______________________________________________</span><br class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-weight:normal; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px">
<span class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-weight:normal; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px; float:none; display:inline!important">API-users
mailing list</span><br class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-weight:normal; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px">
<a href="mailto:API-users@lists.gbif.org" class="" style="color:purple; text-decoration:underline; font-family:Helvetica; font-size:12px; font-style:normal; font-weight:normal; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px">API-users@lists.gbif.org</a><br class="" style="font-family:Helvetica; font-size:12px; font-style:normal; font-weight:normal; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px">
<a href="http://lists.gbif.org/mailman/listinfo/api-users" class="" style="color:purple; text-decoration:underline; font-family:Helvetica; font-size:12px; font-style:normal; font-weight:normal; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px">http://lists.gbif.org/mailman/listinfo/api-users</a></div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</body>
</html>