<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Dear all <br>
<br>
Here is an current problem to solve :-) for the birdhouse WPS and
GBIF developers.<br>
There is a species distribution process in the birdhouse WPS,
processing Climate data based on occurrence coordinates from GBIF.
<br>
<br>
Currently the GBIF data are given to the process via an url of the
GBIF csv (which had to be generated manually before).<br>
I was implementing a direct occurrence search from birdhouse to GBIF
with pygbif. Worked fine BUT: there is a limit of 300 records which
is far to less to train the Species distribution model. <br>
<br>
So here the question: Is there a way to request the species
occurrence coordinates somehow directly (e.g. like a WPS request)? <br>
<br>
(The previous conversation is following)<br>
<br>
And some link for the background informations: <br>
<br>
Birdhouse:<br>
<a class="moz-txt-link-freetext" href="http://birdhouse.readthedocs.io/en/latest/">http://birdhouse.readthedocs.io/en/latest/</a><br>
GBIF: <br>
<a class="moz-txt-link-freetext" href="http://www.gbif.org/">http://www.gbif.org/</a><br>
<br>
Species distribution process: <br>
<a class="moz-txt-link-freetext" href="http://flyingpigeon.readthedocs.io/en/latest/tutorials/sdm.html">http://flyingpigeon.readthedocs.io/en/latest/tutorials/sdm.html</a><br>
<br>
Birdhouse architecture: <br>
<a class="moz-txt-link-freetext" href="https://github.com/bird-house/birdhouse-docs/blob/master/slides/birdhouse-architecture/birdhouse-architecture.pdf">https://github.com/bird-house/birdhouse-docs/blob/master/slides/birdhouse-architecture/birdhouse-architecture.pdf</a><br>
<br>
Merci <br>
Nils<br>
<br>
-------- Forwarded Message -------
<div class="moz-forward-container">
<table class="moz-email-headers-table" border="0" cellpadding="0"
cellspacing="0">
<tbody>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">Subject:
</th>
<td>Re: pygbif for occurrence coordinates</td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">Date: </th>
<td>Thu, 2 Jun 2016 09:00:49 -0300</td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">From: </th>
<td>Mauro Cavalcanti <a class="moz-txt-link-rfc2396E" href="mailto:maurobio@gmail.com"><maurobio@gmail.com></a></td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">To: </th>
<td>Nils Hempelmann <a class="moz-txt-link-rfc2396E" href="mailto:info@nilshempelmann.de"><info@nilshempelmann.de></a></td>
</tr>
<tr>
<th align="RIGHT" valign="BASELINE" nowrap="nowrap">CC: </th>
<td><a class="moz-txt-link-abbreviated" href="mailto:wps-dev@lists.dkrz.de">wps-dev@lists.dkrz.de</a>, Carsten Ehbrecht
<a class="moz-txt-link-rfc2396E" href="mailto:ehbrecht@dkrz.de"><ehbrecht@dkrz.de></a>, <a class="moz-txt-link-abbreviated" href="mailto:Wolfgang.Falk@lwf.bayern.de">Wolfgang.Falk@lwf.bayern.de</a>,
Scott Chamberlain <a class="moz-txt-link-rfc2396E" href="mailto:myrmecocystus@gmail.com"><myrmecocystus@gmail.com></a></td>
</tr>
</tbody>
</table>
<br>
<br>
<p dir="ltr">Nils,</p>
<p dir="ltr">No, there is no other way to get at once a larger
number of records using the GBIF API (although one could be able
to achieve this by sequentially querying the server using
"batches" of 200 or 300 records each). As I said, there are good
operational reasons for the GBIF developers to have imposed such
limit.</p>
<p dir="ltr">As of your other question, it should better be put to
the GBIF developers themselves (in the discussion list, so that
we can all benefit from the answers! :-))</p>
<p dir="ltr">With warmest regards,</p>
<p dir="ltr">--<br>
Dr. Mauro J. Cavalcanti<br>
E-mail: <a moz-do-not-send="true"
href="mailto:maurobio@gmail.com">maurobio@gmail.com</a><br>
Web: <a moz-do-not-send="true"
href="http://sites.google.com/site/maurobio">http://sites.google.com/site/maurobio</a></p>
<div class="gmail_quote">Em 02/06/2016 08:49, "Nils Hempelmann"
<<a moz-do-not-send="true"
href="mailto:info@nilshempelmann.de">info@nilshempelmann.de</a>>
escreveu:<br type="attribution">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> Hi Mauro<br>
<br>
Oh ... <br>
No way to set it unlimited?<br>
The 'manual via browser' option and parsing the returning
csv is the current status. <br>
or any alternative to pygbif?<br>
<br>
If I understood it correctly, GBIF database is organized as
Web Server, so I gues there should be a way to connect to
the birdhouse WPS, am I right? <br>
<br>
(put the web-dev list in copy)<br>
<br>
Merci <br>
Nils <br>
<br>
<br>
<div>On 02/06/2016 12:40, Mauro Cavalcanti wrote:<br>
</div>
<blockquote type="cite">
<p dir="ltr">Nils,</p>
<p dir="ltr">That's a limit imposed (for good operational
reasons) by the GBIF API. If you want a larger number of
records, you'll have to download them "manually" (that
is, via browser) and then parse locally the csv file
returned from the GBIF server.</p>
<p dir="ltr">Hope this helps.</p>
<p dir="ltr">Best regards,</p>
<p dir="ltr">--<br>
Dr. Mauro J. Cavalcanti<br>
E-mail: <a moz-do-not-send="true"
href="mailto:maurobio@gmail.com" target="_blank">maurobio@gmail.com</a><br>
Web: <a moz-do-not-send="true"
href="http://sites.google.com/site/maurobio"
target="_blank">http://sites.google.com/site/maurobio</a></p>
<div class="gmail_quote">Em 02/06/2016 04:48, "Nils
Hempelmann" <<a moz-do-not-send="true"
href="mailto:info@nilshempelmann.de" target="_blank">info@nilshempelmann.de</a>>
escreveu:<br type="attribution">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> Hi Scott <br>
<br>
works fine. Thanks a lot. Just have a question for
the search limits: <br>
<br>
The maximal records seems to be limited to 300. Is
that on purpose?<br>
And requesting more than 200000 gives a 'bad
request'<br>
<br>
Merci <br>
Nils <br>
<br>
<br>
In [68]: len( occurrences.search(taxonKey=key,
limit=100)['results'])<br>
Out[68]: 100<br>
<br>
In [69]: len( occurrences.search(taxonKey=key,
limit=300)['results'])<br>
Out[69]: 300<br>
<br>
In [70]: len( occurrences.search(taxonKey=key,
limit=3000)['results'])<br>
Out[70]: 300<br>
<br>
In [71]: len( occurrences.search(taxonKey=key,
limit=200000)['results'])<br>
Out[71]: 300<br>
<br>
In [72]: len( occurrences.search(taxonKey=key,
limit=200001)['results'])<br>
---------------------------------------------------------------------------<br>
HTTPError Traceback
(most recent call last)<br>
<ipython-input-72-2f7d7b4ccba0> in
<module>()<br>
----> 1 len( occurrences.search(taxonKey=key,
limit=200001)['results'])<br>
<br>
/home/nils/.conda/envs/birdhouse/lib/python2.7/site-packages/pygbif/occurrences/search.pyc
in search(taxonKey, scientificName, country,
publishingCountry, hasCoordinate, typeStatus,
recordNumber, lastInterpreted, continent, geometry,
recordedBy, basisOfRecord, datasetKey, eventDate,
catalogNumber, year, month, decimalLatitude,
decimalLongitude, elevation, depth, institutionCode,
collectionCode, hasGeospatialIssue, issue, q,
mediatype, limit, offset, **kwargs)<br>
251 'collectionCode': collectionCode,
'hasGeospatialIssue': hasGeospatialIssue,<br>
252 'issue': issue, 'q': q, 'mediatype':
mediatype, 'limit': limit,<br>
--> 253 'offset': offset}, **kwargs)<br>
254 return out<br>
<br>
/home/nils/.conda/envs/birdhouse/lib/python2.7/site-packages/pygbif/gbifutils.pyc
in gbif_GET(url, args, **kwargs)<br>
17 def gbif_GET(url, args, **kwargs):<br>
18 out = requests.get(url, params=args,
headers=make_ua(), **kwargs)<br>
---> 19 out.raise_for_status()<br>
20 stopifnot(out.headers['content-type'])<br>
21 return out.json()<br>
<br>
/home/nils/.conda/envs/birdhouse/lib/python2.7/site-packages/requests/models.pyc
in raise_for_status(self)<br>
842 <br>
843 if http_error_msg:<br>
--> 844 raise
HTTPError(http_error_msg, response=self)<br>
845 <br>
846 def close(self):<br>
<br>
HTTPError: 400 Client Error: Bad Request for url: <a
moz-do-not-send="true"
href="http://api.gbif.org/v1/occurrence/search?taxonKey=2882316&limit=200001&offset=0"
target="_blank"><a class="moz-txt-link-freetext" href="http://api.gbif.org/v1/occurrence/search?taxonKey=2882316&limit=200001&offset=0">http://api.gbif.org/v1/occurrence/search?taxonKey=2882316&limit=200001&offset=0</a></a><br>
<br>
In [73]: <br>
<br>
<br>
<div>On 02/06/2016 02:22, Scott Chamberlain wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Fixes to docs and fix to
download_get are up on Github now. Will push new
version to pypi soon. Let me know if some thing
still don't work for you after reinstalling from
github
<div><br>
</div>
<div>S</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Wed, Jun 1, 2016 at 4:51 PM
Nils Hempelmann <<a moz-do-not-send="true"
href="mailto:info@nilshempelmann.de"
target="_blank">info@nilshempelmann.de</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> Hi
Scott and Mauro <br>
<br>
Mauro sended me a snipped of code which
worked fine: <br>
<a moz-do-not-send="true"
href="https://github.com/bird-house/flyingpigeon/blob/develop/scripts/pygbif_occurence.py"
target="_blank">https://github.com/bird-house/flyingpigeon/blob/develop/scripts/pygbif_occurence.py</a><br>
<br>
I found the example here:<br>
<a moz-do-not-send="true"
href="https://github.com/maurobio/pygbif#occurrence-data"
target="_blank">https://github.com/maurobio/pygbif#occurrence-data</a><br>
<br>
and here: <br>
<a moz-do-not-send="true"
href="https://github.com/sckott/pygbif#occurrences-module"
target="_blank">https://github.com/sckott/pygbif#occurrences-module</a><br>
<br>
Thanks a lot very great. That enables a lot
;-) <br>
I ll keep you posted <br>
<br>
merci <br>
</div>
<div bgcolor="#FFFFFF" text="#000000"> Nils <br>
</div>
<div bgcolor="#FFFFFF" text="#000000"> <br>
<br>
<br>
<br>
<br>
<div>On 02/06/2016 01:45, Scott Chamberlain
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">And where are those example
from exactly? I don't see those examples
searching the repo (which includes all
docs).
<div><br>
</div>
<div>`pygbif.name_suggest` wouldn't work
because `name_suggest` is a method in
the `species` module, so you'd have to
do pygbif.species.name_suggest, or
`from pygbif import species`, then
`species.name_suggest`</div>
<div><br>
</div>
<div>Looks like <span>occ.get(taxonKey =
252408386)<span> is a documentation
bug, that should be `key` instead
of `taxonKey`, a copy paste error.
will fix that. </span></span></div>
<div><font color="#212121"
face="helvetica neue, helvetica,
arial, sans-serif"><br>
</font></div>
<div><font color="#212121"
face="helvetica neue, helvetica,
arial, sans-serif">The
`occ.download_get` call has a small
bug, will fix that</font></div>
<div><font color="#212121"
face="helvetica neue, helvetica,
arial, sans-serif"><br>
</font></div>
<div><font color="#212121"
face="helvetica neue, helvetica,
arial, sans-serif">All other calls
work for me<br>
</font>
<div><br>
</div>
<div>S </div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Wed, Jun 1, 2016 at
4:34 PM Scott Chamberlain <<a
moz-do-not-send="true"
href="mailto:myrmecocystus@gmail.com"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:myrmecocystus@gmail.com">myrmecocystus@gmail.com</a></a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div dir="ltr">Hi,
<div><br>
</div>
<div>What version of pygbif are you
on? And what version of Python><br>
<div><br>
</div>
<div>Best, Scott<br>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Wed, Jun 1, 2016
at 4:02 PM Nils Hempelmann <<a
moz-do-not-send="true"
href="mailto:info@nilshempelmann.de"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:info@nilshempelmann.de">info@nilshempelmann.de</a></a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">Hi Mauro
and Scott<br>
<br>
Was checking out pygbif. seems to
be a very useful tool.<br>
<br>
Can you help me with the syntax
(or forward me to the appropriate
person<br>
;-) )<br>
The given snippets of code are
outdated.<br>
<br>
I am basically just looking for
the occurrence coordinates:<br>
here is my first try :<br>
<br>
import pygbif<br>
occ =
pygbif.occurrences.search(scientificName='Fagus
sylvatica')<br>
occ['count']<br>
<br>
... and further? ;-)<br>
<br>
the examples in the docu are
throwing errors:<br>
<br>
key=
pygbif.name_suggest(q='Helianthus
annuus',rank='species')['key']<br>
pygbif.search(taxonKey=key[0]['key'],limit=2)<br>
<br>
from pygbif import occurrences as
occ occ.search(taxonKey = 3329049)<br>
occ.get(taxonKey = 252408386)
occ.count(isGeoreferenced = True)<br>
occ.download_list(user =
"sckott", limit = 5)<br>
occ.download_meta(key =
"0000099-140929101555934")<br>
occ.download_get("0000099-140929101555934")<br>
<br>
<br>
Thanks<br>
Nils<br>
<br>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br>
</div>
<br>
</body>
</html>