<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Dear GBIF API users,<br>
</p>
<p dir="auto"><i>You might prefer to read this email on either
GitHub or the community forum, as the formatting is probably
better:</i></p>
<i> </i>
<p dir="auto"><i>GitHub issue discussion:
<a class="moz-txt-link-freetext" href="https://github.com/gbif/gbif-api/issues/4#issuecomment-1735378954">https://github.com/gbif/gbif-api/issues/4#issuecomment-1735378954</a></i></p>
<i> </i>
<p dir="auto"><i>Community forum discussion:
<a class="moz-txt-link-freetext"
href="https://discourse.gbif.org/t/gbif-api-supporting-ranges-in-occurrence-eventdate/3804">https://discourse.gbif.org/t/gbif-api-supporting-ranges-in-occurrence-eventdate/3804</a></i></p>
<p><b>Event dates — upcoming API change</b><br>
<br>
Early this year we announced a plan to change the way we handle
the "eventDate" Darwin Core term. Date ranges formatted using the
ISO 8601 standard, recommended by Darwin Core, will retain their
meaning, and the API will return values like "2000-05" or
"2007-11-13/2007-11-15", rather than the current behaviour of
changing these values to "2000-05-01" and "2007-11-13".<br>
<br>
These changes are now visible on GBIF's test system,
GBIF-UAT.org. To allow time for you to test this change against
any existing software and scripts you have, we will not implement
these changes on GBIF.org before early November.<br>
<br>
<b>API users</b><br>
<br>
Users of the occurrence API will need to decide how to handle an
eventDate like "1880/1889", "1910", "2000-05", "1999-11/2000-03",
"2007-11-13/2007-11-15" or
"2023-09-22T05:17:10/2023-09-22T12:17:10" — taking the earliest,
latest or middle value, randomizing within the range, excluding
them etc. To make parsing easier ranges will always be formatted
using the full form and never abbreviated — always
"2007-11-13/2007-11-15" and never "2007-11-13/15".<br>
<br>
It may be easier to use the individual "year", "month" and "day"
fields, which will be present if the year/month/day is constant
for the whole range of the eventDate —
eventDate=2010-11-25/2010-12-03 will have year=2010, month=NULL,
day=NULL as only the year is constant. (However, note a date like
2022-12-31/2023-01-01 covers just 2 days, but as is spans two
different years the "year" field will be blank.)<br>
<br>
When searching using a range (e.g. eventDate=2005-01,2005-03) only
occurrences with eventDates *entirely within* the range will be
returned.<br>
<br>
<b>Download users</b><br>
<br>
The "eventDate" column in CSV, Darwin Core and Parquet (cloud
snapshot) downloads will contain the same value as in the API, for
example "2023-09-22T12:17:10", "2023-09-22", "1880/1889", "1910",
"2000-05", "1999-11/2000-03", "2007-11-13/2007-11-15" or
"2023-09-22T05:17:10/2023-09-22T12:17:10".<br>
<br>
As with the search API, when filtering using a range (e.g.
eventDate=2005-01,2005-03) only occurrences with eventDates
*entirely within* the range will be returned<br>
<br>
<b>Data interpretation (for data publishers)</b><br>
<br>
Eight Darwin Core terms record information on when an occurrence
was collected or observed:<br>
<br>
- year<br>
- month<br>
- day<br>
- eventDate<br>
- eventTime<br>
- startDayOfYear<br>
- endDayOfYear<br>
- verbatimEventDate<br>
<br>
Some records will have conflicting information in these fields.
Detailed documentation on how we handle the various cases is being
prepared, but the general approach is to remove parts of the date
that conflict, adding a RECORDED_DATE_MISMATCH issue in this
case. For example, "eventDate=2005-06-01", "year=2005", "month=6"
and "day=NULL" would have eventDate changed to "2005-06" and the
issue added.<br>
<br>
Occurrences published with only one/some fields will have the
other fields filled in automatically, where possible. We will not
add an issue flag for this.</p>
<p>All existing datasets will be reprocessed with the new algorithms
as the change to the API is made for GBIF.org.<br>
</p>
<p><b>Example dataset</b><br>
<br>
A dataset of test occurrences is here:
<a class="moz-txt-link-freetext" href="https://www.gbif-uat.org/occurrence/search?dataset_key=d6167827-973d-429a-a00c-8ea294d62d80">https://www.gbif-uat.org/occurrence/search?dataset_key=d6167827-973d-429a-a00c-8ea294d62d80</a>
providing many examples of consistent and conflicting event date
fields. The scientificName is set to a summary of what the
eventDate is, and the eventRemarks field has more explanation.<br>
<br>
<b>Feedback</b><br>
<br>
Feedback is welcome on the GitHub issue, here on the mailing list,
or on the Discourse forum<br>
<br>
Thanks,<br>
<br>
Matt</p>
<div class="moz-cite-prefix">
<p dir="auto">GitHub issue discussion:
<a class="moz-txt-link-freetext" href="https://github.com/gbif/gbif-api/issues/4#issuecomment-1735378954">https://github.com/gbif/gbif-api/issues/4#issuecomment-1735378954</a></p>
<p dir="auto">Community forum discussion:
<a class="moz-txt-link-freetext"
href="https://discourse.gbif.org/t/gbif-api-supporting-ranges-in-occurrence-eventdate/3804">https://discourse.gbif.org/t/gbif-api-supporting-ranges-in-occurrence-eventdate/3804</a></p>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">On 17/01/2023 15:28, Matthew Blissett
via API-users wrote:<br>
</div>
<blockquote type="cite"
cite="mid:0bf06246-d25c-de16-fbb4-47e6b785effc@gbif.org">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<p dir="auto">Dear GBIF API users,</p>
<p dir="auto"><i>You might prefer to read this email on either
GitHub or the community forum, as the formatting is probably
better:</i></p>
<i> </i>
<p dir="auto"><i>GitHub issue discussion: <a
class="moz-txt-link-freetext"
href="https://github.com/gbif/gbif-api/issues/4#issuecomment-1385497157"
moz-do-not-send="true">https://github.com/gbif/gbif-api/issues/4#issuecomment-1385497157</a></i></p>
<i> </i>
<p dir="auto"><i>Community forum discussion:
<a class="moz-txt-link-freetext"
href="https://discourse.gbif.org/t/gbif-api-supporting-ranges-in-occurrence-eventdate/3804"
moz-do-not-send="true">https://discourse.gbif.org/t/gbif-api-supporting-ranges-in-occurrence-eventdate/3804</a></i></p>
<p><br>
</p>
<p dir="auto">A longstanding issue with the GBIF API is the
interpretation and formatting of the Darwin Core term
"eventDate".</p>
<p dir="auto"><strong>Summary: instead of GBIF changing published
<code>eventDate</code> values like <code>2009-03-18/2009-04-13</code>
and <code>2010</code> to <code>2009-03-18</code> and <code>2010-01-01</code>
respectively, we propose returning the values <code>2009-03-18/2009-04-13</code>
and <code>2010</code> in the occurrence API and in downloads.
Existing code/scripts that use the <code>eventDate</code>
value may need to be updated.</strong></p>
The recommended best practise for the term is "use a date that
conforms to ISO 8601-1:2019" (see <a
href="https://dwc.tdwg.org/terms/#dwc:eventDate" rel="nofollow"
class="moz-txt-link-freetext" moz-do-not-send="true">https://dwc.tdwg.org/terms/#dwc:eventDate</a>).
<p dir="auto">ISO 8601-1:2019 supports date ranges, and some
publishers provide these. Examples are <code>2000-05</code>, or
<code>2007-11-13/2007-11-15</code>. GBIF's current
interpretation changes date ranges like this to the first
possible day in the range (<code>2000-05-01</code> and <code>2007-11-13</code>).</p>
<p dir="auto">At least 64 million occurrences are affected.</p>
<h2 dir="auto"><a id="user-content-change-to-date-interpretation"
class="anchor" aria-hidden="true"
href="https://gist.github.com/MattBlissett/ff06599559ce86302a6e84d2e3e605ec#change-to-date-interpretation"
moz-do-not-send="true"><svg class="octicon octicon-link"
viewBox="0 0 16 16" width="16" height="16"
aria-hidden="true"></svg></a></h2>
<h2 dir="auto">Change to date interpretation</h2>
<p dir="auto">We propose changing the eventDate field in the GBIF
API to support ISO 8601-1 date ranges. A range will be returned
where one was provided by the publisher, either directly as a
range in the <code>eventDate</code> field, or through a
combination of the <code>year</code>, <code>month</code>, <code>day</code>,
<code>startDayOfYear</code> and <code>endDayOfYear</code>
fields.</p>
<p dir="auto">The data quality checks on dates will be improved to
check for consistency between these fields: <code>eventDate</code>,
<code>year</code>, <code>month</code>, <code>day</code>, <code>startDayOfYear</code>
and <code>endDayOfYear</code>. These fields will only be
populated if they are constant for the whole range of dates — a
range spanning several days in January 2020 will have <code>year=2020</code>,
<code>month=January</code> and <code>day=(Blank)</code>.</p>
<p dir="auto"><code>startDayOfYear</code> and <code>endDayOfYear</code>
will also be present if the range is accurate to days.</p>
<p dir="auto">Examples:</p>
<table>
<thead> <tr>
<th>published event date</th>
<th>intepreted eventDate</th>
<th>int. year</th>
<th>int. month</th>
<th>int. day</th>
<th>int. sdoy</th>
<th>int. edoy</th>
</tr>
</thead> <tbody>
<tr>
<td>2023-01-13</td>
<td>2023-01-13</td>
<td>2023</td>
<td>1</td>
<td>13</td>
<td>13</td>
<td>13</td>
</tr>
<tr>
<td>2023-01</td>
<td>2023-01</td>
<td>2023</td>
<td>1</td>
<td><br>
</td>
<td><br>
</td>
<td><br>
</td>
</tr>
<tr>
<td>2023</td>
<td>2023</td>
<td>2023</td>
<td><br>
</td>
<td><br>
</td>
<td><br>
</td>
<td><br>
</td>
</tr>
<tr>
<td>2023-01-13/2023-01-14</td>
<td>2023-01-13/2023-01-14</td>
<td>2023</td>
<td>1</td>
<td><br>
</td>
<td>13</td>
<td>14</td>
</tr>
<tr>
<td>2023-01-13/14</td>
<td>2023-01-13/14</td>
<td>2023</td>
<td>1</td>
<td><br>
</td>
<td>13</td>
<td>14</td>
</tr>
<tr>
<td>2023-01/2023-02</td>
<td>2023-01/2023-02</td>
<td>2023</td>
<td><br>
</td>
<td><br>
</td>
<td><br>
</td>
<td><br>
</td>
</tr>
<tr>
<td>2023-01/02</td>
<td>2023-01/02</td>
<td>2023</td>
<td><br>
</td>
<td><br>
</td>
<td><br>
</td>
<td><br>
</td>
</tr>
<tr>
<td>2023/2024</td>
<td>2023/2024</td>
<td><br>
</td>
<td><br>
</td>
<td><br>
</td>
<td><br>
</td>
<td><br>
</td>
</tr>
<tr>
<td>2023-01-01/2023-12-31</td>
<td>2023-01-01/2023-12-31</td>
<td>2023</td>
<td><br>
</td>
<td><br>
</td>
<td>1</td>
<td>365</td>
</tr>
</tbody>
</table>
<p dir="auto">Other cases where we can unambiguously determine a
date or date range will also be handled, for example a record
with a <code>year</code> and <code>month</code> but no <code>eventDate</code>,
or non-ISO dates like <code>January 2023</code>.</p>
<h3 dir="auto"><a id="user-content-api-example" class="anchor"
aria-hidden="true"
href="https://gist.github.com/MattBlissett/ff06599559ce86302a6e84d2e3e605ec#api-example"
moz-do-not-send="true"><svg class="octicon octicon-link"
viewBox="0 0 16 16" width="16" height="16"
aria-hidden="true"></svg></a></h3>
<h3 dir="auto">API example:</h3>
<p dir="auto"><a
href="https://api.gbif.org/v1/occurrence/1234530937"
rel="nofollow" moz-do-not-send="true">This record</a> (<a
href="https://www.gbif.org/occurrence/1234530937"
rel="nofollow" moz-do-not-send="true">portal link</a>) is
published with <code>eventDate=2009-03-18/2009-04-13</code>, <code>year=2009</code>,
<code>month=3</code>, <code>day=18</code>. We currently change
the <code>eventDate</code>:</p>
<div class="highlight highlight-source-json" dir="auto">
<pre><span class="pl-ent">"year"</span>: <span class="pl-c1">2009</span>,
<span class="pl-ent">"month"</span>: <span class="pl-c1">3</span>,
<span class="pl-ent">"day"</span>: <span class="pl-c1">18</span>,
<span class="pl-ent">"eventDate"</span>: <span class="pl-s"><span
class="pl-pds">"</span>2009-03-18T00:00:00<span class="pl-pds">"</span></span>,</pre>
</div>
<p dir="auto">With this proposal, we would preserve the <code>eventDate</code>
but remove <code>day</code>, as it the event crosses several
days:</p>
<div class="highlight highlight-source-json" dir="auto">
<pre><span class="pl-ent">"year"</span>: <span class="pl-c1">2009</span>,
<span class="pl-ent">"month"</span>: <span class="pl-c1">3</span>,
<span class="pl-ent">"eventDate"</span>: <span class="pl-s"><span
class="pl-pds">"</span>2009-03-18/2009-04-13<span class="pl-pds">"</span></span>,</pre>
</div>
<p dir="auto"><a
href="https://api.gbif.org/v1/occurrence/2382954724"
rel="nofollow" moz-do-not-send="true">This record</a> (<a
href="https://www.gbif.org/occurrence/2382954724"
rel="nofollow" moz-do-not-send="true">portal link</a>) is
published with <code>eventDate=2019-04-06T20:00:00/2019-04-10T05:00:00</code>
and no separate <code>day</code>, <code>month</code> or <code>year</code>
values. Currently, we process it to this:</p>
<div class="highlight highlight-source-json" dir="auto">
<pre><span class="pl-ent">"year"</span>: <span class="pl-c1">2019</span>,
<span class="pl-ent">"month"</span>: <span class="pl-c1">4</span>,
<span class="pl-ent">"day"</span>: <span class="pl-c1">6</span>,
<span class="pl-ent">"eventDate"</span>: <span class="pl-s"><span
class="pl-pds">"</span>2019-04-06T20:00:00<span class="pl-pds">"</span></span>,</pre>
</div>
<p dir="auto">Instead, we propose returning this:</p>
<div class="highlight highlight-source-json" dir="auto">
<pre><span class="pl-ent">"year"</span>: <span class="pl-c1">2019</span>,
<span class="pl-ent">"month"</span>: <span class="pl-c1">4</span>,
<span class="pl-ent">"eventDate"</span>: <span class="pl-s"><span
class="pl-pds">"</span>2019-04-06T20:00:00/2019-04-10T05:00:00<span
class="pl-pds">"</span></span>,
<span class="pl-ent">"startDayOfYear"</span>: <span class="pl-c1">96</span>,
<span class="pl-ent">"endDayOfYear"</span>: <span class="pl-c1">100</span>,</pre>
</div>
<h2 dir="auto"><a id="user-content-searching" class="anchor"
aria-hidden="true"
href="https://gist.github.com/MattBlissett/ff06599559ce86302a6e84d2e3e605ec#searching"
moz-do-not-send="true"><svg class="octicon octicon-link"
viewBox="0 0 16 16" width="16" height="16"
aria-hidden="true"></svg></a></h2>
<h2 dir="auto">Searching</h2>
<p dir="auto">The search and download APIs will be affected by
this change.</p>
<p dir="auto">Occurrences will be returned if the occurrence
date/date range is <strong>completely within</strong> the query
date or date range.</p>
<pre><code>Search: eventDate=2023-01-11
Record: eventDate=2023-01-11 -- included
Record: eventDate=2023-01 -- EXCLUDED
Record: eventDate=2023-01-11/12 -- EXCLUDED
Search: eventDate=2023-01-11,2023-01-12
Record: eventDate=2023-01-11 -- included
Record: eventDate=2023-01 -- EXCLUDED
Record: eventDate=2023-01-11/12 -- included
Search: eventDate=*,2023-01 (meaning "Before end of January 2023")
Record: eventDate=2023-01-11 -- included
Record: eventDate=2023-01 -- included
Record: eventDate=2023-01-11/12 -- included
Search: eventDate=2023-01,2023-01 (meaning "After start of January 2023 AND before end of January 2023")
Search: eventDate=2023-01 (same meaning)
Record: eventDate=2023-01-11 -- included
Record: eventDate=2023-01 -- included
Record: eventDate=2023-01-11/12 -- included
</code></pre>
<p dir="auto">This implementation will avoid returning occurrences
with eventDates like "2010/2021" in many queries. (There are
millions of occurrences with large ranges like this.)</p>
<h2 dir="auto"><a id="user-content-density-maps" class="anchor"
aria-hidden="true"
href="https://gist.github.com/MattBlissett/ff06599559ce86302a6e84d2e3e605ec#density-maps"
moz-do-not-send="true"><svg class="octicon octicon-link"
viewBox="0 0 16 16" width="16" height="16"
aria-hidden="true"></svg></a></h2>
<h2 dir="auto">Density maps</h2>
<p dir="auto">There is a year filter for the density/pixel maps.
An occurrence from 2023-01 will be included, but an occurrence
with an eventDate spanning more than a single year (like
2022-13-31/2023-01-01) will no longer be included.</p>
<h2 dir="auto"><a
id="user-content-quarterly-analytics-globalregional-trends"
class="anchor" aria-hidden="true"
href="https://gist.github.com/MattBlissett/ff06599559ce86302a6e84d2e3e605ec#quarterly-analytics-globalregional-trends"
moz-do-not-send="true"><svg class="octicon octicon-link"
viewBox="0 0 16 16" width="16" height="16"
aria-hidden="true"></svg></a></h2>
<h2 dir="auto">Quarterly analytics, global/regional trends</h2>
<p dir="auto">The quarterly analytics include calculations based
on the individual dwc:year, dwc:month and dwc:day fields. The
statistics will be affected where these values change or become
blank.</p>
<h2 dir="auto"><a id="user-content-rgbif-pygbif" class="anchor"
aria-hidden="true"
href="https://gist.github.com/MattBlissett/ff06599559ce86302a6e84d2e3e605ec#rgbif-pygbif"
moz-do-not-send="true"><svg class="octicon octicon-link"
viewBox="0 0 16 16" width="16" height="16"
aria-hidden="true"></svg></a></h2>
<h2 dir="auto">rGBIF, PyGBIF</h2>
<p dir="auto">Both libraries will be updated as necessary to
support eventDate values containing a date range.</p>
<h2 dir="auto"><a id="user-content-feedback" class="anchor"
aria-hidden="true"
href="https://gist.github.com/MattBlissett/ff06599559ce86302a6e84d2e3e605ec#feedback"
moz-do-not-send="true"><svg class="octicon octicon-link"
viewBox="0 0 16 16" width="16" height="16"
aria-hidden="true"></svg></a></h2>
<h2 dir="auto">Feedback</h2>
<p dir="auto">We have delayed addressing this issue for a long
time, primarily due to concerns about changing the existing
behaviour of the API. However, it's also one of the most
frequently requested improvements to GBIF's interpretation.</p>
<p dir="auto">If you are aware of software or systems which would
have problems adapting to the proposed change, please let us
know, either on this mailing list, the GitHub issue, the
community forum or by email to me.</p>
<p dir="auto">We will alert users in the same places when the
change is ready to be tested on the test system at
api.gbif-uat.org, and when the change is to be made live on
api.gbif.org.</p>
<p dir="auto">Thank you,</p>
<p dir="auto">Matt</p>
<p dir="auto">GitHub issue discussion: <a
class="moz-txt-link-freetext"
href="https://github.com/gbif/gbif-api/issues/4#issuecomment-1385497157"
moz-do-not-send="true">https://github.com/gbif/gbif-api/issues/4#issuecomment-1385497157</a></p>
<p dir="auto">Community forum discussion:
<a class="moz-txt-link-freetext"
href="https://discourse.gbif.org/t/gbif-api-supporting-ranges-in-occurrence-eventdate/3804"
moz-do-not-send="true">https://discourse.gbif.org/t/gbif-api-supporting-ranges-in-occurrence-eventdate/3804</a><br>
</p>
<br>
<fieldset class="moz-mime-attachment-header"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
API-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:API-users@lists.gbif.org">API-users@lists.gbif.org</a>
<a class="moz-txt-link-freetext" href="https://lists.gbif.org/mailman/listinfo/api-users">https://lists.gbif.org/mailman/listinfo/api-users</a>
</pre>
</blockquote>
</body>
</html>