<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
</head>
<body>
<p dir="auto">Dear GBIF API users,</p>
<p dir="auto"><i>You might prefer to read this email on either
GitHub or the community forum, as the formatting is probably
better:</i></p>
<i>
</i>
<p dir="auto"><i>GitHub issue discussion:
<a class="moz-txt-link-freetext" href="https://github.com/gbif/gbif-api/issues/4#issuecomment-1385497157">https://github.com/gbif/gbif-api/issues/4#issuecomment-1385497157</a></i></p>
<i>
</i>
<p dir="auto"><i>Community forum discussion:
<a class="moz-txt-link-freetext" href="https://discourse.gbif.org/t/gbif-api-supporting-ranges-in-occurrence-eventdate/3804">https://discourse.gbif.org/t/gbif-api-supporting-ranges-in-occurrence-eventdate/3804</a></i></p>
<p><br>
</p>
<p dir="auto">A longstanding issue with the GBIF API is the
interpretation and formatting of the Darwin Core term "eventDate".</p>
<p dir="auto"><strong>Summary: instead of GBIF changing published <code>eventDate</code>
values like <code>2009-03-18/2009-04-13</code> and <code>2010</code>
to <code>2009-03-18</code> and <code>2010-01-01</code>
respectively, we propose returning the values <code>2009-03-18/2009-04-13</code>
and <code>2010</code> in the occurrence API and in downloads.
Existing code/scripts that use the <code>eventDate</code> value
may need to be updated.</strong></p>
The recommended best practise for the term is "use a date that
conforms to ISO 8601-1:2019" (see <a
href="https://dwc.tdwg.org/terms/#dwc:eventDate" rel="nofollow"
class="moz-txt-link-freetext">https://dwc.tdwg.org/terms/#dwc:eventDate</a>).
<p dir="auto">ISO 8601-1:2019 supports date ranges, and some
publishers provide these. Examples are <code>2000-05</code>, or <code>2007-11-13/2007-11-15</code>.
GBIF's current interpretation changes date ranges like this to the
first possible day in the range (<code>2000-05-01</code> and <code>2007-11-13</code>).</p>
<p dir="auto">At least 64 million occurrences are affected.</p>
<h2 dir="auto"><a id="user-content-change-to-date-interpretation"
class="anchor" aria-hidden="true"
href="https://gist.github.com/MattBlissett/ff06599559ce86302a6e84d2e3e605ec#change-to-date-interpretation"><svg
class="octicon octicon-link" viewBox="0 0 16 16" width="16"
height="16" aria-hidden="true"></svg></a></h2>
<h2 dir="auto">Change to date interpretation</h2>
<p dir="auto">We propose changing the eventDate field in the GBIF
API to support ISO 8601-1 date ranges. A range will be returned
where one was provided by the publisher, either directly as a
range in the <code>eventDate</code> field, or through a
combination of the <code>year</code>, <code>month</code>, <code>day</code>,
<code>startDayOfYear</code> and <code>endDayOfYear</code> fields.</p>
<p dir="auto">The data quality checks on dates will be improved to
check for consistency between these fields: <code>eventDate</code>,
<code>year</code>, <code>month</code>, <code>day</code>, <code>startDayOfYear</code>
and <code>endDayOfYear</code>. These fields will only be
populated if they are constant for the whole range of dates — a
range spanning several days in January 2020 will have <code>year=2020</code>,
<code>month=January</code> and <code>day=(Blank)</code>.</p>
<p dir="auto"><code>startDayOfYear</code> and <code>endDayOfYear</code>
will also be present if the range is accurate to days.</p>
<p dir="auto">Examples:</p>
<table>
<thead>
<tr>
<th>published event date</th>
<th>intepreted eventDate</th>
<th>int. year</th>
<th>int. month</th>
<th>int. day</th>
<th>int. sdoy</th>
<th>int. edoy</th>
</tr>
</thead>
<tbody>
<tr>
<td>2023-01-13</td>
<td>2023-01-13</td>
<td>2023</td>
<td>1</td>
<td>13</td>
<td>13</td>
<td>13</td>
</tr>
<tr>
<td>2023-01</td>
<td>2023-01</td>
<td>2023</td>
<td>1</td>
<td><br>
</td>
<td><br>
</td>
<td><br>
</td>
</tr>
<tr>
<td>2023</td>
<td>2023</td>
<td>2023</td>
<td><br>
</td>
<td><br>
</td>
<td><br>
</td>
<td><br>
</td>
</tr>
<tr>
<td>2023-01-13/2023-01-14</td>
<td>2023-01-13/2023-01-14</td>
<td>2023</td>
<td>1</td>
<td><br>
</td>
<td>13</td>
<td>14</td>
</tr>
<tr>
<td>2023-01-13/14</td>
<td>2023-01-13/14</td>
<td>2023</td>
<td>1</td>
<td><br>
</td>
<td>13</td>
<td>14</td>
</tr>
<tr>
<td>2023-01/2023-02</td>
<td>2023-01/2023-02</td>
<td>2023</td>
<td><br>
</td>
<td><br>
</td>
<td><br>
</td>
<td><br>
</td>
</tr>
<tr>
<td>2023-01/02</td>
<td>2023-01/02</td>
<td>2023</td>
<td><br>
</td>
<td><br>
</td>
<td><br>
</td>
<td><br>
</td>
</tr>
<tr>
<td>2023/2024</td>
<td>2023/2024</td>
<td><br>
</td>
<td><br>
</td>
<td><br>
</td>
<td><br>
</td>
<td><br>
</td>
</tr>
<tr>
<td>2023-01-01/2023-12-31</td>
<td>2023-01-01/2023-12-31</td>
<td>2023</td>
<td><br>
</td>
<td><br>
</td>
<td>1</td>
<td>365</td>
</tr>
</tbody>
</table>
<p dir="auto">Other cases where we can unambiguously determine a
date or date range will also be handled, for example a record with
a <code>year</code> and <code>month</code> but no <code>eventDate</code>,
or non-ISO dates like <code>January 2023</code>.</p>
<h3 dir="auto"><a id="user-content-api-example" class="anchor"
aria-hidden="true"
href="https://gist.github.com/MattBlissett/ff06599559ce86302a6e84d2e3e605ec#api-example"><svg
class="octicon octicon-link" viewBox="0 0 16 16" width="16"
height="16" aria-hidden="true"></svg></a></h3>
<h3 dir="auto">API example:</h3>
<p dir="auto"><a
href="https://api.gbif.org/v1/occurrence/1234530937"
rel="nofollow">This record</a> (<a
href="https://www.gbif.org/occurrence/1234530937" rel="nofollow">portal
link</a>) is published with <code>eventDate=2009-03-18/2009-04-13</code>,
<code>year=2009</code>, <code>month=3</code>, <code>day=18</code>.
We currently change the <code>eventDate</code>:</p>
<div class="highlight highlight-source-json" dir="auto">
<pre><span class="pl-ent">"year"</span>: <span class="pl-c1">2009</span>,
<span class="pl-ent">"month"</span>: <span class="pl-c1">3</span>,
<span class="pl-ent">"day"</span>: <span class="pl-c1">18</span>,
<span class="pl-ent">"eventDate"</span>: <span class="pl-s"><span class="pl-pds">"</span>2009-03-18T00:00:00<span class="pl-pds">"</span></span>,</pre>
</div>
<p dir="auto">With this proposal, we would preserve the <code>eventDate</code>
but remove <code>day</code>, as it the event crosses several
days:</p>
<div class="highlight highlight-source-json" dir="auto">
<pre><span class="pl-ent">"year"</span>: <span class="pl-c1">2009</span>,
<span class="pl-ent">"month"</span>: <span class="pl-c1">3</span>,
<span class="pl-ent">"eventDate"</span>: <span class="pl-s"><span class="pl-pds">"</span>2009-03-18/2009-04-13<span class="pl-pds">"</span></span>,</pre>
</div>
<p dir="auto"><a
href="https://api.gbif.org/v1/occurrence/2382954724"
rel="nofollow">This record</a> (<a
href="https://www.gbif.org/occurrence/2382954724" rel="nofollow">portal
link</a>) is published with <code>eventDate=2019-04-06T20:00:00/2019-04-10T05:00:00</code>
and no separate <code>day</code>, <code>month</code> or <code>year</code>
values. Currently, we process it to this:</p>
<div class="highlight highlight-source-json" dir="auto">
<pre><span class="pl-ent">"year"</span>: <span class="pl-c1">2019</span>,
<span class="pl-ent">"month"</span>: <span class="pl-c1">4</span>,
<span class="pl-ent">"day"</span>: <span class="pl-c1">6</span>,
<span class="pl-ent">"eventDate"</span>: <span class="pl-s"><span class="pl-pds">"</span>2019-04-06T20:00:00<span class="pl-pds">"</span></span>,</pre>
</div>
<p dir="auto">Instead, we propose returning this:</p>
<div class="highlight highlight-source-json" dir="auto">
<pre><span class="pl-ent">"year"</span>: <span class="pl-c1">2019</span>,
<span class="pl-ent">"month"</span>: <span class="pl-c1">4</span>,
<span class="pl-ent">"eventDate"</span>: <span class="pl-s"><span class="pl-pds">"</span>2019-04-06T20:00:00/2019-04-10T05:00:00<span class="pl-pds">"</span></span>,
<span class="pl-ent">"startDayOfYear"</span>: <span class="pl-c1">96</span>,
<span class="pl-ent">"endDayOfYear"</span>: <span class="pl-c1">100</span>,</pre>
</div>
<h2 dir="auto"><a id="user-content-searching" class="anchor"
aria-hidden="true"
href="https://gist.github.com/MattBlissett/ff06599559ce86302a6e84d2e3e605ec#searching"><svg
class="octicon octicon-link" viewBox="0 0 16 16" width="16"
height="16" aria-hidden="true"></svg></a></h2>
<h2 dir="auto">Searching</h2>
<p dir="auto">The search and download APIs will be affected by this
change.</p>
<p dir="auto">Occurrences will be returned if the occurrence
date/date range is <strong>completely within</strong> the query
date or date range.</p>
<pre><code>Search: eventDate=2023-01-11
Record: eventDate=2023-01-11 -- included
Record: eventDate=2023-01 -- EXCLUDED
Record: eventDate=2023-01-11/12 -- EXCLUDED
Search: eventDate=2023-01-11,2023-01-12
Record: eventDate=2023-01-11 -- included
Record: eventDate=2023-01 -- EXCLUDED
Record: eventDate=2023-01-11/12 -- included
Search: eventDate=*,2023-01 (meaning "Before end of January 2023")
Record: eventDate=2023-01-11 -- included
Record: eventDate=2023-01 -- included
Record: eventDate=2023-01-11/12 -- included
Search: eventDate=2023-01,2023-01 (meaning "After start of January 2023 AND before end of January 2023")
Search: eventDate=2023-01 (same meaning)
Record: eventDate=2023-01-11 -- included
Record: eventDate=2023-01 -- included
Record: eventDate=2023-01-11/12 -- included
</code></pre>
<p dir="auto">This implementation will avoid returning occurrences
with eventDates like "2010/2021" in many queries. (There are
millions of occurrences with large ranges like this.)</p>
<h2 dir="auto"><a id="user-content-density-maps" class="anchor"
aria-hidden="true"
href="https://gist.github.com/MattBlissett/ff06599559ce86302a6e84d2e3e605ec#density-maps"><svg
class="octicon octicon-link" viewBox="0 0 16 16" width="16"
height="16" aria-hidden="true"></svg></a></h2>
<h2 dir="auto">Density maps</h2>
<p dir="auto">There is a year filter for the density/pixel maps. An
occurrence from 2023-01 will be included, but an occurrence with
an eventDate spanning more than a single year (like
2022-13-31/2023-01-01) will no longer be included.</p>
<h2 dir="auto"><a
id="user-content-quarterly-analytics-globalregional-trends"
class="anchor" aria-hidden="true"
href="https://gist.github.com/MattBlissett/ff06599559ce86302a6e84d2e3e605ec#quarterly-analytics-globalregional-trends"><svg
class="octicon octicon-link" viewBox="0 0 16 16" width="16"
height="16" aria-hidden="true"></svg></a></h2>
<h2 dir="auto">Quarterly analytics, global/regional trends</h2>
<p dir="auto">The quarterly analytics include calculations based on
the individual dwc:year, dwc:month and dwc:day fields. The
statistics will be affected where these values change or become
blank.</p>
<h2 dir="auto"><a id="user-content-rgbif-pygbif" class="anchor"
aria-hidden="true"
href="https://gist.github.com/MattBlissett/ff06599559ce86302a6e84d2e3e605ec#rgbif-pygbif"><svg
class="octicon octicon-link" viewBox="0 0 16 16" width="16"
height="16" aria-hidden="true"></svg></a></h2>
<h2 dir="auto">rGBIF, PyGBIF</h2>
<p dir="auto">Both libraries will be updated as necessary to support
eventDate values containing a date range.</p>
<h2 dir="auto"><a id="user-content-feedback" class="anchor"
aria-hidden="true"
href="https://gist.github.com/MattBlissett/ff06599559ce86302a6e84d2e3e605ec#feedback"><svg
class="octicon octicon-link" viewBox="0 0 16 16" width="16"
height="16" aria-hidden="true"></svg></a></h2>
<h2 dir="auto">Feedback</h2>
<p dir="auto">We have delayed addressing this issue for a long time,
primarily due to concerns about changing the existing behaviour of
the API. However, it's also one of the most frequently requested
improvements to GBIF's interpretation.</p>
<p dir="auto">If you are aware of software or systems which would
have problems adapting to the proposed change, please let us know,
either on this mailing list, the GitHub issue, the community forum
or by email to me.</p>
<p dir="auto">We will alert users in the same places when the change
is ready to be tested on the test system at api.gbif-uat.org, and
when the change is to be made live on api.gbif.org.</p>
<p dir="auto">Thank you,</p>
<p dir="auto">Matt</p>
<p dir="auto">GitHub issue discussion:
<a class="moz-txt-link-freetext" href="https://github.com/gbif/gbif-api/issues/4#issuecomment-1385497157">https://github.com/gbif/gbif-api/issues/4#issuecomment-1385497157</a></p>
<p dir="auto">Community forum discussion:
<a class="moz-txt-link-freetext" href="https://discourse.gbif.org/t/gbif-api-supporting-ranges-in-occurrence-eventdate/3804">https://discourse.gbif.org/t/gbif-api-supporting-ranges-in-occurrence-eventdate/3804</a><br>
</p>
<p></p>
</body>
</html>