Dear Scott,
I must have missed your email in march, sorry for that. Let me still answer your important questions about the monthly downloads.
COL aims to produce a monthly release which we will keep at least for a year accessible via the API with all its features. Each release has its distinct datasetKey in ChecklistBank. There is no fixed day when we will issue the release. Moving to the new infrastructure at the end of last year caused a few teething issues which made us skip a few releases in january/february for example. We hope this will not happen again and you should see now regular monthly updates.
One of these releases will be tagged as an Annual release which from the API point of view is just the same as a monthly one. But it will not be removed from ChecklistBank and therefore you have long term access via the API to it. We plan to issue the next annual release in June.
Once a monthly release is deleted, we will still keep the data in various formats for download. But it will be gone from the database.
The download archive contains a DwC archive (YYYY-MM-DD_dwca.zip) and an ACEF archive (YYYY-MM-DD_acef.zip) right now, with the next May release we will also add a new ColDP archive to the supported formats. A native postgres dump is on our list too, but that is not straight forward as COL is just a small part of ChecklistBank and we will need to filter out the relevant bits.
Prior to December 2020 we only had DwC archives, but these has used some slightly different terms than we use today.
Note that we do not export a flattened classification (family, order, etc) at this stage, but we plan to add that back in again in a not too distant future. The same applies also to ColDP.
The ACEF archive is slightly special. We used it to transfer the data to the old systems and uses \N to represent NULL, which is a postgres specific convention. The files are proper CSV files with a header row and not tab separated. There is a short SQL script to load the ACEF files into a postgres database which also has DDL: https://github.com/CatalogueOfLife/backend/blob/master/webservice/src/main/…
The DwCA and ColDP archives on the other hand use just an empty string and also use a header row with the term names.
Files in both dwca and coldp are tab delimited and use the .tsv file extension. For ColDP we maintain a Postgres, MS Access and Excel schema, but that needs some small updates as we are about to freeze the format for a final & fixed 1st version.
Hope thats useful,
Markus
Hi,
I had been using the downloads available at https://download.catalogueoflife.org/col/monthly/ to construct a SQLite version of the database here https://github.com/sckott/col-sql to make it easier for users to use.
Two questions, the 2nd with many parts:
1. Will https://download.catalogueoflife.org/col/monthly/ continue to be updated every 2 or 3 months with a new database dump?
2. If the answer to (1) is yes: The format changed in the last database dump "2020-12-01_acef.zip".
a. The included file names changed, and file types changed from .tsv to .csv (although the data still appears to be tab-sep). Was it intended to change to comma-sep?
b. Will there be more changes to the monthly dump?
c. Will the release cycle be something predictable? Every 2 or 3 months?
d. The files have a lot of "\N" in them. Is this supposed to be a newline character? I've not seen a newline with a capital N.
e. Any schema to use for these various csv files?
Thanks!
Scott Chamberlain
Hi,
I had been using the downloads available at https://download.catalogueoflife.org/col/monthly/ to construct a SQLite version of the database here https://github.com/sckott/col-sql to make it easier for users to use.
Two questions, the 2nd with many parts:
1. Will https://download.catalogueoflife.org/col/monthly/ continue to be updated every 2 or 3 months with a new database dump?
2. If the answer to (1) is yes: The format changed in the last database dump "2020-12-01_acef.zip".
a. The included file names changed, and file types changed from .tsv to .csv (although the data still appears to be tab-sep). Was it intended to change to comma-sep?
b. Will there be more changes to the monthly dump?
c. Will the release cycle be something predictable? Every 2 or 3 months?
d. The files have a lot of "\N" in them. Is this supposed to be a newline character? I've not seen a newline with a capital N.
e. Any schema to use for these various csv files?
Thanks!
Scott Chamberlain
> Mailing list subscription confirmation notice for mailing list
> COL-Users
>
> We have received a request from 161.111.170.108 for subscription of
> your email address, "lado(a)rjb.csic.es", to the
> col-users(a)lists.gbif.org mailing list. To confirm that you want to be
> added to this mailing list, simply reply to this message, keeping the
> Subject: header intact. Or visit this web page:
>
> https://lists.gbif.org/mailman/confirm/col-users/c28b81fc03e038ae6a7f35897f…
>
>
> Or include the following line -- and only the following line -- in a
> message to col-users-request(a)lists.gbif.org:
>
> confirm c28b81fc03e038ae6a7f35897fe3ef1d9f4c7e02
>
> Note that simply sending a `reply' to this message should work from
> most mail readers, since that usually leaves the Subject: line in the
> right form (additional "Re:" text in the Subject: is okay).
>
> If you do not wish to be subscribed to this list, please simply
> disregard this message. If you think you are being maliciously
> subscribed to the list, or have any other questions, send them to
> col-users-owner(a)lists.gbif.org.
--
======================================================================
Dr. Carlos Lado phone: +34 91 420 3017
Real Jardín Botánico, CSIC. fax: +34 91 420 0157
Plaza de Murillo, 2 e-mail: lado(a)rjb.csic.es
E-28014 Madrid http://www.rjb.csic.es
SPAIN
======================================================================
Dear COL users,
we have just put live a new November edition of the Catalogue of Life. This monthly update is special for a variety of reasons:
1) It is only available through the new infrastructure, i.e. the COL Portal, API and COL ChecklistBank.
We have stopped converting the data back into the legacy database structure driving the old Annual Checklist (AC) interface.
2) It uses a new set of identifiers. These are still preliminary and under review in this release and will change with the next release due in a few weeks.
With that next release though, they will remain stable and we will then also provide a mapping from the previous hash based identifers.
3) It is exposed for the first time through the new COL Portal which is getting closer to have all major features in place to replace the old AC interface.
It also hosts and gives dynamic access to all previous annual editions with their original interface at that time.
Please consider this still a beta version. There are still many things to improve, but we think this is a first useful version with all essential features in place and we are happy to receive feedback on bugs, missing features or other kind of enhancements. With the next major release we will also move the new software back to the old and final domain catalogueoflife.org<http://catalogueoflife.org> and deprecate catalogue.life.
Best,
Markus
COL Portal:
https://www.catalogue.life/
COL ChecklistBank
https://data.catalogue.life/
COL API:
https://api.catalogue.life/https://api.catalogue.life/dataset/3LR
Feedback:
feedback(a)catalogueoflife.org<mailto:feedback@catalogueoflife.org>
--
Markus Döring
Software Developer
Global Biodiversity Information Facility (GBIF)
mdoering(a)gbif.org<mailto:mdoering@gbif.org>
http://www.gbif.org
