- COL-Users - lists.gbif.org

Tombstone pages for deleted IDs
by Markus Döring 18 Aug '23

18 Aug '23

Dear COL users, the COL portal now has tombstone pages for once existing, but now deleted COL identifiers. Here is an example: https://www.catalogueoflife.org/data/taxon/64SCW This shows the basic information as it once existed as well as links to the first and last appearance in ChecklistBank. The API still returns a 404 for these ids, but does include a message and an archived usage instance that can be processed: https://api.checklistbank.org/dataset/3LR/taxon/64SCW Btw this is also true for synonyms being looked up as accepted names / taxa: https://api.catalogueoflife.org/dataset/3LR/taxon/7Q8L8 Best wishes, Markus

1 0

CLB API related data
by MANCINI Raffael 17 Aug '23

17 Aug '23

Dear list, I'm developing a national "datawarehouse"/repository for natural history observation/occurrence data integrating several sources including GBIF and iNaturalist. For this purpose I'm generating lists of names and taxa related to the different ingested observation sources. In order to offer unified taxon search and filter functionality, I need equivalence mappings between ID-namespaces of different checklists (GBIF, CoL, internal, ornitho.lu, etc). I noticed that the CLB API offers the call `/dataset/{key}/nameusage/{id}/related` which for key=3LR and datasetKey=139831 [iNat] returns a correspondence. Where does the data for this mapping come from? Examining both the CoL and iNat checklists NameRelation.tsv (from the CoLDP downloads) didn't give a match. Is this data imported into CLB though a different checklist or is it supplemented though some more hidden datasource? The ID-namespace translation does not seem to be universally implemented and seems somehow tied to the dataset (translation from iNat to CoL didn't seem to work), is this correct? Do you plan on supporting my use-case more explicitly in the future? I would prefer not to use the CLB API and instead import the relevant checklists into our postgresql database in order to allow for an efficient and flexible operation. Thanks for the great work that went into CLB and related projects! Best regards, -- Raffael Mancini IT administrator and developer Service d'information digital sur le patrimoine naturel (SIDPNAT) Musée National d'Histoire Naturelle Luxembourg T: +352 247 66667 - https://mnhn.lu

2 3

ChecklistBank Test Environment to be reset on 16th January
by Markus Döring 12 Jan '23

12 Jan '23

Dear ChecklistBank users, we will reset the test environment of ChecklistBank next Monday, 16th January. This removes all data, but all datasets registered in GBIF will reappear quickly. If you need to preserve any data please download it before as we do not keep any backups. Users with editor rights will remain editors and can create new datasets, but will not be linked to any existing (GBIF) datasets. With thanks, Markus https://www.dev.checklistbank.org/

1 0

New monthly edition April 2021 live
by Markus Döring 24 Mar '22

24 Mar '22

Dear COL users, we are happy to announce that we have finally started again to publish monthly releases of the Catalogue of Life. The latest early April 2021 edition is not public on the main portal and available in the API. As this is the first time we have reissued our new identifiers, we have also taken some time to write a small post about stable identifiers in COL which I hope is informative and worth a read: https://www.catalogueoflife.org/2021/04/14/stable-ids Best wishes, Markus & the entire COL team +++ COL RESOURCES +++ COL Website: http://www.catalogueoflife.org/ COL ChecklistBank: http://data.catalogueoflife.org/ COL ChecklistBank API: http://api.catalogueoflife.org/ COL latest release in API: http://api.catalogueoflife.org/dataset/3LR

2 3

Catalogue of Life - call for proposals to prepare documentation
by Chantal Huijbers 01 Mar '22

01 Mar '22

Dear all, We would like to notify you that the Catalogue of Life has published two calls for proposals to prepare documentation on 1) the COL API, and 2) the COLDP format. We have some funding available for people who are familiar with the API and/or COLDP to write up documentation focused on examples and user guidelines. We anticipate that these will be relatively short projects, which should be completed within the next 6 months. For more information about the calls and how to submit a proposal, please visit our website: Call for API documentation <https://www.catalogueoflife.org/2022/03/01/documentation-call-API> Call for COLDP documentation <https://www.catalogueoflife.org/2022/03/01/documentation-call-COLDP> Regards, Chantal Huijbers COL Secretariat

1 0

API Introduction
by Markus Döring 23 Feb '22

23 Feb '22

Dear all, as you all know at the end of last year the Catalogue of Life has moved to a whole new infrastructure and now offers a new and richer API. As I had a few similar requests vie email lately I thought it would be good to give you a quick introduction to get started, while we are still working on better documentation. The basic concept has changed in the sense that the API does not only expose the Catalogue of Life (COL), but all of ChecklistBank (CLB) which hosts a large number of taxonomic or nomenclatural datasets including the sources that we use to assemble the COL. You can read here more about it: https://www.catalogueoflife.org/about/colpipeline The base URL of the API uses Swagger to exposes a rudimentary documentation of all the available REST resources and allows you to test it with real data: https://api.catalogueoflife.org In order to access the latest version of COL you need to use the datasetKey=3LR (note that the swagger generated API docs do not allow strings, please use the currents release key 2242 for testing or curl). Pretty much all calls of interest to you should be scoped under http://api.catalogueoflife.org/dataset/3LR In order to search for a name you could be doing this: http://api.catalogueoflife.org/dataset/3LR/nameusage/search?q=Abies&rank=SP… To get taxon details for a given taxonID this does most of it: Basics: http://api.catalogueoflife.org/dataset/2242/taxon/5QQJ9 Full Infos: http://api.catalogueoflife.org/dataset/2242/taxon/5QQJ9/info To browse the tree you can use the Tree API which in its basic form works like this: Root taxa: http://api.catalogueoflife.org/dataset/3LR/tree Plant Children: http://api.catalogueoflife.org/dataset/3LR/tree/P/children<http://api.catalogueoflife.org/dataset/3LR/tree/P> We now also offer reusable React UI components that you can embed in your own site or application to browse the tree, search and present species pages: https://github.com/CatalogueOfLife/portal-components We have also migrated the previous "legacy" API which you can access with the latest data from https://api.catalogueoflife.org/col/webservice<https://api.catalogueoflife.org/col/webservice?name=Abies%20alba&format=jso…> like this: https://api.catalogueoflife.org/col/webservice?name=Abies%20alba&format=jso… There are a few small differences how we treat references compared to the previous version, otherwise it should work just as before. We do consider this a legacy API and would like to encourage everyone using it still to migrate to our new API above. Please don't hesitate to ask any questions on this list if you want to know more on how to use our API. With best wishes, Markus +++ COL RESOURCES +++ COL Website: http://www.catalogueoflife.org/ COL ChecklistBank: http://data.catalogueoflife.org/ COL ChecklistBank API: http://api.catalogueoflife.org/ COL latest release in API: http://api.catalogueoflife.org/dataset/3LR -- Markus Döring Lead Developer Catalogue of Life Global Biodiversity Information Facility (GBIF) mdoering(a)gbif.org<mailto:mdoering@gbif.org> http://www.gbif.org<http://www.gbif.org/>

3 8

Maintenance work coming up - API in read only
by Markus Döring 25 Jan '22

25 Jan '22

Dear COL users, we are preparing for a larger update of the main database over this weekend for which we will have to turn the COL API into a read only mode. We expect the work to start later today and to stay in read only mode until early Monday. Please excuse any inconveniences during that time. The COL portal and most of the API will work just as usual, but expect the following services to be unavailable: - dataset imports - any active project work incl. sector syncs and tree manipulations - name matching With the changes to come we will be prepared to synchronise COL Checklist Bank with the GBIF registry and thus ingest all GBIF checklist datasets including all of the Plazi work. Best, Markus +++ COL RESOURCES +++ COL Portal: https://www.catalogueoflife.org/ COL ChecklistBank: https://data.catalogueoflife.org/ COL ChecklistBank API: https://api.catalogueoflife.org/ COL latest release in API: https://api.catalogueoflife.org/dataset/3LR

3 2

Download bug fixed
by Markus Döring 14 Nov '21

14 Nov '21

Dear all, in the past few days we had a bug when downloading Darwin Core & ColDP archives. This has been fixed just now so you should be able to repeat the downloads that have failed previously. With thanks, Markus +++ COL RESOURCES +++ COL Website: http://www.catalogueoflife.org/ COL ChecklistBank: http://data.catalogueoflife.org/ COL ChecklistBank API: http://api.catalogueoflife.org/ COL latest release in API: http://api.catalogueoflife.org/dataset/3LR

1 0

Question about Catalogue of Life (COL) Data Package file format
by Cheng, Tiejun (NIH/NLM/NCBI) [E] 29 Sep '21

29 Sep '21

Dear COL team, I'm trying to download the ColDP Archive of THE COL CHECKLIST VERSION 2021-09-21 on this page: https://www.catalogueoflife.org/data/download. However, the actual download link https://download.catalogueoflife.org/col/monthly/2021-09-21_coldp.zip is not working and returns 404. I am able to find another data file here: https://download.catalogueoflife.org/col/. The most recent one I believe is https://download.catalogueoflife.org/col/latest_coldp.zip with a timestamp of 2021-08-27 11:00. According to the description of ColDP file format here: https://www.catalogueoflife.org/about/colusage#data-formats. The ZIP archive is supposed to bundle the following delimited text files: * Name<https://github.com/CatalogueOfLife/coldp/blob/master/README.md#name> * NameRelation<https://github.com/CatalogueOfLife/coldp/blob/master/README.md#namerelation> * Taxon<https://github.com/CatalogueOfLife/coldp/blob/master/README.md#taxon> * Synonym<https://github.com/CatalogueOfLife/coldp/blob/master/README.md#synonym> * NameUsage<https://github.com/CatalogueOfLife/coldp/blob/master/README.md#nameusage> * Reference<https://github.com/CatalogueOfLife/coldp/blob/master/README.md#reference> * TypeMaterial<https://github.com/CatalogueOfLife/coldp/blob/master/README.md#typematerial> * Distribution<https://github.com/CatalogueOfLife/coldp/blob/master/README.md#distribution> * VernacularName<https://github.com/CatalogueOfLife/coldp/blob/master/README.md#vernacularna…> * Media<https://github.com/CatalogueOfLife/coldp/blob/master/README.md#media> * SpeciesInteraction<https://github.com/CatalogueOfLife/coldp/blob/master/README.md#speciesinter…> * TaxonConceptRelation<https://github.com/CatalogueOfLife/coldp/blob/master/README.md#taxonconcept…> * SpeciesEstimate<https://github.com/CatalogueOfLife/coldp/blob/master/README.md#speciesestim…> * Treatments<https://github.com/CatalogueOfLife/coldp/blob/master/README.md#treatment> However, In the latest_coldp.zip I downloaded, I got something different. $zipinfo -1 latest_coldp.zip * NameUsage.tsv * NameRelation.tsv * TypeMaterial.tsv * VernacularName.tsv * Distribution.tsv * Media.tsv * SpeciesEstimate.tsv * SpeciesInteraction.tsv * TaxonConceptRelation.tsv * Reference.tsv * reference.json (<- this is new) Some important files are missing. * Name * Taxon * Synomym * Treatments Could you please help understand it? Thanks. Best regards, Tiejun CHENG, Ph.D. --------------------------- National Center for Biotechnology Information (NCBI) National Library of Medicine (NLM) National Institutes of Health (NIH) Bldg. 38A, Rm. 8S816A 8600 Rockville Pike Bethesda, MD 20894 Phone: 301-402-9527 Email: chengt2(a)ncbi.nlm.nih.gov<mailto:chengt2@ncbi.nlm.nih.gov>

2 2

Re: [COL-Users] COL downloads changes
by Markus Döring 12 May '21

12 May '21

Dear Scott, I must have missed your email in march, sorry for that. Let me still answer your important questions about the monthly downloads. COL aims to produce a monthly release which we will keep at least for a year accessible via the API with all its features. Each release has its distinct datasetKey in ChecklistBank. There is no fixed day when we will issue the release. Moving to the new infrastructure at the end of last year caused a few teething issues which made us skip a few releases in january/february for example. We hope this will not happen again and you should see now regular monthly updates. One of these releases will be tagged as an Annual release which from the API point of view is just the same as a monthly one. But it will not be removed from ChecklistBank and therefore you have long term access via the API to it. We plan to issue the next annual release in June. Once a monthly release is deleted, we will still keep the data in various formats for download. But it will be gone from the database. The download archive contains a DwC archive (YYYY-MM-DD_dwca.zip) and an ACEF archive (YYYY-MM-DD_acef.zip) right now, with the next May release we will also add a new ColDP archive to the supported formats. A native postgres dump is on our list too, but that is not straight forward as COL is just a small part of ChecklistBank and we will need to filter out the relevant bits. Prior to December 2020 we only had DwC archives, but these has used some slightly different terms than we use today. Note that we do not export a flattened classification (family, order, etc) at this stage, but we plan to add that back in again in a not too distant future. The same applies also to ColDP. The ACEF archive is slightly special. We used it to transfer the data to the old systems and uses \N to represent NULL, which is a postgres specific convention. The files are proper CSV files with a header row and not tab separated. There is a short SQL script to load the ACEF files into a postgres database which also has DDL: https://github.com/CatalogueOfLife/backend/blob/master/webservice/src/main/… The DwCA and ColDP archives on the other hand use just an empty string and also use a header row with the term names. Files in both dwca and coldp are tab delimited and use the .tsv file extension. For ColDP we maintain a Postgres, MS Access and Excel schema, but that needs some small updates as we are about to freeze the format for a final & fixed 1st version. Hope thats useful, Markus Hi, I had been using the downloads available at https://download.catalogueoflife.org/col/monthly/ to construct a SQLite version of the database here https://github.com/sckott/col-sql to make it easier for users to use. Two questions, the 2nd with many parts: 1. Will https://download.catalogueoflife.org/col/monthly/ continue to be updated every 2 or 3 months with a new database dump? 2. If the answer to (1) is yes: The format changed in the last database dump "2020-12-01_acef.zip". a. The included file names changed, and file types changed from .tsv to .csv (although the data still appears to be tab-sep). Was it intended to change to comma-sep? b. Will there be more changes to the monthly dump? c. Will the release cycle be something predictable? Every 2 or 3 months? d. The files have a lot of "\N" in them. Is this supposed to be a newline character? I've not seen a newline with a capital N. e. Any schema to use for these various csv files? Thanks! Scott Chamberlain

2 1