PROJ RFC 5: Adopt GeoTIFF-based grids for grids delivered with PROJ

Author:

Even Rouault

Contact:

even.rouault@spatialys.com

Status:

Adopted

Implementation target:

PROJ 7

Last Updated:

2020-01-28

Motivation

This RFC is a continuation of PROJ RFC 4: Remote access to grids and GeoTIFF grids. With RFC4, PROJ can, upon request of the user, download grids from a CDN in a progressive way. There is also API, such as proj_download_file() to be able to download a GeoTIFF grid in the user writable directory. The content of the CDN at https://cdn.proj.org is https://github.com/OSGeo/PROJ-data , which has the same content as https://github.com/OSGeo/proj-datumgrid converted in GeoTIFF files. In the current state, we could have a somewhat inconsistency between users relying on the proj-datumgrid, proj-datumgrid-[world,northamerica,oceania,europe] packages of mostly NTv2 and GTX files, and what is shipped through the CDN. Maintaining two repositories is also a maintenance burden in the long term.

It is thus desirable to have a single source of truth, and we propose it to be based on the GeoTIFF grids.

Summary of work planned by this RFC and related decisions

• https://github.com/OSGeo/PROJ-data/ will be used, starting with PROJ 7.0, to create "static" grid packages.

• For now, a single package of, mostly GeoTIFF grids (a few text files for PROJ init style files, as well as a few edge cases for deformation models where grids have not been converted), will be delivered. Its size at the time of writing is 486 MB (compared to 1.5 GB of uncompressed NTv2 + GTX content, compressed to ~ 700 MB currently)

• The content of this archive will be flat, i.e. no subdirectories

• Each file will be named according to the following pattern ${agency_name}_${filename}[.ext]. For example fr_ign_ntf_r93.tif This convention should allow packagers, if the need arise, to be able to split the monolithic package in smaller ones, based on criterion related to the country.

  The agency name is the one you can see from the directory names at https://github.com/OSGeo/PROJ-data/. ${agency_name} itself is structure like ${two_letter_country_code_of_agency_nationality}_${some_abbreviation} (with the exception of eur_nkg, for the Nordic Geodetic Commission which isn't affiliated to a single country but to some European countries, and follows the general scheme)

• https://github.com/OSGeo/proj-datumgrid and related packages will only be maintained during the remaining lifetime of PROJ 6.x. After that, the repository will no longer receive any update and will be put in archiving state (see https://help.github.com/en/github/creating-cloning-and-archiving-repositories/about-archiving-repositories)

• PROJ database grid_alternatives table will be updated to point to the new TIFF filenames. It will also maintain the old names as used by current proj-datumgrid packages to be able to provide backward compatibility when a PROJ string refers to a grid by its previous name.

• Upon adoption of this RFC, new grids referenced by PROJ database will only point to GeoTIFF grid names.

• Related to the above point, if a PROJ string refers to a grid name, let's say foo.gsb. This grid will first be looked for in all the relevant locations under this name. If no match is found, then a lookup in the grid_alternatives table will be done to retrieve the potential new name (GeoTIFF file), and if there's such match, a new look-up in the file system will be done with the name of this GeoTIFF file.

• The package_name column of grid_alternatives will no longer be filled. And url will be filled with the direct URL to the grid in the CDN, for example: https://cdn.proj.org/fr_ign_ntf_r93.tif

• The Python scripts to convert grids (NTv2, GTX) to GeoTIFF currently available at https://github.com/rouault/sample_proj_gtiff_grids/ will be moved to a grid_tools/ subdirectories of https://github.com/OSGeo/PROJ-data/ Documentation for those utilities will be added to PROJ documentation.

• Obviously, all the above assumes PROJ builds to have libtiff enabled. Non-libtiff builds are not considered as nominal PROJ builds (if a PROJ master build is attempted and libtiff is not detected, it fails. The user has to explicitly ask to disable TIFF support), and users deciding to go through that route will have to deal with the consequences (that is that grid-based transformations generated by PROJ will likely be non working)

Backward compatibility

This change is considered to be mostly backward compatible. There might be impacts for software using proj_coordoperation_get_grid_used() and assuming that the url returned is one of the proj-datumgrid-xxx files at https://download.osgeo.org. As mentioned in https://lists.osgeo.org/pipermail/proj/2020-January/009274.html , this assumption was not completely bullet-proof either. There will be impacts on software checking the value of PROJ pipeline strings resulting proj_create_crs_to_crs(). The new grid names will now be returned (the most impacted software will likely be PROJ's own test suite)

Although discouraged, people not using the new proj-datumgrid-geotiff-XXX.zip archives, should still be able to use the old archives made of NTv2/GTX files, at least as long as the PROJ database does not only point to a GeoTIFF grid. So this might be a short-term partly working solution, but at time goes, it will become increasingly non-working. The nominal combination will be PROJ 7.0 + proj-datumgrid-geotiff-1.0.zip

Testing

PROJ test suite will have to be adapted for the new TIFF based filenames.

Mechanism to auto-promote existing NTv2/GTX names to TIFF ones will be exercised.

Proposed implementation

https://github.com/OSGeo/PROJ/pull/1891 and https://github.com/OSGeo/PROJ-data/pull/5

Adoption status

The RFC was adopted on 2020-01-28 with +1's from the following PSC members

• Kristian Evers

• Even Rouault

• Thomas Knudsen

• Howard Butler

• Kurt Schwehr