Create your own importer plugin¶
Since version 1.4, pysteps allows the users to add new importers by installing external packages, called plugins, without modifying the pysteps installation. These plugins need to follow a particular structure (described next) to allow pysteps to discover and integrate the new importers to the pysteps interface without any user intervention. For a short description of how the plugins work, see How do the plugins work?. There are two ways of creating your plugin. The first one involves building the plugin from scratch. An easier alternative is using a Cookiecutter template that easily builds the skeleton for the new importer plugin.
There are two ways of creating a plugin. The first one is building the importers plugin from scratch. However, an easier alternative is using this Cookiecutter template to create the skeleton for the new importer plugin, and then customize it. However, this can be a daunting task if you are creating your first plugin. Hence, before customizing the cookiecutter template, let’s review the main components of the plugin architecture by describing how to build an importers plugin from scratch.
After you are familiar with the plugin fundamentals, you can build your plugin from the cookiecutter template. For a detailed description of the template, see Description of the pysteps-plugin template.
Minimal plugin project¶
Let’s suppose that we want to add two new importers to pysteps for reading the radar composites from the “Awesome Bureau of Composites”, kindly abbreviated as “abc”. The composites provided by this institution are available in two different formats: Netcdf and Grib2. The details of each format are not important for the rest of this description. Just remember the names of the two formats.
Without further ado, let’s create a python package (a.k.a. the plugin) implementing the two importers. For simplicity, we will only include the elements that are strictly needed for the plugin to be installed and to work correctly.
The minimal python package to implement an importer plugin has the following structure:
pysteps-importer-abc (project name)
├── pysteps_importer_abc (package name)
│ ├── importer_abc_xyz.py (importer module)
│ └── __init__.py (Initialize the pysteps_importer_abc package)
├── setup.py (Build and installation script)
└── MANIFEST.in (manifest template)
Project name¶
pysteps-importer-abc (project name)
For the project name, our example used the following convention: pysteps-importer-<institution short name>. Note that this convention is not strictly needed, and any name can be used.
Package name¶
pysteps-importer-abc
└── pysteps_importer_abc (package name)
This is the name of our package containing the new importers for pysteps. The package name should not contain spaces, hyphens, or uppercase letters. For our example, the package name is pysteps_importer_abc.
__init__.py¶
pysteps-importer-abc
├── pysteps_importer_abc
└───── __init__.py
The __init__.py files are required to inform python that a given directory contains a python package. This is also the first file executed when the importer plugin (i.e., the package) is imported.
Importer module¶
pysteps-importer-abc
├── pysteps_importer_abc
└───── importer_abc_xyz.py (importer module)
Inside the package folder (pysteps_importer_abc), we place the python module (or modules) containing the actual implementation of our new importers. Below, there is an example of an importer module that implements the skeleton of two different importers (the “grib” and “netcdf” importer that we are using as an example):
# -*- coding: utf-8 -*-
"""
One line description of this module. E.g. "Awesome Bureau of Composites importers."
Here you can write a more extensive description (optional). For example, describe the
readers that are implemented (e.g. Grib2 and Netcdf) and any other thing that you
consider relevant.
"""
# Here import your libraries
import numpy as np
### Uncomment the next lines if pyproj is needed for the importer.
# try:
# import pyproj
#
# PYPROJ_IMPORTED = True
# except ImportError:
# PYPROJ_IMPORTED = False
from pysteps.decorators import postprocess_import
from pysteps.exceptions import MissingOptionalDependency
# Function import_abc_grib to import Grib files from the Awesome Bureau of Composites
# ===================================================================================
# IMPORTANT: The name of the importer should follow the "importer_institution_format"
# naming convention, where "institution" is the acronym or short-name of the
# institution. The "importer_" prefix to the importer name is MANDATORY since it is
# used by the pysteps interface.
#
# Check the pysteps documentation for examples of importers names that follow this
# convention:
# https://pysteps.readthedocs.io/en/latest/pysteps_reference/io.html#available-importers
#
# The definition of the new importer functions should have the following form:
#
# @postprocess_import()
# def import_institution_format(filename, keyword1="some_keyword", keyword2=10, **kwargs):
#
# The "filename" positional argument and the "**kwargs" argument are mandatory.
# However, the keyword1="some_keyword", keyword2=10 are optional shown in the above
# example are optional. In the case that the new importer needs optional keywords to
# fine control its behavior, it is strongly encouraged to define them explicitly
# instead of relying on the "**kwargs" argument.
#
#
# The @postprocess_import operator
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#
# The `pysteps.io.importers` module uses the `postprocess_import` decorator to easily
# define the default data type and default value used for the missing/invalid data.
# The allowed postprocessing operations are
# - Data type casting using the `dtype` keyword.
# - Set invalid or missing data to a predefined value using the `fillna` keyword.
# The @postprocess_import decorator should always be added immediately above the
# importer definition to maintain full compatibility with the pysteps library.
# Adding the decorator @add_postprocess_keywords() without any keywords will ensure that
# the precipitation data returned by the importer has double precision, and the
# invalid/missing data is set to `np.nan`.
# For more information on the postprocessing decorator, see:
# https://pysteps.readthedocs.io/en/latest/generated/pysteps.decorators.postprocess_import.html
#
# Function arguments
# ~~~~~~~~~~~~~~~~~~
#
# The function arguments should have the following form:
# (filename, keyword1="some_keyword", keyword2=10,...,keywordN="something", **kwargs)
# The `filename` and `**kwargs` arguments are mandatory to comply with the pysteps
# interface. To fine-control the behavior of the importer, additional keywords can be
# added to the function.
# For example: keyword1="some_keyword", keyword2=10, ..., keywordN="something"
# It is recommended to declare the keywords explicitly in the function to improve the
# readability.
#
#
# Return arguments
# ~~~~~~~~~~~~~~~~
#
# The importer should always return the following fields:
#
# precipitation : 2D array (ndarray or MaskedArray)
# Precipitation field in mm/h. The dimensions are [latitude, longitude].
# quality : 2D array or None
# If no quality information is available, set to None.
# metadata : dict
# Associated metadata (pixel sizes, map projections, etc.).
#
#
# Now, let's implement the importer.
@postprocess_import()
def import_abc_grib(filename, keyword1="some_keyword", keyword2=10, **kwargs):
"""
Here you need to add the documentation for the importer. A minimal documentation is
strictly needed since the pysteps importers interface expect docstrings.
For example, a documentation may look like this:
Import a precipitation field from the Awesome Bureau of Composites stored in
Grib format.
Parameters
----------
filename : str
Name of the file to import.
keyword1 : str
Some keyword used to fine control the importer behavior.
keyword2 : int
Another keyword used to fine control the importer behavior.
{extra_kwargs_doc}
####################################################################################
# The {extra_kwargs_doc} above is needed to add default keywords added to this #
# importer by the pysteps.decorator.postprocess_import decorator. #
# IMPORTANT: Remove this box in the final version of this function #
####################################################################################
Returns
-------
precipitation : 2D array, float32
Precipitation field in mm/h. The dimensions are [latitude, longitude].
quality : 2D array or None
If no quality information is available, set to None.
metadata : dict
Associated metadata (pixel sizes, map projections, etc.).
"""
### Uncomment the next lines if pyproj is needed for the importer
# if not PYPROJ_IMPORTED:
# raise MissingOptionalDependency(
# "pyproj package is required by import_abc_grib
# "but it is not installed"
# )
####################################################################################
# Add the code to read the precipitation data here. Note that only cartesian grids
# are supported by pysteps!
# In this example, we are going create a precipitation fields of only zeros.
precip = np.zeros((100, 100), dtype="double")
# The "double" precision is used in this example to indicate that the imaginary
# grib file stores the data using double precision.
# Quality field, should have the same dimensions of the precipitation field.
# Use None if not information is available.
quality = None
# Adjust the metadata fields according to the file format specifications.
# For additional information on the metadata fields, see:
# https://pysteps.readthedocs.io/en/latest/pysteps_reference/io.html#pysteps-io-importers
# The projection definition is a string with a PROJ.4-compatible projection
# definition of the cartographic projection used for the data
# More info at: https://proj.org/usage/projections.html
# For example:
projection_definition = (
"+proj=stere +lon_0=25E +lat_0=90N +lat_ts=60 +a=6371288 "
"+x_0=380886.310 +y_0=3395677.920 +no_defs",
)
metadata = dict(
xpixelsize=1,
ypixelsize=1,
cartesian_unit="km",
unit="mm/h",
transform=None,
zerovalue=0,
institution="The institution that created the file",
projection=projection_definition,
yorigin="upper",
threshold=0.03,
x1=0,
x2=100,
y1=0,
y2=100,
)
# IMPORTANT! The importers should always return the following fields:
return precip, quality, metadata
# Function import_abc_netcdf to import NETCDF files from the Awesome Bureau of Composites
# =======================================================================================
@postprocess_import()
def import_abc_netcdf(filename, keyword1="some_keyword", keyword2=10, **kwargs):
"""Import a precipitation field from the Awesome Bureau of Composites stored in
Netcdf format.
Parameters
----------
filename : str
Name of the file to import.
keyword1 : str
Some keyword used to fine control the importer behavior.
keyword2 : int
Another keyword used to fine control the importer behavior.
{extra_kwargs_doc}
####################################################################################
# The {extra_kwargs_doc} above is needed to add default keywords added to this #
# importer by the pysteps.decorator.postprocess_import decorator. #
# IMPORTANT: Remove these box in the final version of this function #
####################################################################################
Returns
-------
precipitation : 2D array, float32
Precipitation field in mm/h. The dimensions are [latitude, longitude].
quality : 2D array or None
If no quality information is available, set to None.
metadata : dict
Associated metadata (pixel sizes, map projections, etc.).
"""
### Uncomment the next lines if pyproj is needed for the importer
# if not PYPROJ_IMPORTED:
# raise MissingOptionalDependency(
# "pyproj package is required by import_abc_grib
# "but it is not installed"
# )
# Add the code to read the precipitation data (cartesian grid!) here.
precip = np.zeros((100, 100))
quality = np.ones((100, 100))
projection_definition = (
"+proj=stere +lon_0=25E +lat_0=90N +lat_ts=60 +a=6371288 "
"+x_0=380886.310 +y_0=3395677.920 +no_defs",
)
metadata = dict(
xpixelsize=1,
ypixelsize=1,
cartesian_unit="km",
unit="mm/h",
transform=None,
zerovalue=0,
institution="The institution that created the file",
projection=projection_definition,
yorigin="upper",
threshold=0.03,
x1=0,
x2=100,
y1=0,
y2=100,
)
return precip, quality, metadata
setup.py¶
pysteps-importer-abc (project name)
└── setup.py (Build and installation script)
The setup.py file contains all the definitions for building, distributing, and installing the package. A commented example of a setup.py script used for the plugin installation is shown next:
# -*- coding: utf-8 -*-
"""
Setup script for installing the Awesome Bureau of Composites (abc) importers plugin.
"""
# IMPORTANT: Note that for this example we are referring to the plugin also as the
# "package".
# This setup script uses setuptools to handle the package installation and distribution.
# Setuptools is a fully-featured and stable library that facilitates packaging Python
# projects. See more at: https://setuptools.readthedocs.io/en/latest/
from setuptools import setup, find_packages
# The long description is used to explain to the users how the package is installed and
# how it is used. Typically, a python package includes this long description in a
# separate README file.
# The following lines use the contents of the readme file as long
# description.
with open("README.rst") as readme_file:
long_description = readme_file.read()
# Add the plugin dependencies here. This dependencies will be installed along with your
# package.
requirements = []
# Add the dependencies needed to build the package.
# For example, if the package use compile extensions (like Cython), they can be included
# here.
setup_requirements = []
setup(
author="Your name",
author_email="Your email",
python_requires=">=3.6", # Pysteps supports python versions >3.6
# Add the classifiers to the package.
classifiers=[
"Intended Audience :: Developers",
"Natural Language :: English",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.6",
"Programming Language :: Python :: 3.7",
"Programming Language :: Python :: 3.8",
],
description="Pysteps plugin for importing the ABC composites.", # short description
install_requires=requirements,
license="MIT license",
long_description=long_description,
include_package_data=True,
keywords=["pysteps-importer-abc", "pysteps", "plugin", "importer"],
name="pysteps-importer-abc",
packages=find_packages(),
setup_requires=setup_requirements,
# Entry points
# ~~~~~~~~~~~~
#
# This is the most important part of the plugin setup script.
# Entry points are a mechanism for an installed python distribution to advertise
# some of the components installed (packages, modules, and scripts) to other
# applications (in our case, pysteps).
# https://packaging.python.org/specifications/entry-points/
#
# An entry point is defined by three properties:
# - The group that an entry point belongs indicate the kind of functionality that
# provides. For the pysteps importers use the "pysteps.plugins.importers" group.
# - The unique name that is used to identify this entry point in the
# "pysteps.plugins.importers" group.
# - A reference to a Python object. For the pysteps importers, the object should
# point to a importer function, and should have the following form:
# package_name.module:function.
# The setup script uses a dictionary mapping the entry point group names to a list
# of strings defining the importers provided by this package (our plugin).
# The general form of the entry points dictionary is:
# entry_points={
# "group_name": [
# "entry_point_name=package_name.module:function",
# "entry_point_name=package_name.module:function2",
# ]
# },
# For this particular example, the entry points are defined as:
entry_points={
"pysteps.plugins.importers": [
"import_abc_grib=pysteps_importer_abc.importers:import_abc_grib",
"import_abc_netcdf=pysteps_importer_abc.importers:import_abc_netcdf",
]
},
version="0.1.0", # Indicate the version of the plugin
# https://setuptools.readthedocs.io/en/latest/userguide/miscellaneous.html?highlight=zip_safe#setting-the-zip-safe-flag
zip_safe=False, # Do not compress the package.
)
Manifest.in¶
If you don’t supply an explicit list of files, the installation using setup.py will include the minimal files needed for the package to run (the *.py files, for example). The Manifest.in file contains the list of additional files and directories to be included in your source distribution.
Next, we show an example of a Manifest file that containing a README and the LICENSE files located in the project root. Lines starting with # indicate comments, and they are ignored.
# This file contains the additional files included in your plugin package
include LICENSE
include README.rst
###You can also add directories with data, tests, etc.
# recursive-include dir_with_data
###Include the documentation directory, if any.
# recursive-include doc
For more information about the manifest file, see https://docs.python.org/3/distutils/sourcedist.html#specifying-the-files-to-distribute
Get in touch¶
If you have questions about the plugin implementation, you can get in touch with the pysteps community on our pysteps slack. To get access to it, you need to ask for an invitation or use the automatic invitation page here. This invite page can sometimes take a while to load so, please be patient.