Friday, November 25, 2016

Python resources for Hydrologists

Python is a modern object oriented language. Occasionally I wrote about it in my posts, also for remarking that I went in a different direction. However, I cannot deny the evidence that more and more people are choosing it, and there are good reasons, as their language of choice for doing research and hydrological applications. In fact since 2017, I am using it in place of R for scripting and data managing. Below you will find a list of resources. Please do not hesitate to bring my attention to yours or others' contributions which I have not yet in my group.

Motivations for Python use, over other choices, can be found in this blog post or in this paper.

To understand how to start you can follow Python programming for hydrology students that starts with indicating how to install it.

For who wants to start with Python (for hydrologists), I suggest to give a look to my blog post Python general resources. For others, please give a look below.

Python is especially use as a glue for existing program, either written in C or FORTRAN. We have the cases of
  • CFM is a programming library to create hydrological models. Although written in C++, it has a Python interface 
  • ESMF regridding has been interfaced with Python ESMPy
  • GRASS GIS has been interfaced with Python
  • Python is also interfaced to gvSIG, as you can see here
  • HPGL a High Performance Geostatistics Library. Written in C++ is glued together by Python 
  • MODFLOW the groundwater model is interfaced by FloPy. Documentation and other information is here
  • PcRaster - Is a collection of software targeted at the development and deployment of spatio-temporal environmental models. It has a python interface which is constantly being enhanced. 
  • OpenHydrology is a library of open source hydrological software written in Python to operate as packages under an umbrella interface 
  • PyHSPF Python extensions to the Hydrological Simulation Program in Fortran (HSPF
  • PyQGIS: A Python interface to QGIS 
  • RhessysWorkflow  RHESSysWorkflows provides Python scripts for building RHESSys models. Other Pythonic material on RHESSys can be found here.
  • UWHydro tools for connecting University of Washington hydrological models, and, in particolar, the VIC driver PythonDriver

In the reign of hydrologic applications entirely written in Python, we remind:

  • ANUGA 2 - package for modelling dam breaks, riverine flooding, storm-surge or tsunamis. In Python and C. 
  • EcoHydrolib provides a series of Python scripts for performing ecohydrology data preparation workflows. 
  • evaplib: Python library containing functions for calculation of evaporation rates. Functions include Penman open water evaporation, Makkink reference evaporation, Priestley Taylor evaporation Penman Monteith (1965) evaporation and FAO's Penman Monteith ET0 reference evaporation for short, well-watered grass. In addition there is a function to calculate the sensible heat flux from temperature fluctuation measurements. View documentation of evaplib module functions. Module documentation is also available as a PDF document. Author: M.J. Waterloo. 
  • A GLUE, Generalised Likelihood Uncertainty Estimation (GLUE) developed by Framework Joost Delsman, at Deltares, 2011 
  • Groundwater flow modelling manual for Python written by Vincent post 
  • Hydro-conductor: A set of Python scripts and modules written to couple a hydrologic model with a regional glacier model
  • ODMToolsPython and ODMTools ODMTools is a python application for managing observational data using the Observations Data Model. ODMTools allows you to query, visualize, and edit data stored in an Observations Data Model (ODM) database.ODMTools was originally developed as part of the CUAHSI Hydrologic Information System. YOu can find a presentation about here
  • PyETo is a package for calculating reference/potential evapotranspiration (ETo). 
  • Python script for rectangular Piper plot (version December 2014): Python script for plotting chemical data in a rectangular python plot (see image) according to Ray and Mukherjee (2008) Groundwater 46(6): 893-896. Also download the example data file watersamples.txt. Author: B.M. van Breukelen. 
  • Python script for multiple Stiff plots (version June 2011): Python script for preparing multiple Stiff diagrammes (see image). Also download the example data file watersamples.txt. Author: B.M. van Breukelen. 
  • Haran Kiruba tools for hydrology Not really clear what he does. 
  • USEPA site contains various python (and other languages) tools, including an interface to Epanet and SWMM (a connection to swmmtoolbox is also avilable here). 
  • Also USGS has its python tools
  • sMAP 2.0 is a tutorial will cover how to retrieve data from a sMAP archiver using Python. 
  • ulmo clean, simple and fast access to public hydrology and climatology data 


Specific hydrological Hydrological Models are enumerated below.

  • EXP-HYDRO Model is a catchment scale hydrological model that operates at a daily time-step. 
  • Landlab Landlab is a python-based modeling environment that allows scientists and students to build numerical landscape models. Designed for disciplines that quantify earth surface dynamics such as geomorphology, hydrology, glaciology, and stratigraphy, it can also be used in related fields. 
  • LHMP - lumped hydrological models playground - tiny docker container with complete environment for predictions.
  • PyCatch is a component based hydrological model of catchments built within the PCRaster Python framework. The code is here. A related paper, here
  • PyTOPKAPI is a BSD licensed Python library implementing the TOPKAPI Hydrological model (Liu and Todini, 2002). The model is a physically-based and fully distributed hydrological model, which has already been successfully applied in several countries around the world 
  • SPHY. See for details on model and publications (HESS, Nature, etc) here. Just recently a new paper on climate change and mountain hydrology in PloS came out, using SPHY model, more info here.
  • Topoflow a python hydrologic model by Scott Peckham 
  • WOFpy is an implementation of CUAHSI's Water One Flow service stack in python 
  • wflow is a distributed hydrological model platform that currently includes two models: the wflow_sbm model (derived from the topog_sbm soil concept) and the wflow_hbv model which is a distributed version of the HBV model. This is actully part of a larger Deltares project called OpenStream

GIS capabilities are also present:


Also tools to deal with Meteorology:

  • meteolib: Python library containing meteorological functions for calculation of atmospheric vapour pressures, air density, latent heat of vapourisation, heat capacity at constant pressure, psychrometric constant, day length, extraterrestrial radiation input, potential temperature and wind vector. The documentation for this module is presented at here (meteolib module functions web site). Functions to convert event-based data records to equidistant time-spaced records (event2time) and to convert date values to day-of-year values (date2doy) are now in a separate meteo_util module. Documentation is presented here (meteo_util module functions web site). Module documentation is also available as a PDF document. Author: M.J. Waterloo. 
  • MetPy is An Open Source Python Toolkit for Meteorology 
  • Melodist (MEteoroLOgical observation time series DISaggregation Tool) is an open-source software package written in Python for temporally downscaling (disaggregating) daily meteorological time series to hourly data. It is documented in a GMD paper by Forster et al., 2016. 
  • Various resources for meteorology can be found in the pyaos blog
Statistical and data analysis tools are abundant
  • CUAHSI time series viewer
  • The basic cheatshit
  • NetCDF file operations are available here. However, there is also txt2netcdf which containsvarious Python functions for importing text into NetCDF data files (creating files, adding variables, listing structure, etc.), developed by Ko van Huissteden. 
  • Pandas is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language. (A short tutorial is also here
  • Extreme distribution (from scipy.stats) is here
  • An example of use of Pandas for analysing time series 
Visualisation is well served
  • ggplot is a plotting system for Python based on R's ggplot2 and the Grammar of Graphics. It is built for making profressional looking, plots quickly with minimal code. 
  • An impressive tour of Python possibilities in this field is given by “Regress to Impress” 
  • VisTrails an open-source scientific workflow and provenance management system that supports data exploration and visualization. Its website is here
  • uvcmetrics metrics aka diagnostics for comparing models with observations or each other. This is part of the Uv-CDAT website which contains also other visualisation tools. 
Tools for dealing with uncertainty and sensitivity analysis
Yet another repository of Python models and resources
A final comment

I am actually impressed by the quality of the contributions in Python. I think there is not anymore reason to use commercial program like Matlab in Universities (see this review here). What Matlab does, also Python does. Compared to R, it has a much more clear sintax and is, certainly, a better language. So I would suggest to use it with students (or R). I like R but I never built program on it, because its object orientation is really poor. Python is better and, as it is known its syntax is clean. Python is great to link FORTRAN and C/C++ native libraries, so actually many uses it to assembled libraries they wrote in those more performant languages.
As you know, however, my group uses Java as its principal programming language. Java, in comparison with Python, is less immediate and more verbose, but it allows to build many framework to work with, and is usually faster than Python. Probably because, Java is supported by magnificent building tools (Maven, Gradle) that allow to manage large projects in a way that probably cannot be done in Python.

25 comments:

  1. Wow! I've been working with python for hydrology for two years, but have in my bookmarks only a half of your links! Thank you a lot for amazing contribution! And some little bonus from me: LHMP - lumped hydrological models playground - tiny docker container with complete environment for predictions (https://github.com/hydrogo/LHMP).

    ReplyDelete
  2. Great and unique collection of sources and tools, thanks a lot for putting this together and sharing it with the community!

    Another model fully coded in Python using PcRaster library among others, is open-source SPHY: www.sphy.nl. See for details on model and publications (HESS, Nature, etc) here: http://www.sphy.nl/. Just recently a new paper on climate change and mountain hydrology in PloS came out, using SPHY model, more info here: http://www.futurewater.eu/2016/11/impacts-of-climate-change-on-the-upper-indus-basin/

    Cheers, Johannes

    ReplyDelete
    Replies
    1. I added to main text. Please let me know any news. Cheers, ric

      Delete
  3. Nice post! As a proposal, wouldn't it be beneficial to make this a curated list on github, e.g. awesome-python-hydrology, as they exist for more general topics (https://github.com/vinta/awesome-python)? It would allow for people to contribute and keep the list up to date as a community (when projects are no longer maintained or new projects appear). I can set it up or you can do it yourself?

    P.S. I think the link from openhydrology wrongly directs to http://open-hydrology.org/ instead of http://open-hydrology.org/

    ReplyDelete
    Replies
    1. Thank you, Stijn. You probably misspelled the last link. However, I redirected my own link directly to the Github repository. Yes, I could write and maintain a list in Github, but this post is "liquid" too. I can change and edit it as I desire, so, just tell me what to add or correct.

      Delete
    2. You can verify that I easily added your own tools, on the bottom of the post ;-).

      Delete
    3. Dear Stijn, in any case, as soon as, you cite this blog post appropriately, I encourage you to get all this material and put it on a proper Github site, if you feel it can be useful for the community.

      Delete
    4. Ok, I'll fix that in the following week. I think the main advantage is that the community itself can contribute to the repository and keep the list up to date (while some maintainers, I propose that you'll be collaborator as well, keep track of the pull requests of additions/adaptations).

      Delete
  4. Great! I indeed misspelled it as well, link to github is perfect.

    Concerning "my" tools, no need to provide a link to my name/Github as such (would be better if it would be a community effort). Maybe more interesting to point to specific repositories under the proper topic. I would advice the following adaptations:
    - Sensitivity analysis: https://github.com/SALib/SALib is currently more intensive maintained, contains almost all algorithms pystran has and just went through a review process in JOSS, https://github.com/openjournals/joss. I would recommend this link for SA. You can add https://github.com/stijnvanhoey/pystran as well in this section.
    - I would also add http://fb09-pasig.umwelt.uni-giessen.de/spotpy/ to the list with SA/UA/optimization tools, as this provides lots of useful algorithms to hydrological modellers.
    - https://github.com/stijnvanhoey/hydropy could be interesting in the section "Statistical and data analysis tools are abundant", as it aims to use the power of Python Pandas and add hydrological functionalities (will be further maintained).
    - https://github.com/stijnvanhoey/hydrus_wrapper could be provided as example of a 'wrapped' model in the section "glue for existing program, either written in C or FORTRAN."

    Other repositories are less relevant for this blog.

    Thanks, Stijn

    ReplyDelete
  5. Dear Riccardo,
    Great post! An inspiring one. If you will be looking for a partner in a hydrological project linked with python do not hesitate to contact me.

    Regards
    M.R.

    ReplyDelete
  6. Great post! Do you happen to know of any python or R programs to help automate comparing contaminants in groundwater to their respective standards/screening levels?

    ReplyDelete
  7. Wow Nice informative blog ....The blog containing an unique collection of the tools and resources...Thanks for sharing..

    ReplyDelete
  8. Thanks for sharing the information about the Python and keep updating us.This information is really useful to me.

    ReplyDelete
  9. This comment has been removed by a blog administrator.

    ReplyDelete
  10. You can also add pyMODIS - a python package for dealing with MODIS data.

    ReplyDelete
  11. This comment has been removed by a blog administrator.

    ReplyDelete
  12. Bokeh is really cool if you want to create interactive plots..

    ReplyDelete
  13. This comment has been removed by a blog administrator.

    ReplyDelete
  14. I appreciate your work on Python. It's such a wonderful read on Python course. Keep sharing stuffs like this. I am also educating people on similar Python training so if you are interested to know more you can watch this Python tutorial:-https://www.youtube.com/watch?v=e9p0_NB3WrM

    ReplyDelete
  15. Does anyone know which python model/package would be useful if I wanna simulate water contamination detection in water bodies like pond, lake etc?

    ReplyDelete