Wednesday, September 17, 2014

GEOtop binaries for Mac OS

Now (August 2015) we have new binaries compiled for Mac OS X. Please follow:

This is, however, the old post:
After a few hacking I was able to transport my compiled version of GEOtop and libraries to a different machine. It is still a rough approximation of what it can be. However, it works. No compiling is required as before, just installations:

#You have to know, first, that in Mac OS there exist an application called "Terminal" in /Applications/Utilities/ folder. If you do not know browse a little the web to understand what is is about.  From Terminal, then,

# Install the following packages (or verify you have them already):

Install,  Homebrew and other required libraries.

#Install needed libraries

brew install boost
brew install gdal
brew install proj

#Install GEOtop components and libraries: this require, at the moment, a little of hand working

Download the executable zipped file from here.

Once unzipped the file look at the directories created and place the files you find in the "/Libraries" and "/usr" directories into the directories with the same name in your Mac hard disk. Both the directories can be found from the root with the Finder and while Library is visible, "/usr" is usually not. To make it visible, follow the instructions here.

As you can see, once unzipped the files, I chose to create a "/usr/local/GEOtop" directory where all the real executable are (has to be placed). This directory is actually not a standard one, However, I placed a symbolic link into /usr/local/bin, which is a directory usually already present in the Terminal path (to make a symbolic link by yourself, see here). Differently of the /Library and the directory "/usr/local/lib" and "/usr/local/include" that already exist, "/usr/local/GEOtop" has to be created, and therefore you can simply drag and drop the unzipped directory to copy it.

At this point everything should be set, if everything works fine, you should just execute GEOtop from any place simply by issuing:

$ geotop

If it runs, at this point it should give a runtime error:

Error::Run time error
Error::You need to export the WorkingPath environment variable before to run the program
........exiting...

But this is what it should be^1.  Now, that you are doing it right:

#Download the examples

You can find them here. Unzip them  in your directory, for instance ~/GEOtop, under your Home directory ("~" stands for your Home directory or,  given as absolute path for  "/Users/YourHomeNameHere/GEOtop").

#Try geotop examples

cd to the directory (folder) where the tests are), for instance:

~/GEOtop/tests/test_sample_run/small_example

Issue:

$ geotop ./

If you did it well and the goddess of programmers assisted you, it should work, as it did for me and in Tim's machine.

Please observe that now (2015) we have:



_________________________________________________________________________________

^1 - If it does not run try:


/usr/local/GEOtop/geotop/geotop

and you should have the same runtime error. This just means that all is correct except for, maybe, the symbolic link or your PATH in Terminal shell, that, for some reason, is not containing  "/usr/local/bin".

Monday, September 15, 2014

Opinions on the GEOtop roadmap

Just opinions indeed. For who landed here occasionally, information on GEOtop can be found here.

1 - Infrastructure

Mountain-eering and Exact-lab companies are doing a visible and positive effort in having a clean C++ code, after the huge work made by Stefano Endrizzi and Stephan Gruber to arrive to GEOtop 2.0. However, I believe the work on C++  should make some further steps, which I delineate in one of my previous posts here.
I deem necessary to do at least the first step described there, but the others would be also necessary in order that many people can work collaboratively on the project, and to maintain what we have while producing alternatives, and progress. My solution is, obviously just a proposal, and others (that I do not know) could exist. I plan to work myself part time on this from now to February, but if I can have help from someone this will be great. While changing the basic structure of the model's main( ), if  I will be really able to do it in this favourable astral conjunction,  I will take notes, and provide information to everybody. 

2 - Algorithms

On the side of the algorithms I believe we can work in several directions.  Using an unstructured grid will be a necessary intermediate step to pass through. 
Obviously this would imply a subsequent complete rewrite of the code, and in this rewriting we could take the occasion to: 1 - solve Richards equation with Casulli’s method, and 2 - better integrate surface waters with it. If resources would be available, even freezing soil could be rewritten, and the successful 2011 paper already contains how to do it.

3 - Processes

Among the processes, it would be important if the work by Stefano Dellachiesa on vegetation, and if the work of Florian Marshall on contaminant transport could be really embedded in the GEOtop main stream. My claim for more OO structure above, will go also in the direction to include these achievements, without cluttering the code and/or making more complicate the IO.
Personally, and in perspective, I am interested to include in GEOtop a modeling of the low atmosphere, integrating Navier-Stokes equation. I did some preliminary approach to the problem working conjointly with Dino Zardi and Michael Dumbser. A lot of resources are needed but I can work to grab them.

4 - OMS - Web services

My personal academic interest is also to push further  the evolution of GEOtop’s (beyond step 1) informatics for embedding GEOtop in OMS. This would allow splitting GEOtop  in components that can be joined at run-time through a scripting language, i.e. Groovy. This would enormously  increase the ability to run different modelling solutions, facilitate simulations reports, incapsulate even more the code. I am not claiming here, or pursuing, that this must be the main road followed by the GEOtop community. A certain “genetic variability” in the GEOtop versions and ecology would be indeed of benefit, and in favour of the model eventual survival. A passage necessary for OMS integration will be that task 1above will be pursued. That I think, should be really accomplished right away.

5 - Commitment and Executables

I already manifested my opinion to the developer group. Which is the following: in this moment using GEOtop require the total commitment of the guy/gal the want to embrace it. This is not acceptable if we want that our product would have some spreading. In this moment compiling GEOtop, still remains a challenge for a new users, and even for expert users, because they usually have a lot of other things to do, and they are pissed out when something that they consider trivial does not work as they expect. 
Downloading the source code directly from trunk should work for developers, but not for users. A tagged version, known to compile under the three major operating systems, should be used instead. More than that, a tarred zipped file, with the source code should be made available without passing through git. The best experience for a user would be, however, to download directly the executable for her/his machine and run it. I think that this move, together with improving the manual and documentation, will really boost GEOtop use. And, IMHO is the presence of a lot users that create a fertile soil for businesses.  Certainly a different pathway would be to implement GEOtop as a remote web-service. Recently at CSU,  I was impressed on what te eRams platform can offer, and how it helps in getting data and simulations together, and make easier collaborative work of groups.

6 -Testing

GEOtop comes with a series of tests, and, I verified, all of them compile, and all except one converge to the right solutions. This is useful for making users confident of what they are doing, and developers certain that they did not screwed up the system with their last minute modifications.  However, a special attention should be given to compare GEOtop with the tests provided by the increasing community of process-based model developers. They call their virtual experiment “the superslab” and you can find its definition here.  Being engaged with those guys would be really important, and would bring furhter recognition the our model. We could possibly be able also to challenge that community and propose tests that cope with the cryosphere, since none of its models does it. 

7 - Manual

The manual is not out-of -date but not up-to-date too. Efforts to get it improved would be necessary. I will try to put some students on it, if the occasion comes.

8 -License and Community

GEOtop is GPL and I believe it has to remain GPL. However, not in name of a single Author, but most probably in name of many. The creation of a GEOtop by components, could help in this direction, in the sense that single components, could be copyrighted then by single persons or group different from the original ones. The establishment of some GEOtop foundation with some organisation, and which determines some rules of behavior and/or an etiquette to follow for GEOtop use, will be a desired goal for future. I consider myself not eligible for any position in it though. 

9 - Courses

If a few of the conditions above would be met, it would not be difficult to start some schools either in Europe and in the Americas (or elsewhere). This also should be a goal to be pursued. Schools for Ph.D. students and Researchers, could not be very expensive, however some income can be generated.  Courses for professionals should be more expensive ... but maybe these could be directed to use web-services, and therefore the real income could arrive from subsequent use. 


10 - And then ?


Then it is matter to find resources and man/months to have all of it working. 

Project CLIMAWARE

Today we submitted the CLIMAWARE project: CLIMatic change impacts on future Availability of WAter REsources and hydro- geological risks.
It is a proposal internal to UNITN but it involves so many different disciplines and Departments that represent a variety that normally can be found only in EU projects. Please below, find the abstract of the project, while we cross the fingers for its approval.
                                       


The pressure of human activities on the earth system, including the water cycle, is multifaceted and intrinsically interlinked with climate change due to the inherent complexity of natural and human processes. The interaction of humans with the environment is strongly non-linear and is dominated by multiple feedbacks and non-stationarities, which are difficult to identify and assess. Despite these complexities, the existing tools to deal with the many hazards connected to the interaction of humans with the environment are based on simple approaches, founded on the concept that the earth system is in a steady state, so that predictions on its evolution can be safely made either statistically, or through simplified models founded on the hypothesis the processes are stationary.
This proposal focuses on the interactions between climate change and human activities related to water, with a holistic view embracing physical, social and economic processes. It will consider, in particular, changes in water cycle components related to extremes and their implications in contiguous sectors. Here extremes include floods and land instabilities triggered by extreme precipitations, such as debris-flows and snow avalanches, but also different stress factors threatening the integrity of freshwater services, with adverse effects on agriculture, tourism, and energy production. New paradigms, approaches and tools will be developed in order to cope with the non-stationarity of water cycle processes and to study the entanglement between the physical processes and human activities. The proposal is interdisciplinary also regarding the physical aspects, and involves interaction between water related disciplines, such as climate science, meteorology, hydrology, fluid mechanics, and information technology for aspects related to distributed monitoring. Because of the challenges that climate change imposes, there is a need to increase general awareness of climate-related risks, and find ways to mitigate the impact on economic activities as well as to adapt to new scenarios. For mitigation and adaptation strategies to be effective, knowledge of the physical aspects of the water cycle should be combined and complemented with knowledge of social and economic processes that crucially affect the use and management of freshwater. Furthermore, it should consider that national governments, on the one hand, and the European Union, on the other, have started to enact laws aimed at countering the effects of climate change. The implementation of these measures and their harmonization with existing laws, however, is not without problems. This might require some legal creativity – and the interaction of law with other hard- and social sciences – to bring about effective results.

Project Description, highlighting the interdisciplinarity (excerpt)

Climate change mitigation and adaptation require substantial changes in production and consumption patterns, as well as in individual behaviors. This is particularly true for water- related sectors, which are a linchpin in the production of beverages, food and energy, and for many recreational activities. However, in economics, estimates of water resource availability are rarely made on a solid quantitative basis. Also, the systematic preemptive estimation of extremes caused by hydrological events is a science in its infancy. The recent advances in hydrology and water related sciences open interesting perspective for new methods and models to support a better informed decision making process at all levels, from local to national and international. Moreover, societal and economic concerns receive only marginal attention in the debate [1]. Arguably, the gap between science, society and policymakers undermines the decision-making process.

The interdisciplinary approach of this project, which combines insights from hard sciences with sociology, economics and law, represents an important step towards bridging the gap between science and society, and raising awareness of the benefits of cooperation between disciplines. The project studies the complex interplay between physical and human processes in controlling the distribution, in space and time, of water resources by considering both actual and possible future climatic and societal scenarios. The project is organized in five interconnected Tasks (Figure 1): Task 1 evaluates the drivers of water resource availability; Task 2 assesses climate change and human impacts on water resources and local hydro-geological extremes (underestimated according to a report by WMO [2]); Task 3 addresses the development of suitable tools for describing and modeling environmental granular flow extremes and sediment transport; Tasks 4 and 5 aim for better management of water resources, specifically, Task 4 uses the paradigm of virtual water to assess the regional impact of the food trade, while Task 5 tries to envisage effective societal responses to climate change.

A full description of the project, financed by UNITN5, can be found here.

References

1. Malnes, R.; Environmental Politics 17(4), 660-672, 2008.
2. http://www.wmo.int/pages/prog/drr/transfer/2014.06.12WMO1123_Atlas_120614.pdf

Wednesday, September 10, 2014

My CV and Five Papers that represent me

For who it may concern, I am posting my up-to-date CV here. It collects what I did,  and contains information already present here (where, in addition, you have a link to the papers).

For a shorter version of my CV, please take this.
If I would asked to choose five papers that more represent my work (and me), they would be not the most cited (among them), not those in Journals with the highest impact (well ... among them), and I would select probably:
Obviously I did decent work also between 1992 and 2006. Enjoy!



Monday, September 8, 2014

Improve GEOtop informatics !

I take the occasion of the actual structure of GEOtop main to argue about best practices in programming and introduce the topic of design patterns. For browsing the actual code you can see here , while my more specific comments are available at the GEOtop Developers forum

I argued that a much simpler  form for the main, should be represented by a scheme like the following (dots represent missing code, the code is assumed to be C++ - please be merciful with me: my knowledge of C++ details is very limited -  but, the arguments are for every generic language with support of object orientation):

where three main phases are distinguished: an initialization phase (with allocation and initialisation of variables), an execution phase (where the real stuff is done), and a closure phase (where all the memory if freed, and whatever is needed is performed). This best practice is a suggestion that comes from many recent numerical modeling efforts, like ESMF, OpenMI, OMS and allows an encapsulation of the processes in their own part of the project, but everybody, I think, can appreciate its simplicity and cleanness. 
A structure like this should be not very difficult to obtain in a few days of work of a proficient programmer. This structure does not alter the internal of the code and rationalise what is present now, introduce some encapsulation of the procedure,  but IT IS NOT object oriented (OO) programming.
Object oriented programming has to do with the choices of appropriate classes, and instantiation of the appropriate objects, and doing OO design, for implementing the code.

Who is suited to procedural programming, for instance interprets algorithms as “procedure calls”. My view, and understanding of the matter, is instead that procedures should be instantiation of classes, a.k.a. “objects”. Therefore, a proper evolution of the above could be:

In this second case, the organisation of the code remains the same. However the execution of the single procedure is written differently, i.e., for instance:

EnergyBalance.execute( ); 

This is intended to mean that EnergyBalance is an object and .execute() (please notice without arguments) is a method called on this object. One of the advantage of making the each procedure an object, has several argument in its favour. The first is that each object, now, contains its own inputs as fields, and encapsulation is certainly improved. This makes all the code easier to maintain, and the input data for each of the procedures easy to control and parse (out of the time loop, obviously). For a simpler, but a conceptually similar, example in Java, please see here.
Doing this step is not matter of a day, it certainly require more thinking and a sequence of modifications also in the structure of the data containers actually present in GEOtop which propagates down into the inner routines.

In case, also the print( ) method is part of a class, with the rational that with proper software engineering, it can be sent in parallel with the core calculation, on different processors, and therefore, interfering the less possibile with the ‘core’ computational time. I personally bet that, at moment, a consistent part of computational time of GEOtop is wasted in printing to screen data. So, maybe we spent months to have a fast algorithm for finding the solution, and that we use that time saved to put thousands of number on the screen or to write them to disk subtracting power to the computation. 

Actually, the one above is just the beginning of the story. Opening the way to classes, open the way to use some well known  Design Patterns to make the code more maintainable, flexible and evolvable. One case, obvious to all GEOtop aficionados is that, in reality, either the water budget and the other procedures, are a bundle of different alternatives. With OO there exists at least one neat scheme to treat this case, which is called, the “Strategy pattern” where, the object really chosen is one out of a group of options which are implementation of a common interface  (I am going to use the technical OO slang here). So actually, the "main( )" can be programmed to accept a generic type of solution (algorithm) to that problem (as specified by the abstract interface), and eventually this solution can be chosen at run time among the subclasses that implement it.  Solutions can then be added to the code without the need to modify (almost) anything else of the code than just adding a new subclass.

References (you will not find them easy to read ... but take it quietly, cross-check with the web, and something will pass slowly). Besides the basic, also adressed here fro Java, try to give a look to:

Gamma, E; Helm, R.; Johnson, R; Vlissides, J; Design Patterns: Elements of Reusable Object Oriented Software, Addison-Wesley, 1995

Martin Fowler's, UML Distilled: A Brief Guide to the Standard Object Modeling Language, Addison Wesley, 1999

Freeman, Eric T; Elisabeth Robson, Bert Bates, Kathy Sierra (2004). Head First Design Patterns. O'Reilly Media. ISBN 0-596-00712-4

Bruce Eckel, Thinking in patterns', notes. 

Various Authors, Design Pattern,  from Indian Student Association of University of Nebraska at Omaha
 

Friday, September 5, 2014

Installing GEOtop 2.0 on Mac OS X

I receive from Matteo Dall'Amico and I publish the following instruction to compile GEOtop 2.0 on Mac OS X. For Linux see this other post.  Now, actually, there is a more recent guide that you can find here. Refer to this one just in case of problems.

# Install the following packages (or verify you have them already):

#Install and compile MeteoIO libraries.

This will include to install the Developers tools,  Homebrew and other required libraries. See instructions here

#Check if you have git, otherwise install
brew install git

#Install libraries

brew install pkg-config
brew install boost

# download GEOtop
git clone https://code.google.com/p/geotop/

# create the makefile

If you do not have installed before, from CMake website. When you start it a window as the one below appears:
Please, observe that, on top the path to source code has been specified, and just below, the path to binaries. click the Configure button to configure the Cmake. If not already present, add the path to MeteoIO libraires: /usr/local. After configuration, the screen should appear as below:

To process the file, Press Generate. This will generate, in the directory (folder) selected for binaries, the Makefile

# make the project

In terminal cd to the directories were binaries will be created anc check if the Makefile is present. Issue the command:

$ make

# copy either using Finder Window or Terminal commands the binaries files generated to a standard place.
I chose "/usr/local/GEOtop".
This directory is actually not a standard one, However, I placed a symbolic link into /usr/local/bin, which is a directory usually already present in the Terminal path.

$cd /usr/local/bin
$ln -s ../GEOtop/geotop geotop

#try geotop example. Due to previous operation

cd to the directory (folder) tests/test_sample_run/small_example (for instance this, among the others)
Issue:


$ geotop ./

If you did it well and the goddess of programmers assisted you, it should work, as it did for me.

Installing meteoIO on Mac OS

Matteo Dall'Amico (see previous post on GEOtop installation) also provided instructions for compiling MeteoIO on Mac and Linux). While Mac instructions can be found in this post, Linux instructions can be seen in a previous post

# Check the environment compiler: You are suppose to have installed the appropriate Developers tools (search for them in the App store). Be sure that you also download the command line tools.

#For Mac OS use Homebrew  (which needs to be installed first)
#Then write at the console, for instance for Make:
brew install make

#For any other package:
brew install subversion
brew install gdal
brew install proj
brew install doxygen

#install CMake from CMake website

# After having selected your directory for installing meteoIO libraries with the usual console commands
# download the source code from the SVN:

svn co https://models.slf.ch/svn/meteoio/trunk
#Start Cmake and a window like the one below will appear:


# Please observe that you have to indicate the directory (folder) where the source code is
# and the directory where you want the binaries. Is in this directory that the Makefile will be produced
#Click on Configure and a window like the following should appear:

#Please note that you can modify the flags just clicking on them. Be sure that you have also checked the proj.  The following fields should be "ON"


BUILD_SHARED_LIBS
PLUGIN_A3DIO
PLUGIN_ARCIO
PLUGING_ARPSIO
PLUGIN_GEOTOPIO
PLUGIN_GRASSIO
PLUGIN_PGMIO
PLUGIN_SMETIO
PLUGIN_SNIO
PROJ4 (then you have to verify the paths to LIBPROJ and to LIBPROJ_INCLUDE_DIR)

All the rest can be set to "OFF".

# Then, press  the butto to generate the makefile and exit from the console.

# In the terminal, goes in the directory where you create the Makefile and type:

make install

# (with "sudo" if you want to install it as super user).

Now move the libraries into /usr/local/lib.  They shoud be libmeteoio.2.4.3.dylib and the two symbolic links: libmeteoio.2.dylib and libmeteoio.dylib

After having installed meteoIO, in the needs to understand what it does, one can browse the examples in the "doc" directory. However, s/he can also built the doxygen documentation. For doing it, please first check you have doxygen installed. Subsequently:
  • using a terminal move to the directory where the meteoIo source code is located. 
  • execute "doxygen -g <configfile>" to create a configuration file, "configfile" for the project
  • execute doxygen <configfile> to create the documentation
  • Find the documentation in a new html directory created
  • Access it by clicking on the index.html file

Thursday, September 4, 2014

Atlas of Mortality and Economic Losses from Weather, Climate, and Water Extremes (1970-2012)

With the limits that these large scale project have, which is neglecting most of the micro-phenomena, as small landslides, that causes huge economical losses, and several life losses, this is, however, a reference to keep in mind when talking about climate related hazards.

It was produced by WMO and it is available here. As it reports in introduction:
"Weather, climate and water-related disasters are on the rise worldwide, causing loss of life and setting back economic and social development by years, if not decades. From 1970 to 2012, 8 835 disasters, 1.94 million deaths, and US$ 2.4 trillion of economic losses were reported globally as a result of hazards such as droughts, extreme temperatures, floods, tropical cyclones and related health epidemics, according to a new report."

Tuesday, September 2, 2014

New Course on statistics and data inference on Coursera

Verbatim from DataCamp Blog:

" Yesterday (Monday 1st of September), a new session of Data Analysis and Statistical Inference, taught by Doctor Mine Çetinkaya-Rundel from Duke university, has started on Coursera. Just like with the previous run, all labs take place in DataCamp’s interactive learning environment.

Data Analysis and Statistical Inference will teach you how to make use of data in the face of uncertainty. Throughout the course, you will learn how to collect, analyze, and use data to make inferences and conclusions about real world phenomena. No formal background is required, but mathematical skills are definitely a plus."

Further information here.

Compiling GEOtop with Eclipse (on Mac)

[The three  operations below are in common with the compilation of GEOtop by using a makefile, and they are better described here in a previous post]

1- Follow the instructions to install meteoIO. Install and compile it. 

This create the dynamic libraries to be linked to GEOtop. Saving the libraries in a “standard place” could be useful. The other information to annotate is the position of the meteo include files (/usr/local/include, at least under Mac OS) since Eclipse needs them to compile the code.

2 - Install the boost libraries

3 - Download GEOtop from the Git:

[The above operations are in common with the compilation of GEOtop by using a makefile, here]

4 - Set up the Eclipse project^1 

Select:

File -> New -> C++ Project 




Then follow instructions from the pop-up window. If you never did it before, you better think, to browse Eclipse on-line help. 

Next you have to indicate the path of the meteoio libraries and includes. These are not the only one includes that are supposed to be made visibile, as you will learn below. 

Right click on the project. From the menu, select ‘Properties’. A pop-up window will appear. Select and open (clicking on the triangle) the C/C++ General Menu. Go and select ‘Path and Symbols’. On the right side of the pop-up window will appear. The window presents several button. The ones important for us are the ‘Include’ button and the ‘Libraries *’ buttons. 
If you click on ‘Includes’, a selection of languages will be shown. Select GNU C++ (that’s my case). Click on the ‘Add’ button at the right side of the window for adding libraries. A further pop-up menu will appear.

Select the ‘File System’ button. This will drive you to the selection of the directory you need.
Repeat the operation all the times it is required. 

Subsequently, always in the same pop-up menu, click on ‘Libraries’. Click on ‘Add’ and add the names of the libraries required: meteoio, meteoio.2.4.3, meteoio.2. These names correspond to the files called lib*.dylib where the ‘*’ stands for the name of the library required. Repeat the operation for each one of the libraries name. [I do not think it is necessary to link all the three libraries, since the real library is one. The others are symbolic links. However, I did not experimented differently]. 

Third, go back on the Path and Symbols window, and select ‘Libraries Paths’. With the same type of procedures as before, please add the path to the libraries. 

Then, you have to import manually the file into Eclipse. GEOtop “core” related files are contained in the root folder of the project, i.e. only the file ‘config.h’, and in the directory: ‘geotop’, ‘gt_utilities’, ‘libraries’, ‘meteoio_plugin’. Directory (folder) ‘libraries’ has two subdirectories (subfolders): ‘ascii’ and ‘fluidturtle’.  Therefore, these directories needed to be created inside the just created GEOtop folder:

File -> New -> Folder, as it appears in the Figure below


Now, what you have to do is to populate the folders with the files:

Right click on the folder in the Project Explorer -> Select Import, and the following window will appear:

Select ‘File System’, and then ‘Next’. Then follow the instructions. Clicking on the ‘Browse’ button a pop-up window will guide you. 

Another operation to consider is that some include files in the code were declared with < > instead than  “ “. This implies that these includes are recognised as “system include” or of this type. Therefore the path to these include has to be added to the project. This addition works the same way as in point #4. The files of which we are talking about  are 'inputKeywords.h' and  'config.h' that has to be added since it has been place directly in the root directory. [ I am not sure that this is necessary but, at the moment, I could not find a better solution. ]

Now you should be almost done. Just


5 - Build and Compile.  [To do it, get the instructions here. When I did it, I got a conflict in linking which was solved by observing that there were some duplicate classes. I just cancelled:part_snow.cc and part_snow.h from the project]

_________________________________________________________________________________
^1 - One would probably get the project directly inside Eclipse using Egit. This is not an option at moment. But I am thinking to create a clone of the git from where it will be possible. In meanwhile, you can get the project from here.