#########################################
Cookbook for the ParFlow JUBE environment
#########################################

*************
Prerequisites
*************

- execution on JURECA and JUQUEEN
- use the standard module environment provided by on JURECA or JUQUEEN
- knowledge of the hydrologic model ParFlow

*************************
Setup of your environment
*************************

1. Unpack this archive under ``JUST:$WORK``; cd into the test directory. There are run control, preprocessing, postprocessing and benchmark driver files under driver/, there are template files for parflow input, platform, job submission and profiling, and there are parflow model examples located inmodel which are run by the various examples located in custom/  
2. Edit and adjust custom xml files as necessary under custom/: and there is a further example: ``vim ParFlowJUBE2ctrl.xml``
3. To run using the driver files use the following python script: jubeRun_script.py. To run using profiling use driver/ParFlowJUBE2_BMBase.xml. Eg: python jubeRun_script.py 'idealised_100x100x50_48taskspernode_jureca' 'ParFlowWeakScalingBM' 'ParFlowJUBE2_BMBase.xml' 'ParFlowBM_Runs' 'weakScalingIdealised_Job_Jureca.xml'
4. Postprocessing: To get the timing result or user nominated result, collated by JUBE2, run the following python file: jubeResult_script.py


   - PDF summary: ``darshan-job-summary.pl 'darshan.log'`` > does not work at
     the moment
   - TXT summary: ``darshan-parser 'darshan.log' > darshan.log.txt``

   - Extrae does not work at the moment on JUQUEEN
 
   - Alinea does not work at the moment on JUQUEEN

   - Full list of profiling tools available for each platform is in templates/profiler.xml

With the compilation one might choose whether the tgz should be downloaded, or the tgz should be extracted,
whether a clean compile is desired and whether a compile is to be done at all.

ParFlow is controlled via the TCL-script. The ParFlow compilation consists of
two steps: the pf-simulator and the pf-tools.

Running JUBE2 will create an ``outputDir`` directory as well as an xml file.
It will also output the results to the newly created $WORK/JUBE directory.

********************************************
How to run the ParFlow testcases
********************************************

The standard modeluration JUBE comes with a sinusoidal testcase, a eurocordex clm model, a eurocordex precipitation minus evaprotranspiration model, a eurocordex ponding model and the little washita test case. All these cases can be run with existing custom xml files in custom/

ParFlow r693 is compiled
with ``-O3 -xCORE-AVX2`` on JURECA and ``-O3 -qhot -qarch=qp -qtune=qp`` on JUQUEEN, without clm, and undistributed file output, by default.

You can nominate weak or strong scaling in the custom files (templates/scaling.xml) and use the driver file driver/ParFlowJUBE2_BMBase.xml. The parameter ``scaling`` introduces a nice concept of JUBE:
`parameter dependencies <https://apps.fz-juelich.de/jsc/jube/jube2/docu/advanced.html#parameter-dependencies>`_.
In the job submit step JUBE will substitute the flags in the tcl that we can
influence.


You will find the resulting output data in the respective JUBE benchmark directory.

************************************
Customizing the benchmarking process
************************************

If you open the custom xml and driver xml files you will see a lot of parameters
that influence the testcase type, compilation and execution of ParFlow.

To understand the concepts you should at least be familiar with the
`JUBE tutorial <https://apps.fz-juelich.de/jsc/jube/jube2/docu/tutorial.html>`_.
There will be content that also focuses on the
`Advanced tutorial <https://apps.fz-juelich.de/jsc/jube/jube2/docu/advanced.html>`_.


User defined vars for ParFlow
-----------------------------

``nx``, ``ny`` and ``nz`` define the subdomain every task has to solve.

With the ``calcnx``, ``calcny`` and ``calcnz`` parameters you can set the way the
``Computational Grid`` for ParFlow is calculated. ``NP``, ``NQ``, ``NR`` are the
amount of tasks in x-, y- and z-direction.

Feel free to change the subdomain for your tests or get creative with the
``Computational Grid`` calculation.

You can influence the time variables and output format of the
produced ParFlow data. If you want to change other more specific ParFlow
parameter you will have to add to the substitution parameters of the ``templates/ParFlow_model_input.xml``. 
You are able to specify these ParFlow parameters because they directly influence the results of the profiler
(hence the benchmarking environment) and enable you to quickly increase/decrease
the problem size. 

The real data test cases have a predifined problem size driverd on the topography and geometry files
so these must remain unchanged.

You have the option to output either ``Pfb`` or ``Silo`` files. When you choose an output format be careful to only set one value to ``True``.
JUBE will not check these values and the ParFlow run will produce errors.

Using profiler
--------------

Enabling the usage of different profilers on the fly was one of the main
motivations to set this JUBE environment up. You can just change the
``$profiler`` parameter to your desired tool and it will be instrumented at
runtime.

.. caution::
	If you switch the profiler regularly you have to think about how ParFlow was
	compiled. For example, if you want to use ``Score-P`` it needs to be instrumented at
	compile time. So whenever you switch profilers, think whether you need to set your
	``PARFLOW_COMPILE`` parameter to 1.

This are the available options:

========== =========== ========
 Profiler   Value       result
========== =========== ========
 Allinea    'allinea'   |done on JURECA|
 DARSHAN    'darshan'   |done|
 Extrae     'extrae'    |done on JURECA|
 Memusage   'memusage'  |done on JUQUEEN|
 Vector Advise 'vectoradvise' |done on JUQUEEN|
 NONE       'none'      |done|
 Score-P    'scorep'    |done|
 time       'time'      |done|
========== =========== ========

To use the produced profiler data (especially ``Extrae`` and ``Score-P``) you
should make yourself familiar with the tools that can read the data:

- `Cube <http://www.vi-hps.org/tools/cube.html>`_
- `Paraver <http://www.vi-hps.org/tools/paraver.html>`_
- `Periscope <http://www.vi-hps.org/tools/periscope.html>`_
- `Scalasca <http://www.vi-hps.org/tools/scalasca.html>`_
- `TAU <http://www.vi-hps.org/tools/tau.html>`_
- `Vampir <http://www.vi-hps.org/tools/vampir.html>`_

In general the whole `site <http://www.vi-hps.org/>`_ focuses on improving
the finetuning and optimal hardware usage for HPC applications.

Allinea
^^^^^^^

See the `homepage <http://www.allinea.com/products/allinea-performance-reports>`_
for detailed information about Allinea performance reports.

When you choose this profiler, after ParFlow has finished Allinea will produce two
job summaries. One pure ASCII ``txt`` file and a ``html`` file you can view in
your browser.

This summary can give you a quick overview in which way your program utilizes
the available hardware. It focuses on the four topics CPU, MPI, I/O and Memory.
See this `example <http://www.allinea.com/sites/default/files/uploads/products/allinea-performance-reports-cp2k_256p.pdf>`_
from Allineas website to get the idea. The content of the pdf file equates to
the ``html`` output.

DARSHAN
^^^^^^^

See the `homepage <http://www.mcs.anl.gov/research/projects/darshan/>`_
for detailed information about DARSHAN.

DARSHAN is scalable HPC characterization tool that focuses on the application
I/O. It will produce a binary logfile which by default will be converted to an
ASCII log file through the JUBE environment. You can do this manually this way:

.. code-block:: bash

	darshan-parser.pl darshan.log > darshan.txt

There is also a command to create a nice ``pdf`` file but this is not available
on the JURECA machine due to a missing pdflatex dependency. If you want to
create the ``pdf`` file locally nonetheless you can do it this way:

.. code-block:: bash

	darshan-job-summary.pl darshan.log

Extrae
^^^^^^

See the `homepage <http://www.bsc.es/computer-sciences/extrae>`_
for detailed information about Extrae.

Extrae uses different interposition mechanisms to probe the target application
and gather the applications performance data. It produces trace files that are
aimed to be used by `Paraver <http://www.bsc.es/computer-sciences/performance-tools/paraver>`_

In this environment Extrae uses the Linker preload interposition mechanism to
gather the data.

It is possible to fully customize the filter for the gathered data. The filter
that is used in this environment is located under
``profiler_data/extrae/extrae.xml``.

NONE
^^^^

Like the name suggests this does nothing additionally. JUBE just generates the
ParFlow datadriver and runs the problem.

Score-P
^^^^^^^

See the `homepage <http://www.vi-hps.org/projects/score-p/>`_
for detailed information about Score-P.

Score-P is another tracing/profiling tool for HPC applications. It directly
supports `Scalasca <http://www.scalasca.org/>`_,
`Vampir <https://www.vampir.eu/>`_,
`Periscope <http://periscope.in.tum.de/>`_,
and `TAU <https://computing.llnl.gov/code/tau.html>`_. Score-P compiles an
application with its own wrapped compiler (hence the caution on the usage of
profilers within this environment) and gathers trace data driverd on the filter
located under ``profiler_data/scorep/filter``.

Memusage
^^^^^^^^
This tool is driverd on the routine given in: http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUQUEEN/FAQ/juqueen_FAQ_node.html
This routine is incorporated into richards solver routine, upon compilation of ParFlow. Once all the arrays 
have been set up, it ascertains and prints out how much memory is used on each processor into the ParFlow log file.

Vector Advise
^^^^^^^^^^^^^
See `Vector Advisor <https://software.intel.com/en-us/get-started-with-advisor>`_
for detailed information about Vector Advise

Vectorization Advisor is a vectorization analysis tool that lets you identify loops 
that will benefit most from vectorization, identify what is blocking effective 
vectorization, explore the benefit of alternative data reorganizations, and 
increase the confidence that vectorization is safe.

***************
Result timings:
***************

If you include the analyse and result JUBE steps into your ParFlow run, this adds a ``time`` command before
``srun`` is executed. This enables you to use the jubeResult_script.py to compile the timings of your run/s into
CSV format.

******************************
Use your own ParFlow testcases
******************************

You need to add your tcl files, distribution scripts and associated files to the 
model/, templates/ParFlow_model_input.xml and corresponding driver files. There is a shell cmd option 
in the parflow input struct in the custom files so that you can do a bit of preprocessing
if needed.

Parflow with CLM
^^^^^^^^^^^^^^^^
At present clm compilation can only be done on JURECA.

**********************
Use a different system
**********************

If you want to use this environment on a different system than either Jureca or Juqueen, you most
likely will have to change the ``env.ini`` and the ``ParflowInstall.ksh`` files- see jureca_files/ 
as an example. Either change the versions of the desired software modules or change the 
module loading completely. Don't forget to change the module load commands in the 
``templates/profiler.xml`` to get the profiler on your system working.

You'll also need to check your JUBE installation provides a ``platform.xml``
for your system (check the ``JUBE_INIT_PATH``). If it doesn't you can orientate
yourself with the ``templates/platform.xml`` file and work out what you need to add to the 
relevant structs yourself.

***************************************
How to check your personal JUBE changes
***************************************

The fastest and probably easiest way to check a JUBE xml file for syntax
errors is with ``xmllint``. The following command will only output the errors it
finds with your xml file:

.. code-block:: bash

	xmllint --noout --dtdvalid $EBROOTJUBE/schema/jube.dtd ParFlowJUBE2ctrl.xml

It is important that you tell ``xmllint`` which ``dtd`` file it should use. In
this example I pointed it to the schema file of the standard JUBE installation.

****************
Copywrite Notice
****************

Copyright (C) 2016  Wendy Sharples, Lukas Poorthuis, Klaus Goergen

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see <http://www.gnu.org/licenses/>.

.. |done| unicode:: U+2713
