Workshop participants (left to right): Jim Jones, Ioannis Athanasiadis, Dean Holzworth, Rob Knapen, Julien Cufi, Domi Ripoche, Sander Janssen, Mauricio Fernandes, Johnny teRoller, Chris Villalobos, Meng Zhang, Cheryl Porter, Yunchul Jung, Dirk Raes
An AgMIP Development Sprint was held in Gainesville, Florida, USA July 9-13, 2012. The purpose of the workshop was to develop AgMIP data translation tools which would allow data from the AgMIP Crop Experiment (ACE) database to be exported into model-ready formats needed by the APSIM, AquaCrop, STICS and WOFOST crop models. Crop model developers were paired with java programmers to rapidly develop the data translation tools using open source techniques. These development tools allow AgMIP crop modellers, at subsequent AgMIP modelling workshops, to run their models using the same experimental data, hence allowing true inter-model comparisons. In addition to the software development, the group discussed and improved the standardized format of an AgMIP Crop Experiment Data Structure using JSON (Java Script Object Notation). Examples of this data structure are posted elsewhere in this wiki.
The following teams and participants were represented:
- APSIM - Dean Holzwoth (CSIRO, Australia), Ioannis Athanasiadis (Democritus University of Thrace, Greece) and Mauricio Fernandes (EMBRAPA, Brazil)
- AquaCrop - Dirk Raes (KU Leuven University, Belgium) and Rob Knapen (Alterra, Wageningen UR, Netherlands)
- STICS - Domi Ripoche and Julien Cufi (both from INRA, France)
- WOFOST - Johnny te Roller and Sander Janssen (both from Alterra, Wageningen UR, Netherlands)
- DSSAT - Cheryl Porter and Meng Zhang (both from University of Florida, USA)
- User Interface - Yunchul Jung (University of Florida, USA)
- Overall support - Chris Villalobos (University of Florida, USA)
The structure of the meeting allowed for daily discussions on an AgMIP-related topic, as outlined in the Agenda. In addition, each day included time for review of the previous day's activities, team progress, difficulties encountered and goals for the day. This format provided an excellent means of "course correction" to improve the final products for all teams and to streamline the process for future development sprints.
The AgMIP Crop Experiment Data JSON structure is divided into 6 sections:
- metadata and location data
- initial conditions
- management events
- observed data
Each of these sections provides data that describe a historical field experiment in sufficient detail that a crop model may be used to simulate the experiment. For very extensive experiments, this may include much more data than would be used by a typical model. However, more often, there is a lack of data to drive the models and assumptions must be made. Each of the crop modeling teams is encouraged to document the assumptions in a text document that will accompany the translated crop model input files to inform the crop model user. The crop modeler may then elect to accept the assumptions or override them with expert knowledge of the experiment. Use of this wiki to document model input assumptions is also encouraged. A sample documentation for DSSAT can be seen here.
The ICASA JSON object that resulted from the workshop will be considered v1.0 and is to be fully documented in this wiki.
The progress of each team is briefly described below.
By the end of the week, the APSIM team had a working first version of a translator that was capable of writing APSIM formatted files containing weather, crop, soil and management experimental data. While not completed, it provides a solid foundation for further development. The team was able to produce model-ready inputs, which were then used in a simulation. The team used a Java plug-in, Velocity, to use templates to quickly replace values with template variable codes. This allowed the data translation tools to be developed with a minimal amount of programming, thereby decreasing development time. The source code can be accessed from https://github.com/agmip/translator-apsim.
The APSIM team autogenerated classes for the ICASA data object using Jackson JSON processor. Weather data were translated to an APSIM .met template using the velocity template engine. Metadata, field data, management events, initial conditions, and soils data were similarly inserted into the .apsim template.
Additional work is required for testing with additional datasets, more crop types, dealing with missing data, writing an warning file for the user, writing outputs to the ACMO format.
The data extracted from the JSON object need to be stored into a number of model-ready formats (i.e. AquaCrop data files) which are accessed by the model during simulation runs. AquaCrop uses the following files, each which has a specific format:
- Climate files (file with extension CLI),
- Air Temperate files (extension TMP),
- Reference evapotranspiration files (extension ETo),
- Rainfall files (extension PLU),
- CO2 files (extension CO2),
- Crop files (extension CRO),
- field management files (extension MAN),
- irrigation files (extension IRR),
- soil profile files (extension SOL),
- groundwater table files (extension GWT),
- files with initial (extension SW0) and
- off-season (extension OFF) conditions, single (extension PRO) and multiple (extension PRM) project files, and
- measurement files (extension OBS).
The following tasks were achieved at the workshop:
- Installation of development tools for AgMIP;
- Documentation (word documents) for translation tools for the climate (CLIM, ETo, PLU, TMP), soil profile (SOL), groundwater table (GWT), field management (MAN), irrigation (IRR), and initial soil water content (SW0) files;
- Data translation tools (using Velocity Engine and templates) for:
- Climatic files (Clim)
- Preliminary version of an ETo calculator (utility tool)
- Reference evapotranspiration files (ETo)
- Rainfall files (PLU)
- Air temperature files (TMP)
- Tables and conversion rules for initial soil water content
- Initial soil water content files (SW0)
The following tasks are under development or remain to be done:
- Partially complete data translation tools for:
- Soil files (SOL)
- Irrigation files (IRR)
- Field management files (MAN)
- Groundwater files (GWT)
- Data translation tools for:
- Crop files (CRO)
- Project files (PRO and PRM)
- Measurement files (OBS).
- Tables and conversion rules for soil profile characteristics (discussion in cooperation with STICS team)
- Documentation for all files and applications
Planned follow up by FAO’s AquaCrop team. FAO will give further documentation for the development of rules for the data conversion tools and the translation tools for crop model output (ACMO). It will be checked if FAO has the expertise in house to develop the data translation tools. If expertise is not available in FAO, it should be checked if expert(s) can be attracted from outside FAO (Alterra ?).
Document a generic utility tool to estimate reference evapotranspiration (ETo) from climatic data. The tool should be able to process data available in a wide variety of units and specified in various commonly used climatic parameters which might be available in the ACE database. Guidelines described in FAO Irrigation and Drainage Paper Nr 56 and used in FAO’s ‘ETo calculator’ will be fully documented. The document will be send to Cheryl Porter for follow up and translation in a generic utility tool;
Fine tuning of conversion rules used in the data translator tools (e.g. estimation of soil fertility stress (used in AquaCrop) from the fertility data available in the ACE database).
Source code for AquaCrop data translation tools can be accessed at https://github.com/agmip/translator-aquacrop.
The STICS team was able to complete the conversion of metadata, field data, weather and initial conditions. Work on translation of soil data is well underway, but not complete. A separate utility was written by the STICS group to flatten the soil layer data from the ICASA format to a maximum of four layers. This utility will be shared as a generic utility for use by other crop model translators, including AquaCrop. Work was begun with the velocity template routines which were also used by the APSIM group to fill in model parameters in a template management file. The STICS model does not require the use of observed data and so these will be ignored by the translators for now.
The work remaining is estimated to take approximately 2 weeks plus additional time for testing. Generation of the crop model outputs file will take additional time.
The STICS data translation tool repository is located at https://github.com/agmip/translator-stics.
The WOFOST team was at a distinct disadvantage in that they did not have expert knowledge of the WOFOST crop model. Most of the week Johnny te Roller worked solo on building the infrastructure of the data translation tools using the Velocity template engine. He was able to complete a data translator for the weather data. The template files were constructed and the overall structure of the translator was developed. But the mapping of variable names from ICASA to WOFOST was not possible at this workshop. This will require the help of the WOFOST crop model development team. Follow-up on accomplishing this and the remaining programming tasks will be initiated by Sander Janssen.
The GitHub source code repository for WOFOST data translation tools can be found at https://github.com/agmip/translator-wofost.
The changes to the JSON object that resulted from the discussions during the workshop will require modifications to the previously developed DSSAT data translation tools. The projected tasks for finalizing DSSAT tools are:
- August 1
- complete DSSAT import and export translators
- populate the ACE database with at least 10 JSON objects
- finalize format of ACMO.csv file for harmonized crop model output data
- August 15
- online ACE database demo available
- December 1
- ACMO data translation tool functional
DSSAT data translation tool source code is maintained on the GitHub/agmip repository at https://github.com/agmip/translator-dssat.
Recommendations and Discussion Items
- A generic translation tool should provide text values for the ICASA codes.
- AgMIP should consider the development of more generic utility tools such as general accepted pedo-transfer functions to estimate soil water contents at saturation, field capacity (upper limit) and permanent wilting point (lower limit) from soil textural data.
- The generic utilities should be provided for quality check of data in the data base (e.g. look for large deviations between listed soil water contents and generated soil water content from soil textural data).
- Specific data required by models other than APSIM and DSSAT might not be available in the date base, requiring a lot of assumptions to be made and requiring data conversion. This can make fully automated model runs difficult/dangerous, especially if the generated data files are not carefully checked the simulation runs. The results might suffer from wrong input. Data translation tools should include generation of a warning file that list the assumptions so that the crop modeler may revise inputs as necessary.
- Many assumptions had to be made to generate input for AquaCrop because the ICASA database does not contain the required variables. It might be considered to increase the variables in the ICASA database (see recommendation 1). (Note from CHP: the ICASA database is a living document. We will continue to update and add variables as needed. However, that is no guarantee that those parameters will exist in the datasets that are stored. Model-specific data translation tools will still have to make assumptions in most cases.
- Cultivar parameters are model-specific. There are no direct means of translating these parameters from one model to another. Crop modelers will need to decide how to handle cultivars for each specific model. We may need to generate some generic descriptions for cultivars based on maturity group, vernalization requirements, relative maturity durations, etc.