The fourth AgMIP Development Sprint, also known as the Austin Hackathon, was sponsored by the iPlant Collaborative. The event was held at the Texas Advanced Computing Center (TACC) on February 4-8, 2013 and was focused on developing the software needed to translate data from the AgMIP Crop Experiment (ACE) database to model-ready formats for seven crop models.This event was similar in format to previous AgMIP Development Sprints where participants work in small groups consisting of both model developers and software developers. The combination of great venue, talented and focused participants, and real, working software tools generated during the week, made this a very successful development sprint.
L to R: Dean Holzworth, Ioannis Athanasiadis, Kelly Thorp, Henri Songoti, Dirk Raes, Naveen Kalra, Julien Cufi, Sander Janssen, Chris Villalobos, Cheryl Porter, Roger Nelson, Meng Zhang, Rob Knapen, Domi Ripoche, Jeff White, Johnny teRoller, Sudheer Pamuru, Daniel vanKraalingen (kneeling), John Fonner, Dan Stanzione
The development teams at the Austin Hackathon were:
- APSIM - Dean Holzworth (CSIRO, Australia) and Ioannis Athanasiadis (Democritus University of Thrace, Greece)
- AquaCrop - Dirk Raes (KU Leuven University, Belgium) and Rob Knapen (Alterra WUR, Netherlands) and
- CropSyst - Roger Nelson (Washington State University, USA)
- InfoCrop - Naveen Kalra (Tata Chemicals, India) and Sudheer Pamuru (Concat Systems, Inc., USA)
- SarraH - Henri Songoti (Agrhymet, Niger) and Meng Zhang (University of Florida, USA)
- STICS - Domi Ripoche (INRA, France) and Julien Cufi (INRA, France)
- WOFOST - Sander Janssen, Johnny teRoller and Daniel van Kraalingen (all from Alterra WUR, Netherlands)
In addition to the data translator development teams, who worked directly with the software development, the following additional support roles contributed to the workshop:
- Technical support - Chris Villalobos (University of Florida, USA), Cheryl Porter (University of Florida, USA), Steve Welch (Kansas State University, USA) and Jeff White (USDA-ARS, USA).
- ICASA standards review and updates - Jeff White and Cheryl Porter
- Demonstrations of large-scale computation using crop models on the TACC - Kelly Thorp (USDA-ARS, USA), Joshua Elliot (University of Chicago, USA), John Fonner (TACC, USA)
- Institutional support - Jim Jones (University of Florida, USA), Dan Stanzione (iPlant, USA), and Jay Boisseau (TACC, USA)
Several of these groups have worked together at previous development sprints and worked to expand and enhance the capabilities of their applications at this workshop. For example, the APSIM team worked together at the Gainesville Development Sprint in July, 2012. This team, consisting of Dean Holzworth and Ioannis Athanasiadis, has the distinction of being the only team at any of the sprints to have a functioning translator by the end of the week. That does not mean that their job was finished. During the Austin Hackathon, they fixed several issues discovered by crop modelers at previous AgMIP crop modeling workshops. These fixes enhanced the software's usability and robustness. They also implemented software to convert simulated model outputs to a harmonized format so that the economic modelers can use these data as input to their models (see description of AgMIP translation products here). Finally, Dean and Ioannis also commenced work on a tool that can "reverse translate" model input data and to the harmonized AgMIP format so that datasets which are already in APSIM format can be shared with the entire AgMIP Community through an online database. This tool, while not complete, allows APSIM soils and weather files to be imported into the AgMIP database. Future effort will focus on this tool.
The STICS development team also attended the workshop in Gainesville, Florida and has a working translator for their model. The team of Domi Ripoche and Julien Cufi, both of INRA in France, worked to enhance the existing capability of the translators, to link these with the desktop user interfaces for AgMIP data translation and to create the ACMO translators for simulated model outputs. Julien also spent some time creating a generic function to aggregate soil properties. The team began work on an ACMO translator to read STICS simulation outputs and harmonize the data for import to the ACMO database. The team must fix a few minor bugs, finish the ACMO translator and handle observed data.
The AquaCrop model was also represented at the Gainesville Sprint. The translators for this model, developed at FAO, are crucial to the success of several of the AgMIP regional teams which are performing integrated assessments in Sub-Saharan Africa. Dirk Raes, one of the model developers teamed up with Rob Knapen, a programmer from Alterra, Wageningen UR to develop these data translation tools. Because of some of the unique challenges facing this group due to the more generalized format of some of the input data, we did not anticipate that the task would be completed at this workshop. However, the team was able to bring the translator up to current AgMIP standards for datasets consisting of multiple sites and multiple years. They also helped with the creation of generic functions for calculation of evapotranspiration and other energy balance variables and soil layer aggregation. The team added field management events to the translator and mapped additional ICASA variables with the Aquacrop model inputs. It is estimated that the translator can be completed and tested with an additional 2 weeks of programming work and 1 week of model developer work. This is being coordinated with FAO for spring or summer of 2013.
Sander Janssen, Johnny teRoller and Daniel van Kraalingen, all of Alterra, Wageningen UR, continued the work started by Johnny teRoller at the Gainesville Sprint on the data translators for the WOFOST model. This model is in use in Europe and other parts of the world, particularly with the Global Yield Gap Atlas project. This group also updated the previous translators to work with multiple sites and multiple years of data in a single object. They also modified the WOFOST model to handle batch simulations in a command line mode. The current translator is functional for production level 1 only with no irrigation or fertilizer inputs. This group wrote the ACMO translator into the WOFOST model and it is complete.
The smallest software development group consisted of just one person, who is both a model developer of the CropSyst model and is also an accomplished software developer for numerous applications related to agricultural systems. Roger Nelson attended the development sprint in Ames, Iowa in September 2012 and worked at the Austin Hackathon to expand and enhance his existing data translation tools for CropSyst. He worked mainly on handling initial conditions data, observed data and testing the translator with additional data sets. This translator works in a different way than others being developed in that the raw AgMIP-format data are supplied to CropSyst and the translation will occur within the CropSyst user interface, rather than into the AgMIP translation tools. This was done so that Roger could use his existing data utilities written in C++. Future work on the CropSyst translators includes more testing using additional datasets and development of an ACMO translator.
New data translators will be started for the InfoCrop and SarraH models at this development sprint. Both of these models are proposed to be used by AgMIP research teams for integrated regional assessments.
AgMIP teams in South Asia will use the InfoCrop model, developed at the Indian Agricultural Research Institute. Naveen Kalra, one of the developers of the model, worked with Sudheer Pamuru to develop the data translation tools. They were able to develop the entire framework of the translator for weather, soils and management data using a template approach. This framework must now be populated with the appropriate ICASA variable names. The mapping of variable names will be done by Naveen Kalra when he returns to India. Additional work includes adding management variables appropriate to rice modeling such as puddling and flooded field management, bed planting, multi-nutrient effects and model improvement for InfoCrop. (It should be noted here that handling of the unique management events for rice is an area that all AgMIP teams need to address. It is particularly relevant to InfoCrop, which deals with South Asian systems.) ICRISAT has indicated that they will help to maintain the InfoCrop data translation tools.
AgMIP teams in Sub-Saharan Africa have proposed use of the SarraH model, developed by CIRAD in France. Henri Songoti of Agrhymet in Niamey, Niger, is very familiar with this model and worked with Meng Zhang to develop the data translation tools for this model. This team also used a template approach and was able to develop the entire framework of the translator for weather, soils and management data. Henri Songoti was to meet with the developers of the model at CIRAD later in February to map the model variables to ICASA variables and populate the framework with the appropriate variable names. This data mapping exercise will form the bulk of the work required to finish the translator.
In addition to work on specific models, there were several inter-model working groups to discuss the development of common functions that are of use to all or several of the models and other topics of general interest:
- Soil layer aggregation. This common function was written by Julien Cufi and takes layered soil property data from the JSON object and aggregates to the number of layers required by the calling model translator. Concentration data are aggregated using a weighted average, while mass data are summed. The algorithm determines the layer thickness based on profile thickness and total number of soil layers needed.
- Reference evapotranspiration. A common function was discussed that would compute reference evapotranspiration by various different algorithms depending on the input data provided. The methods used are outlined in FAO-56 (Allen, et al., 1998) and will also generate other weather-related variables including vapor pressure deficit, dewpoint temperature and relative humidity. The methods were outlined by Dirk Raes and Roger Nelson and the function will be programmed by Chris Villalobos.
- Computation of TAV and TAMP, average annual temperature and average amplitude of temperature. Several of the modeling groups use these parameters to estimate temperature through the soil profile. Chris Villalobos will write a common function for use by all models.
- Storage and curation of crop and cultivar coefficients. A means of registering, linking, storing and retrieving data associated with crops and crop varieties was discussed. The library must be flexible enough to handle data of different formats and types, including both documents and JSON objects. Metadata would allow data to be linked to actual experiments and simulated outputs and users would be able to search and retrieve crop coefficients for new simulations based on the experience of other AgMIP researchers. The concept will be refined after discussion with AgMIP crop modelers.
- The use of high performance computer systems was discussed in the context of AgMIP research and crop modeling requirements in general.
- The use of GITFlow methods to standardize how groups work with version control.
- Error logging. A uniform method of error logging was discussed so that users can be alerted to data omissions that will likely affect their simulations, and other information, warnings and errors that may occur during translation. This will be implemented by Chris and made available to all translators.
ICASA data review
The ICASA group helped the groups to interpret the ICASA variables as each group mapped their model variables to the ICASA master list. In some cases, modifications were made to the ICASA variable to accommodate use by multiple models or to clarify the definitions or units. Some of these modifications include:
- suites of sites that can be grouped together for analysis purposes
- field landscape position
- initial NO3 and NH4 mass for entire profile
- fertilizer NO3 and NH4 specified separately, rather than total N
- simulation end date added to list
- soil albedo, both wet and dry measurements are now accomodated
- psychrometric constant
- psychrometric ventilation (ventilated or natural air flow)
- maximum and minimum relative humidity
- daily water table depth
Future work with the ICASA data list will include documentation with links to relevant in-depth information, a search tool, links to an ontology or semantic wiki, addition of livestock and crop rotation variables.
TACC / iPlant demos
Three groups worked on putting together demos of crop model simulations on the TACC computers. Kelly Thorp demonstrated the use of a precision agriculture application using Quantum GIS, an open source GIS package, where data were collected for many locations on a single field. An optimization routine was used to calibrate the soil parameters for each location using Rosetta pedotransfer functions. The application, which took more than 5 hours on his Linux machine, took less than 20 minutes using 320 processors on the Ranger computer (the slower machine which was just replaced by Stampede) at TACC. Kelly will continue this work from his home office and plans to publish on the topic.
Joshua Elliot demonstrated, via a remote connection, crop model simulations using a gridded system at various scales using the SWIFT script language to control the parallelization. Simulations were done using a DSSAT-CSM for a region in Niger, the Sahel region of Africa, and for the globe. All simulations were on a 0.5 arc-minute grid size, resulting in varying magnitudes of processing, depending on the region. The demonstration was supported by iPlant and TACC personnel and showed the relative ease of porting an existing application using SWIFT and an off-the-shelf crop model to the super computer.
A third demonstration project was done by John Fonner, who also worked with Kelly Thorp and Joshua Elliot on the previous demonstrations. John was able to implement a version of the QuadUI data translation tool into the iPlant Discovery Environment (DE). With standard DE tools, he very quickly created a user interface that can be accessed by any user in the iPlant system.
Recommendations and future work
It was suggested that we form an AgMIP data design standards group which could make recommendations on data structures, variable names and definitions and priorities for future work. This team could convene at Global AgMIP meetings, regional workshops, special sessions and through regular teleconferences.
A discussion about a special issue of Environmental Modeling and Software was discussed to include papers on:
- Model improvements for individual AgMIP crop models
- AgMIP progress paper
- AgMIP data format and translation tools
- other relevant modeling papers
The journal will be edited by Dean Holzworth, Sander Janssen and Ioannis Athanasiadis (and others who may be interested).
The hard work of many individuals made this workshop successful. Special thanks to Steve Welch for conceiving the idea and making it happen. And to Valerie Shilling for organizing the event, the catering and the never-ending chocolate supply! And to Jayme Brown for making all the travel arrangements and last-minute re-arrangements.
Allen, R. G., Pereira, L. S., Raes, D., & Smith, M. 1998. Crop Evapotranpiration: Guildlines for computing crop water requirements, FAO Irrigation and Drainage Paper No 56. Food and Agriculture Organisation, Land and Water. Rome, Italy: