One reason for the long pause since the last post is that our team members have been traveling to conferences and workshops. But now, we are happy to report from some particularly insightful events!

In September, I participated in a course on teaching agent-based modeling given by Volker Grimm and Steve Railsback in Leipzig. This one-week course addressed mainly scientists who will at some point supervise or teach students who develop agent-based models. The content was aligned with the book by Grimm and Railsback, which is rich in practical examples and as such suitable for self-study as well.

One example of a NetLogo implementation, simulating butterfly movement in a hilly landscape.

One specific topic I wanted to learn more about is related to the valid critique against the use of agent-based modelling in science: It is hard to test. So how can we improve testing?

The good news is that there are a couple of strategies one can use to build trust in agent-based models, validate them in a step-wise procedure and make these tests transparent via a proper documentation. Here are some hints from the course:

• Code review: The best approach would be that two people implement the code together from the beginning. The second best alternative is to let somebody else review the code with respect to the specification. In general, the current practice that one person is designing, coding, validating and documenting a model alone can be easily improved by delegating these roles to different people. In this way, one can prevent a lot of errors during these steps, which would be much harder (and costly in terms of time!) later.
• Hypothesis testing: It’s always good practice to have some idea what kind of outcome to expect before running simulations. The more these expectations are specified as a hypothesis, the more can be learned from the model. This is already true in the early phase of model implementation when smaller units are tested. Syntax and runtime errors can easily be found in the coding environment. But to detect logical errors, one should insert assertions directly into the code.  These are simple checks of assumptions such as “check whether population size > 0”, typically inserted between different steps of calculations. This enables more confidence in the code and that the developed model is reproducing the expected behavior. Writing test procedures or “unit tests” is another way of checking parts of the implementation. Particularly efficient for testing are of course also visual outputs, but they are most efficient if one tests honestly against prior expectations.
• Stress tests: Some people might say “never change a running system” – however, the scientific attitude should rather be “try to break the model as hard as you can”. This might provoke the right attitude to think of critical test procedures and challenge the model behavior to its limits. One way to do so is via extreme parameter settings beyond reasonable ranges (in terms of the represented system). Doing this exercise enables one to specify more precisely for which range of parameter values the simulation or analysis outcome holds.
• Pattern-oriented modeling: If you belong to the lucky ones that have some data to test their model against it – use it. This data might be very coarse but there are often ways to analyze your model output statistically and somehow relate it to observed patterns of your targeted system.
• One of my favorites: Document the tests. It sounds very simple, yet this is often neglected for model or even code publications. These tests are not only valuable during development and while exchanging insights with close colleagues, but they are also worth to be published to increase transparency and trust in the model application. A prominent place to do this exists in the submodel section of the ODD-model documentation. Each submodel should be accompanied by its own test.
• Publish the code of your model: This may again sound simple, but it is not always done, sometimes because of restrictions from journals, but more often due to a lack of interest or lack of resources from authors. One very nice platform for discussing research on agent-based models is the OpenABM website. They recently added functionality for uploading and sharing models after a thorough review process. Tested models are then published in the OpenABM-library and certified if they comply to current scientific standards of documentation and testing.

These strategies might not be completely new after all, but they certainly deserve more of our time and effort such that agent-based models can be applied more soundly in a scientific context. If you are aware of other strategies, we would be happy to read your comments!

Some suggestions for further readings:

• Annie Waldherr and Nanda Wijermans: Communicating Social Simulation Models to Sceptical Minds (2013).

• Jan C. Thiele, Winfried Kurth and Volker Grimm: Facilitating Parameter Estimation and Sensitivity Analysis of Agent-Based Models: A Cookbook Using NetLogo and R (2014).