Dry-lab experimentation is being increasingly used to complement wet-lab experimentation. However, conducting dry-lab experiments is a challenging endeavor that requires the combination of diverse techniques. JAMES II, a plug-in-based open source modeling and simulation framework, facilitates the exploitation and configuration of these techniques. The different aspects that form an experiment are made explicit to facilitate repeatability and reuse. Each of those influences the performance and the quality of the simulation experiment. Common experimentation pitfalls and current challenges are discussed along the way.