Resources on Experimental Computer Science

Activity related to Experimental Computer Science

• The 2007 ACM Workshop on Experimental Computer Science
• Evaluate collaboratory on experimental evaluation of software and computer systems.
• RepliCHI discussion on replication in the HCI community
• SC 2011 workshop on Support for Experimental Computer Science

Papers on Experimental Computer Science

• Todd Mytkowicz, Amer Diwan, Matthias Hauswirth, and Peter Sweeney, “Producing wrong data without doing anything obviously wrong!” 14th Architectural Support for Prog. Lang. & Op. Syst. 2009.
• Ron Kohavi and Roger Longbotham, “Unexpected Results in Online Controlled Experiments” SIGKDD Explorations 12(2), pp. 31-35, Dec 2010.
• Jens Gustedt, Emmanuel Jeannot, and Martin Quinson, “Experimental methodologies for large-scale systems: a survey”. Parallel Processing Let. 19(3) pp. 399-418, 2009.
• Ron Kohavi, Roger Longbotham, Dan Sommerfield, and Randal M. Henne, “Controlled experiments on the web: survey and practical guide”. Data Mining and Knowledge Discovery 18(1), pp. 140-181, Feb 2009.
• Todd Mytkowicz, Amer Diwan, Matthias Hauswirth, and Peter Sweeney, “We have it easy, but do we have it right?”. Intl. Parallel and Distributed Processing Symp. 2008.
• L.D. Jackel, Douglas Hackett, Eric Krotkov, Michael Perschbacher, James Pippine, and Charles Sullivan, “How DARPA structures its robotics programs to improve locomotion and navigation”. Comm. ACM 50(11) pp. 55-59, Nov 2007.
• D. G. Feitelson, Experimental Computer Science: The Need for a Cultural Change. Manuscript, 2005.
• E. Frachtenberg and D. G. Feitelson, “Pitfalls in parallel job scheduling evaluation”. In Job Scheduling Strategies for Parallel Processing, pp. 257-282, Springer-Verlag, 2005. Lecture Notes in Computer Science Vol. 3834.
• V. Paxson, “Strategies for Sound Internet Measurement”. Internet Measurement Conference, Oct 2004.
• B. A. Kitchenham, S. L. Pfleeger, L. M. Pickard, P. W. Jones, D. C. Hoaglin, K. El Emam, and J. Rosenberg, “Preliminary Guidelines for Empirical Research in Software Engineering”. IEEE Trans. Softw. Eng. 28(8), pp. 721-734, Aug 2002.
• D. S. Johnson, A Theoretician's Guide to the Experimental Analysis of Algorithms. Nov 2001.
• W. F. Tichy, “Should Computer Scientists Experiment More?”. Computer 31(5) pp. 32-40, May 1998.
• W. F. Tichy, P. Lukowicz, L. Prechelt, and E. A. Heinz, “Experimental Evaluation in Computer Science: A Quantitative Study”. J. Systems and Software 28(1) pp. 9-18, Jan 1995.
• A. Brooks, J. Daly, J. Miller, M. Roper, and M. Wood, Replication's Role in Experimental Computer Science. Technical report EFoCS-5-94 [RR/94/172], University of Strathclyde, 1994.
• National Research Council, Academic Careers for Experimental Computer Scientists and Engineers. US National Academy of Sciences Report, 1994.
• N. Fenton, S. L. Pfleeger, and R. L. Glass, “Science and Substance: A Challenge to Software Engineers”. IEEE Software 11(4) pp. 86-95, Jul/Aug 1994.

Venues Related to Experimental Computer Science

• The ACM Journal of Experimental Algorithmics.
• Empirical Software Engineering (Springer).
• The SIGMETRICS Conf. on Measurement and Modeling of Computer Systems.
• The Internet Measurement Conf. (IMC).
• The Passive & Active Measurement Conf. (PAM).
• The Annual Workshops on Duplicating, Deconstructing, and Debunking, held with ISCA.
• The Intl. Symp. on Empirical Software Engineering and Measurement (ESEM).
• IEEE Intl. Symp. on Workload Characterization (IISWC).
• Workshop on Experimental Evaluation of Software and Systems
• Symposia (previously Workshops) on Experimental Algorithms.
• The SIAM Workshops on Algorithm Engineering and Experiments (ALENEX).
• The Symposia on Experimental Robotics.
• Workshop on Analysis of System Logs (WASL).
• The Workshop on the Peformance and Evaluation of Data Management Systems
• DIMACS Implementation Challenges: using experimentation to provide guides to realistic algorithm performance where analysis fails.

Large-Scale Experimentation

• PRObE: An experimental testbed for large-scale systems research based on decommissioned supercomputers at LANL.
• PlanetLab: An open platform for developing, deploying, and accessing planetary-scale services.
• Emulab: a network emulation testbed.
• The Parallel Reconfigurable Observational Environment (PRObE) for large-scale systems research.
• RAMP: Research Accelerator for Multiple Processors, using FPGAs to create the infrastructure to study multicore architectures.
• Data Center Observatory at CMU, used to observe and study daily data center operation.
• TREC: A series of conferences dedicated to competitive evaluation of large-scale text retrieval systems.
• The MediaEval Benchmark: A series of workshops using predefined tasks and datasets to evaluate multimedia-related applications.
• RoboCup: using soccer playing as a common problem for robotics and AI research.

Publicly Available Data Sets

• The Parallel Workloads Archive.
• USENIX Failure Data Repository.
• LANL operational and failure data.
• Wikipedia page view counts.
• UMass Trace Repository, with data related to various papers including on networking and multimedia.
• The Game Trace Archive.
• The Internet Traffic Archive.
• NLANR Internet traces.
• CAIDA: the Cooperative Association for Internet Data Analysis.
• LBNL/ICSI Enterprise Tracing Project.
• MAWI Backbone Traffic Archive.
• Waikato Internet Traffic Storage.
• DIMES structural Internet data.
• CRAWDAD repository of wireless data.
• Pablo I/O traces from parallel applications.
• Unix file sizes distribution.
• The BYU Performance Evaluation Laboratory Trace Collection Center.
• New Mexico State Univ Trace Database: address traces for processor architecture studies.
• Input data sets for the MiBench benchmark for embedded systems.
• Video Frame Size Traces.
• Email corpus from Enron Corp.
• Web services QoS data.
• Yahoo!'s Webscope datasets (for academics only).

Tools

• Paradyn: dynamic instrumentation of applications and even a running kernel.
• coNCePTuaL: a high-level language to define communication tests and benchmarks.
• Klogger: a logging framework for the Linux kernel.
• PAPI: API for access to hardware performance counters.
• The Auto-pilot tools for running benchmarks.
• The DaCapo benchmark suite for Java benchmarking.

Educational Material

• Course on designing computer science experiments from Stanford (2011)
• Experimental approaches in computer science by Dror Feitelson, Hebrew University (2011)
• Experimental methods in computer systems by Carla Ellis, Duke University (2004)
• Quantitative Methods & Experimental Design in CS by Sanjeev Setia, George Mason University (2009)
• Computer Networking Laboratory from UIUC (2008)
• Networking Technology laboratory course from the Helsinki University of Technology (2007)
• A proof of concept course in experimental techniques for computer science majors by Steve Robbins, The University of Texas at San Antonio (2003)
• Ronny Kohavi's Experimentation Platform work at Microsoft (2006-2010)
• Guidelines for controlled experiments and surveys from Ben Shneiderman's courses.
• Basics of Software Engineering Experimentation, by Natalia Juristo and Ana M. Moreno. Springer, 2001.
• Measuring Computer Performance: A Practitioner's Guide, by David J. Lilja. Cambridge University Press, 2005. ISBN: 05216-4670-7.
  
  provided by AddALL.com
• Internet Measurement: Infrastructure, Traffic, And Applications, by Balachander Krishnamurthy and Mark Crovella. John Wiley and Sons, Inc., 2006. ISBN: 04700-1461-x
  
  provided by AddALL.com
• The Art of Computer Systems Performance Analysis by Raj Jain. John Wiley & Sons, Inc., 1991. ISBN: 0-471-50336-3
  
  provided by AddALL.com