Resources on Experimental Computer Science

Papers on Experimental Computer Science

• Todd Mytkowicz, Amer Diwan, Matthias Hauswirth, and Peter Sweeney, ``We have it easy, but do we have it right?''. Intl. Parallel and Distributed Processing Symp. 2008.
• D. G. Feitelson, ``Experimental Computer Science: The Need for a Cultural Change''. Manuscript, 2005.
• V. Paxson, ``Strategies for Sound Internet Measurement''. Internet Measurement Conference, Oct 2004.
• D. S. Johnson, ``A Theoretician's Guide to the Experimental Analysis of Algorithms''. Nov 2001.
• W. F. Tichy, ``Should Computer Scientists Experiment More?''. Computer 31(5) pp. 32-40, May 1998.
• W. F. Tichy, P. Lukowicz, L. Prechelt, and E. A. Heinz, ``Experimental Evaluation in Computer Science: A Quantitative Study''. J. Systems and Software 28(1) pp. 9-18, Jan 1995.
• A. Brooks, J. Daly, J. Miller, M. Roper, and M. Wood, ``Replication's Role in Experimental Computer Science''. Technical report EFoCS-5-94 [RR/94/172], University of Strathclyde, 1994.
• National Research Council, Academic Careers for Experimental Computer Scientists and Engineers. US National Academy of Sciences Report, 1994.
• N. Fenton, S. L. Pfleeger, and R. L. Glass, Science and Substance: A Challenge to Software Engineers. IEEE Software 11(4) pp. 86-95, Jul/Aug 1994.

Venues Related to Experimental Computer Science

• The ACM Journal of Experimental Algorithmics.
• Empirical Software Engineering (Springer).
• The SIGMETRICS Conf. on Measurement and Modeling of Computer Systems.
• The Internet Measurement Conf. (IMC).
• The Passive & Active Measurement Conf. (PAM).
• The Annual Workshops on Duplicating, Deconstructing, and Debunking, held with ISCA.
• The Intl. Symp. on Empirical Software Engineering and Measurement (ESEM).
• IEEE Intl. Symp. on Workload Characterization (IISWC).
• Workshop on Analysis of System Logs (WASL).
• The Workshop on the Peformance and Evaluation of Data Management Systems
• DIMACS Implementation Challenges: using experimentation to provide guides to realistic algorithm performance where analysis fails.
• Workshops on Experimental Algorithms.
• The SIAM Workshops on Algorithm Engineering and Experiments (ALENEX).
• The Symposia on Experimental Robotics.

Large-Scale Experimentation

• TREC: A series of conferences dedicated to competitive evaluation of large-scale text retrieval systems.
• PlanetLab: An open platform for developing, deploying, and accessing planetary-scale services.
• Emulab: a network emulation testbed.
• RAMP: Research Accelerator for Multiple Processors, using FPGAs to create the infrastructure to study multicore architectures.
• Data Center Observatory at CMU, used to observe and study daily data center operation.
• RoboCup: using soccer playing as a common problem for robotics and AI research.

Publicly Available Data Sets

• The Parallel Workloads Archive.
• LANL operational and failure data.
• The Internet Traffic Archive.
• NLANR Internet traces.
• CAIDA: the Cooperative Association for Internet Data Analysis.
• LBNL/ICSI Enterprise Tracing Project.
• MAWI Backbone Traffic Archive.
• Waikato Internet Traffic Storage.
• DIMES structural Internet data.
• CRAWDAD repository of wireless data.
• Pablo I/O traces from parallel applications.
• Unix file sizes distribution.
• The BYU Performance Evaluation Laboratory Trace Collection Center.
• New Mexico State Univ Trace Database: address traces for processor architecture studies.
• Input data sets for the MiBench benchmark for embedded systems.
• Video Frame Size Traces.
• Email corpus from Enron Corp.

Tools

• Paradyn: dynamic instrumentation of applications and even a running kernel.
• coNCePTuaL: a high-level language to define communication tests and benchmarks.
• Klogger: a logging framework for the Linux kernel.
• PAPI: API for access to hardware performance counters.
• The Auto-pilot tools for running benchmarks.
• The DaCapo benchmark suite for Java benchmarking.

Educational Material

• A proof of concept course in experimental techniques for computer science majors by Steve Robbins, The University of Texas at San Antonio (2003)
• Experimental methods in computer systems by Carla Ellis, Duke University (2004)
• Quantitative Methods & Experimental Design in CS by Sanjeev Setia, George Mason University (2005, 2007)
• Experimental approaches in computer science by Dror Feitelson, Hebrew University (2007)
• Computer Networking Laboratory from UIUC (2007)
• Networking Technology laboratory course from the Helsinki University of Technology (2007)
• Guidelines for controlled experiments and surveys from Ben Shneiderman's courses.
• Measuring Computer Performance: A Practitioner's Guide, by David J. Lilja. Cambridge University Press, 2005. ISBN: 05216-4670-7.
  
  provided by AddALL.com
• Internet Measurement: Infrastructure, Traffic, And Applications, by Balachander Krishnamurthy and Mark Crovella. John Wiley and Sons, Inc., 2006. ISBN: 04700-1461-x
  
  provided by AddALL.com
• The Art of Computer Systems Performance Analysis by Raj Jain. John Wiley & Sons, Inc., 1991. ISBN: 0-471-50336-3
  
  provided by AddALL.com