Resources on Experimental Computer Science
See also the more general page of
Performance Evaluation Links
Activity related to Experimental Computer Science
The 2007 ACM Workshop on Experimental Computer Science
on experimental evaluation of software and computer systems.
RepliCHI discussion on replication in the HCI community
SC 2011 workshop on Support for Experimental Computer Science
Papers on Experimental Computer Science
Todd Mytkowicz, Amer Diwan, Matthias Hauswirth, and Peter Sweeney,
Producing wrong data without doing anything obviously wrong!
14th Architectural Support for Prog. Lang. & Op. Syst.
Ron Kohavi and Roger Longbotham,
Unexpected Results in Online Controlled Experiments
, pp. 31-35, Dec 2010.
Jens Gustedt, Emmanuel Jeannot, and Martin Quinson,
Experimental methodologies for large-scale systems: a survey
Parallel Processing Let.
pp. 399-418, 2009.
Ron Kohavi, Roger Longbotham, Dan Sommerfield, and Randal M. Henne,
Controlled experiments on the web: survey and practical guide
Data Mining and Knowledge Discovery
, pp. 140-181, Feb 2009.
Todd Mytkowicz, Amer Diwan, Matthias Hauswirth, and Peter Sweeney,
We have it easy, but do we have it right?
Intl. Parallel and Distributed Processing Symp.
L.D. Jackel, Douglas Hackett, Eric Krotkov, Michael Perschbacher, James Pippine, and Charles Sullivan,
How DARPA structures its robotics programs to improve locomotion and navigation
pp. 55-59, Nov 2007.
D. G. Feitelson,
Experimental Computer Science: The Need for a Cultural Change
. Manuscript, 2005.
E. Frachtenberg and D. G. Feitelson,
Pitfalls in parallel job scheduling evaluation
Job Scheduling Strategies for Parallel Processing
, pp. 257-282, Springer-Verlag, 2005. Lecture Notes in Computer Science Vol. 3834.
Strategies for Sound Internet Measurement
Internet Measurement Conference
, Oct 2004.
B. A. Kitchenham, S. L. Pfleeger, L. M. Pickard, P. W. Jones, D. C. Hoaglin, K. El Emam, and J. Rosenberg,
Preliminary Guidelines for Empirical Research in Software Engineering
IEEE Trans. Softw. Eng.
, pp. 721-734, Aug 2002.
D. S. Johnson,
A Theoretician's Guide to the Experimental Analysis of Algorithms
. Nov 2001.
W. F. Tichy,
Should Computer Scientists Experiment More?
pp. 32-40, May 1998.
W. F. Tichy, P. Lukowicz, L. Prechelt, and E. A. Heinz,
Experimental Evaluation in Computer Science: A Quantitative Study
J. Systems and Software
pp. 9-18, Jan 1995.
A. Brooks, J. Daly, J. Miller, M. Roper, and M. Wood,
Replication's Role in Experimental Computer Science
. Technical report EFoCS-5-94 [RR/94/172], University of Strathclyde, 1994.
National Research Council,
Academic Careers for Experimental Computer Scientists and Engineers
. US National Academy of Sciences Report, 1994.
N. Fenton, S. L. Pfleeger, and R. L. Glass,
Science and Substance: A Challenge to Software Engineers
pp. 86-95, Jul/Aug 1994.
Venues Related to Experimental Computer Science
The ACM Journal of Experimental Algorithmics
Empirical Software Engineering
Conf. on Measurement and Modeling of Computer Systems
Internet Measurement Conf.
Passive & Active Measurement Conf.
Workshops on Duplicating, Deconstructing, and Debunking
, held with ISCA.
Intl. Symp. on Empirical Software Engineering and Measurement
IEEE Intl. Symp. on Workload Characterization
Workshop on Experimental Evaluation of Software and Systems
Symposia (previously Workshops) on Experimental Algorithms
The SIAM Workshops on
Algorithm Engineering and Experiments
Symposia on Experimental Robotics
Workshop on Analysis of System Logs
Workshop on the Peformance and Evaluation of Data Management Systems
DIMACS Implementation Challenges
: using experimentation to provide guides to realistic algorithm performance where analysis fails.
: An experimental testbed for large-scale systems research based on decommissioned supercomputers at LANL.
: An open platform for developing, deploying, and accessing planetary-scale services.
: a network emulation testbed.
The Parallel Reconfigurable Observational Environment (PRObE)
for large-scale systems research.
: Research Accelerator for Multiple Processors, using FPGAs to create the infrastructure to study multicore architectures.
Data Center Observatory
at CMU, used to observe and study daily data center operation.
: A series of conferences dedicated to competitive evaluation of large-scale text retrieval systems.
The MediaEval Benchmark
: A series of workshops using predefined tasks and datasets to evaluate multimedia-related applications.
: using soccer playing as a common problem for robotics and AI research.
Publicly Available Data Sets
The Parallel Workloads Archive
USENIX Failure Data Repository
LANL operational and failure data
Wikipedia page view counts
UMass Trace Repository
, with data related to various papers including on networking and multimedia.
The Game Trace Archive
The Internet Traffic Archive
NLANR Internet traces
CAIDA: the Cooperative Association for Internet Data Analysis
LBNL/ICSI Enterprise Tracing Project
MAWI Backbone Traffic Archive
Waikato Internet Traffic Storage
DIMES structural Internet data
CRAWDAD repository of wireless data
Pablo I/O traces
from parallel applications.
Unix file sizes distribution
The BYU Performance Evaluation Laboratory Trace Collection Center
New Mexico State Univ Trace Database
: address traces for processor architecture studies.
Input data sets
benchmark for embedded systems.
Video Frame Size Traces
from Enron Corp.
Web services QoS data
Yahoo!'s Webscope datasets
(for academics only).
: dynamic instrumentation of applications and even a running kernel.
: a high-level language to define communication tests and benchmarks.
: a logging framework for the Linux kernel.
: API for access to hardware performance counters.
tools for running benchmarks.
DaCapo benchmark suite
for Java benchmarking.
Course on designing computer science experiments
from Stanford (2011)
Experimental approaches in computer science
by Dror Feitelson, Hebrew University (2011)
Experimental methods in computer systems
by Carla Ellis, Duke University (2004)
Quantitative Methods & Experimental Design in CS
by Sanjeev Setia, George Mason University (2009)
Computer Networking Laboratory
from UIUC (2008)
Networking Technology laboratory course
from the Helsinki University of Technology (2007)
A proof of concept course in experimental techniques for computer science majors
by Steve Robbins, The University of Texas at San Antonio (2003)
work at Microsoft (2006-2010)
Guidelines for controlled experiments and surveys
from Ben Shneiderman's courses.
Basics of Software Engineering Experimentation
, by Natalia Juristo and Ana M. Moreno. Springer, 2001.
Measuring Computer Performance: A Practitioner's Guide
, by David J. Lilja. Cambridge University Press, 2005. ISBN: 05216-4670-7.
Internet Measurement: Infrastructure, Traffic, And Applications
, by Balachander Krishnamurthy and Mark Crovella. John Wiley and Sons, Inc., 2006. ISBN: 04700-1461-x
The Art of Computer Systems Performance Analysis
by Raj Jain. John Wiley & Sons, Inc., 1991. ISBN: 0-471-50336-3
dgf / 11 Jun 2012