To promote the discovery, integration, dissemination, and employment of new knowledge of computer and information science and engineering by supporting experimental investigations conducted by research teams dedicated to making fundamental advances through innovations in software and software engineering
To provide opportunities for investigators from different research areas of computer and information science and engineering to collaborate on addressing a research question that requires the multiple perspectives and talents of the team members
To improve the experimental computer science expertise of principal investigators and students in research institutions.
Conducting experiments comparing the performance of information retrieval algorithms based on extent and nature of structure in scientific data bases
Investigating the scalability of promising ideas for process management support to teraflops-level scientific computing
Designing, implementing, and evaluating novel policies for managing network traffic required by distributed multimedia applications
Prototyping and evaluating innovative user interfaces and environments for computation and problem-solving by scientists
Conducting experimental evaluations to compare and contrast specific visual programming technologies
Designing and implementing a novel language and support environment for vision and robotics tasks
Experimentally investigating the merits of alternative software architectures for cognitive processing systems.
The following descriptions summarize two broad types of projects envisioned as appropriate for ESS support. These descriptions are not intended to be restrictive, but rather to provide examples of types of projects, team composition, and impacts. Build and Evaluate a Novel Software System -- implement a working system that provides demonstrable capabilities to identified users, where the software in the system exhibits a novel concept, and implementing the system is essential to demonstrate the feasibility or scalability of the concept.
The research team may include software architects and builders, and experts from application domains and functional specialties addressed by the software. Researchers are encouraged to use available components (e.g., commercial products or components accessible to the research community), where possible, in building software systems, to focus effort on the novel elements and to facilitate cooperative research through a component-based infrastructure. Any proposed implementation of a software system must be strongly motivated, when compared to alternative approaches, as being an appropriate course of action to address the fundamental research questions. The implemented system must be instrumented to facilitate experimentation with its constituent elements and evaluation of its operational and performance characteristics. This technical evaluation of the system must be an integral part of the proposed research. Examples of potential impacts include --- demonstrating to a user community an experimental software support system with significantly enhanced capabilities
- calibrating the performance of a novel algorithm
- creating infrastructure artifacts usable on subsequent experimental software systems
- improving the systems-building and experimentation expertise in research institutions
Experiments in Software Engineering -- design and execute experiments to test hypotheses about software performance and usability, and about the effectiveness of processes, environments, methods, and tools for software development, integration, and evolution.
The research team may include specialists in software architecture, system performance, usability, and other software engineering technologies addressed by the experiments; industrial software development professionals; and specialists in the computational models, domains, or functional areas addressed by the software. The ESS program intends to enable controlled and reproducible experiments, which have been difficult to conduct, in contrast to other kinds of empirical studies, such as case studies, surveys, or observational studies. Examples of potential impacts include --- settling open questions concerning the relative effectiveness of methods and tools for similar purposes
-- stimulating software development organizations to modify their processes because the experiments produced quantitative evidence concerning the nature and extent of improvement potentially realizable through the use of particular practices or tools
-- encouraging the adoption of a novel architecture by software producers, because the experiments showed it to have superior properties to alternatives
-- strengthening the experimentation expertise of research institutions
What is the intellectual merit and quality of the proposed activity?
The following are suggested questions that the reviewer will consider in assessing how well the proposal meets this criterion. Each reviewer will address only those questions which he/she considers relevant to the proposal and for which he/she is qualified to make judgments. How important is the proposed activity to advancing knowledge and understanding within its own field and across different fields? How well qualified is the proposer (individual or team) to conduct the project? (If appropriate, the reviewer will comment on the quality of prior work.) To what extent does the proposed activity suggest and explore creative and original concepts? How well conceived and organized is the proposed activity? Is there sufficient access to resources? What are the broader impacts of the proposed activity?
The following are suggested questions that the reviewer will consider in assessing how well the proposal meets this criterion. Each reviewer will address only those questions which he/she considers relevant to the proposal and for which he/she is qualified to make judgments. How well does the activity advance discovery and understanding while promoting teaching, training, and learning? How well does the proposed activity broaden the participation of underrepresented groups (e.g., gender, ethnicity, geographic, etc.)? To what extent will it enhance the infrastructure for research and education, such as facilities, instrumentation, networks, and partnerships?Will the results be disseminated broadly to enhance scientific and technological understanding?
What may be the benefits of the proposed activity to society?
Additional ESS-Specific Criteria: In addition to these generic NSF review criteria, reviewers will be asked to use the following additional criteria when reviewing proposals that respond to this announcement: Regular ESS Proposals (2 additional criteria): - Match of the proposed activities to the objectives and description of the ESS Program, as stated in this program announcement - Quality of the proposed plans to (as appropriate) conduct experiments, build software, and evaluate software systems. Planning Grants (1 additional criterion): Likelihood that successful completion of the planning grant will provide benefits to the research community and provide preparation for that community to conduct experimental research.