Preproduction Simulation Stage

The Office of Information and Technology (OIT) Mobile Application Program (MAP) has tasked Booz Allen Hamilton with supporting tasks under the Mobile Applications (Apps) Verification and Validation (V&V) contract. The V&V team is tasked with supporting the comprehensive testing of multiple Apps from two projects within the Mobile Applications Program: Mobile Development and Mobile Health External Development. They provide the management, business/functional, and technical services needed to assist in providing VA-directed programmatic planning, analysis, design and oversight to guide the V&V of these Apps.

The V&V team sees Apps through the Test Intake process, which includes a Functional Review Findings process to verify that the App is ready to test; then through Test Execution and Analysis, which includes Functional, 508, and Security testing. Once Test Execution and Analysis is complete, a Test Summary Debrief meeting is held to deliver all documentation of test analysis and results so that the App may proceed into compliance review, Pilot/Initial Operating Capability (IOC), and National Release. Throughout the process, the V&V team is tasked with engaging compliance review bodies to ensure that Apps have been reviewed for compliance during development, go through V&V compliance testing, and that compliance best practices are documented and utilized for Apps.

Review Purpose

V&V performs testing, following Software Quality Assurance (SQA) development to ensure that Apps function as desired and meet security and compliance standards without defects. V&V is required in order for an App to enter Pilot, Initial Operating Capability (IOC) and National Release.

V&V Prerequisites

1. Fully Development Application
1. Prior to V&V, an App must be fully developed.
2. Software Quality Assurance (SQA) testing must be completed.
3. It is the expectation of the V&V team that once the App is submitted, testing may be completed successfully (e.g., No Critical or High Severity Level defects or anticipated compliance failures). Therefore, it is highly encouraged that Development teams and Project Managers (PMs) worked closely with compliance reviewers during development.
2. Completed Test Intake Form
1. A Test Intake Form requires the following documentation (preferably stored in JIRA or Wiki):
1. Business Requirements Document
2. Concept Paper
3. Design Document
4. Data Model
5. Release Management Plan
6. Requirements Traceability Matrix
7. Requirements Specification Document (RSD)/ARD Mobile Apps Addendum
8. SQA Checklist and/or Unit Test Results
9. Sustainment Plan
10. System Design Document Mobile Apps Addendum
11. System Security Plan Addendum
12. User Guide
13. User Stories/Acceptance Criteria
2. If an intake form is missing required documentation, the form will be returned until the documentation and/or information can be provided before an App is able to enter into the Test Planning and Execution stage.
3. It is highly recommend that the App Project already exists in the Mobile Application Environment (MAE) JIRA and documentation is stored there.

When to Engage

1. When an application has been approved the Business Owner and is entering development, alert the V&V team through your Project Manager. If possible, the V&V team may attend scrum calls for awareness.
2. Once development and all associated documentation and testing have been completed, the Test Intake Form and Addendum should be submitted via email. See your Project Manager for more information.

V&V Review Process

The V&V Test Framework is divided into three main stages, all of which are covered and further defined in the Master Test Plan.

1. Test Preparation

   1. Test Intake

      1. Receive Apps test request

      2. Receive completed intake sheet and all relevant Apps materials from VA

      3. Review and validate test request with stakeholder and prioritize application

      4. Acquire Apps software

      5. Deliverable: Test Intake Form

   2. Requirements Analysis

      1. Receive requirements and conduct Functional Product Review

      2. Document and submit Functional Review Findings Report

      3. Deliverable: Functional Review Findings Report

   3. Test Planning

      1. Develop/Modify Master Test Plan

      2. Create Test Plan & Test Procedures for Functional, 508 and Security Testing

      3. Create Requirements Traceability Matrix (RTM)

      4. Generate test data

      5. Test Plan Review

      6. Identify and select tools and test techniques.

      7. Establish automated test framework, as needed

      8. Manage configuration of test devices, test data, test procedures and associated test artifacts

      9. Set-up test environment and perform dry-run

      10. Deliverables: Requirements Traceability Matrix, Test Plan Package

2. Test Execution and Evaluation

   1. Test Execution

      1. Execute Test Procedures

      2. Record test results

      3. Document incidents and findings in JIRA

      4. Track incidents and findings in JIRA

      5. Generate defect report

      6. Perform additional testing, as necessary

      7. Deliverables: Test Plan Package, Defect Fix Prioritization List

   2. Test Analysis and Reporting

      1. Collect data and analyze test results

      2. Document and submit Test Summary Reports

      3. Complete and submit SQA Checklist

      4. Complete and submit Enterprise Testing Services (ETS) Intake Assessment

      5. Deliverables: Test Summary Report, RTM Updates, SQA Checklist

   3. Support User Acceptance Testing (UAT) and Compliance

      1. Create UAT Test Scripts

      2. Participate in UAT

      3. Hold daily scrum meetings with Compliance Approvers

      4. Conduct Compliance Debrief meeting

3. Initial Operating Capability (IOC)
   1. 1. Participate in IOC Entry Meeting
      2. Participate in IOC Evaluation
      3. Participate in IOC Exit Meeting
      4. Deliverable: IOC Entry and Exit Request Summary

Note that there will be Go/No Go decisions before proceeding to:

• Test Planning (decided during Functional Review Findings Meeting)
• Test Execution (decided during Test Readiness Review Meetings)
• UAT/Compliance Review (decided during Test Summary Review Meetings)
• IOC Entry (decided during Compliance De-Brief Meeting)

(No Go decisions may be made due to lack of documentation, critical defects, etc.)

References

• Mobile Apps V&V Wiki Home Page
• Mobile Apps V&V Test Intake Forms