and Management Findings for the Public and Indian Housing Information Center System 18 Table 4: Requirements Development and Management Findings for the Real Estate Management System 20 Table 5: Requirements Development and Management Findings for the Resident Assessment Subsystem 22 Table 6: Requirements Development and Management Findings for HUD 9 s Central Accounting and Program System 24 Table 7: Requirements Development and Management Findings for the Empowerment Information System 26 Table 8: Project Management Findings for the Public and Indian Housing Information Center System 31 Table 9: Project Management Findings for the Real Estate Management System 33 Table 10: Project Management Findings for the Resident Assessment Subsystem 35 Table 11: Project Management Findings for HUD 9 s Central Accounting and Program System 37 Table 12: Project Management Findings for the Empowerment Information System 39 Table 13: Contract Tracking and Oversight Findings for the Public and Indian Housing Information Center System 44 Table 14: Contract Tracking and Oversight Findings for the Real Estate Management System 46 Table 15: Contract Tracking and Oversight Findings for the Resident Assessment Subsystem 48 Table 16: Contract Tracking and Oversight Findings for HUD 9 s Central Accounting and Program System 50 Table 17: Contract Tracking and Oversight Findings for the Empowerment Information System 52 Table 18: Software Evaluation Findings for the Public and Indian Housing Information Center System 57 Table 19: Software Evaluation Findings for the Real Estate Management System 59 Page iii GAO-01-962 HUD Information Systems Table 20: Software Evaluation Findings for the Resident Assessment Subsystem 61 Table 21: Software Evaluation Findings for HUD 9 s Central Accounting and Program System 63 Table 22: Software Evaluation Findings for the Empowerment Information System 65 Figures Figure 1: SA-CMM Levels and Descriptions 10 Figure 2: Requirements Development and Management Summary 17 Figure 3: Project Management Summary 30 Figure 4: Contract Tracking and Oversight Summary 43 Figure 5: Software Evaluation Summary 56 Abbreviations CIO Chief Information Officer EIS Empowerment Information System GAO General Accounting Office HUD Department of Housing and Urban Development HUDCAPS HUD Central Accounting and Program System PIC Public and Indian Housing Information Center system RASS Resident Assessment Subsystem REMS Real Estate Management System SA-CMM Software Acquisition Capability Maturity Model SEI Software Engineering Institute Page 1 GAO-01-962 HUD Information Systems September 14, 2001 The Honorable Wayne Allard Ranking Minority Member Subcommittee on Housing and Transportation Committee on Banking, Housing, and Urban Affairs United States Senate Dear Senator Allard: The accompanying report is the first in a series of reports responding to your February 2001 request that we examine a range of management issues at the Department of Housing and Urban Development. This report discusses our assessment of the department 9 s capability to acquire software.<br><br> We are making recommendations to the Secretary of Housing and Urban Development to assist the department in improving this capability. We are sending copies of this report to the Secretary of Housing and Urban Development, the Director of the Office of Management and Budget, and other congressional committees. We will also make copies available to other interested parties upon request.<br><br> If you have questions or wish to discuss the issues in this report, please contact me at (202) 512-6240 or by email at koontzl@gao.gov. An additional GAO contact and staff acknowledgements are listed in appendix II. Sincerely yours, Linda D.<br><br> Koontz Director, Information Management Issues United States General Accounting Office Washington, DC 20548 Executive Summary Page 2 GAO-01-962 HUD Information Systems The Department of Housing and Urban Development (HUD) routinely acquires new information systems and enhancements to manage and support its various programs and operations. GAO has designated HUD 9 s major program areas as high risk, in part because the department 9 s information and financial management systems were poorly integrated, ineffective, and generally unreliable. 1 HUD has been endeavoring to improve its systems to better support its missions and management reforms.<br><br> Because of the importance of information and financial management systems and related improvement efforts to meeting HUD 9 s mission, the Ranking Minority Member, Subcommittee on Housing and Transportation, Senate Committee on Banking, Housing, and Urban Affairs, requested that GAO review the maturity of HUD 9 s software acquisition processes. More mature processes increase the likelihood that the software acquired will meet an organization 9 s needs. HUD is the principal federal agency responsible for housing and community development programs.<br><br> HUD 9 s mission includes making housing affordable (through mortgage insurance for multifamily housing and through rental assistance), helping to revitalize localities, and encouraging home ownership. In fiscal year 2001, the department 9 s budget was about $32 billion, including $360 million for information technology investments. High-quality software is essential for HUD 9 s information systems to provide reliable management, financial, and administrative information and support the department 9 s many programs.<br><br> The quality of software is governed largely by the quality of the processes involved in developing or acquiring it and in maintaining it. Carnegie Mellon University 9 s Software Engineering Institute (SEI), recognized for its expertise in software processes, has developed models and methods that define and determine organizations 9 software process maturity. Together, these models and 1 GAO High-Risk Program ( GAO/AIMD-94-72R , January 27, 1994); High-Risk Series: Department of Housing and Urban Development ( GAO/HR-95-11 , February 1995); High- Risk Series: Department of Housing and Urban Development ( GAO/HR-97-12 , February 1997); Major Management Challenges and Program Risks: Department of Housing and Urban Development ( GAO/OCG-99-8 , January 1999); Major Management Challenges and Program Risks: Department of Housing and Urban Development ( GAO-01-248 , January 2001).<br><br> Executive Summary Purpose Background Executive Summary Page 3 GAO-01-962 HUD Information Systems methods provide a logical framework for baselining an organization 9 s current process capabilities (i.e., determining what practices are effectively implemented (strengths), not effectively implemented (weaknesses), or contain mixed or inconclusive evidence (observations)) and providing a structured plan for incremental process improvement. Using SEI 9 s software acquisition capability maturity model SM and SEI 9 s software capability evaluation method, 2 GAO analysts trained at SEI evaluated HUD 9 s software acquisition maturity in four of seven key process areas 3 that are necessary to attain a c repeatable d level of process maturity. The c repeatable d level of process maturity is the second level on SEI 9 s five-level scale.<br><br> An organization at the repeatable level of process maturity has the necessary process discipline in place to repeat earlier successes on similar projects. Organizations that do not satisfy the requirements for the repeatable level are by default judged to be at the c initial d level of maturity. This means that their processes are immature, ad hoc, and sometimes even chaotic, with few of the processes defined and success dependent mainly on the heroic efforts of individuals.<br><br> In this evaluation, GAO examined five ongoing projects selected by HUD as being representative of the department 9 s current software acquisition practices. 4 These were the Public and Indian Housing Information Center system, the Real Estate Management System, the Resident Assessment Subsystem, HUD 9 s Central Accounting and Program System, and the Empowerment Information System. HUD did not fully satisfy the requirements for any of the c repeatable d key process areas we reviewed.<br><br> While HUD has several software acquisition 2 Capability Maturity Model SM is the service mark of Carnegie Mellon University, and CMM is registered in the U.S. Patent and Trademark Office. GAO used Software Acquisition Capability Maturity Model (SA-CMM) Version 1.2 (CMU/SEI-99-TR-002, April 1999), the latest version of the model.<br><br> 3 The four key process areas GAO evaluated are requirements development and management, project management, contract tracking and oversight, and software evaluation. GAO did not evaluate the remaining three key process areas 4 software acquisition planning, solicitation, and transition to support 4 because HUD 9 s project teams had not recently performed activities in these areas. 4 GAO also asked that the five projects be (1) major efforts with large software acquisition components, (2) managed by integrated project teams, (3) at different stages of the life cycle, and (4) among HUD 9 s best managed projects.<br><br> Results in Brief Executive Summary Page 4 GAO-01-962 HUD Information Systems process strengths, GAO found a large number of software process weaknesses in all key process areas evaluated: requirements development and management, project management, contract tracking and oversight, and software evaluation. As a result, its processes for acquiring software are immature and can be characterized as ad hoc, sometimes chaotic, and not repeatable across projects. These weaknesses can lead to systems that do not meet the information needs of management and staff, do not provide support for needed programs and operations, and cost more and take longer than expected to complete.<br><br> HUD is aware that it has software acquisition weaknesses, has stated its commitment to improving its software and system acquisition processes, and will begin a process improvement effort in the near future. HUD did not meet the requirements for the repeatable level of maturity in the four key process areas we reviewed. Of the 310 key practices GAO evaluated, 50 percent constituted strengths (i.e., key practice was effectively implemented), 39 percent were weaknesses (i.e., key practice was not effectively implemented), 10 percent were rated as observations, and 1 percent were not rated.<br><br> Certain weaknesses were systemic, recurring in most or all of the key process areas. For example, HUD has no overall policy for the acquisition of software; the majority of project teams did not develop plans to conduct various activities; none of the teams measured the status of activities; the majority of the teams did not report regularly to management on progress and problems; and most project managers did not require regular reports on progress and problems. These weaknesses can lead to systems that do not meet HUD 9 s information needs, do not effectively support HUD 9 s programs and services, and cost more and take longer to complete.<br><br> To reach the repeatable level of maturity, HUD must overcome the key practice weaknesses identified in this report. Table 1 summarizes the strengths and weaknesses for the five projects. Principal Finding: HUD 9 s Software Acquisition Processes Are Immature Executive Summary Page 5 GAO-01-962 HUD Information Systems Table 1: Total Number of Key Process Area Strengths, Weaknesses, and Observations for the Five Projects Key practice ratings Key process area Strengths Weaknesses Observations Not rated Requirements development and management 27 31 12 0 Project management 45 25 10 0 Contract tracking and oversight 45 33 4 3 Evaluation 37 33 5 0 Total 154 122 31 3 HUD acknowledged that it has software acquisition weaknesses and stated its commitment to improving its software and system acquisition processes.<br><br> The department has prepared a statement of work to obtain contractor support to begin a software process improvement effort. One of the tasks the contract will include is development of a software process improvement plan. To successfully complete such an effort, HUD 9 s plan must address several important issues.<br><br> To be comprehensive, this plan should include the results of this review, measurable goals and time frames, estimates of resource requirements, and time frames to reach the repeatable level. In addition, it should address the systemic weaknesses that GAO found. Finally, the plan should include steps to address the key process areas GAO did not review, as well as steps to ensure that all ongoing and new software acquisition projects adopt processes that meet the requirements for the repeatable level.<br><br> To improve HUD 9 s software acquisition capabilities, GAO recommends that the Secretary of Housing and Urban Development direct the HUD Chief Information Officer to develop and implement a comprehensive plan for software acquisition process improvement that is based on the software capability results in this report and specifies measurable goals and time frames, sets priorities for initiatives, estimates resource requirements (for trained staff and funding), and defines a process improvement management structure. Also, to address the systemic weaknesses mentioned above, the plan should contain steps to Recommendations for Executive Action Executive Summary Page 6 GAO-01-962 HUD Information Systems " develop a comprehensive policy for the acquisition of software, " require plans for specific acquisition activities, " measure and track the status of activities performed by the software acquisition teams, and " report regularly to management on progress and problems. In addition, to ensure that all aspects of software acquisition are addressed, we recommend that the Secretary direct the CIO to " assess HUD 9 s maturity in the three key process areas that could not be evaluated by GAO and include any needed improvement actions in the comprehensive plan for software acquisition process improvement; " ensure that all new software acquisition projects in HUD adopt processes that satisfy at least SA-CMM level 2 requirements; and " ensure that process improvement activities are initiated for all ongoing software acquisition projects.<br><br> In providing written comments on a draft of our report, HUD agreed with our assessment of its software acquisition processes and our recommendations for strengthening them, and stated that our assessment and recommendations will be addressed as part of its process improvement effort. This effort should help the department improve its ability to acquire systems that meet management and staff information needs and efficiently support programs and operations. Agency Comments Chapter 1: Introduction Page 7 GAO-01-962 HUD Information Systems The Department of Housing and Urban Development (HUD), established in 1965, has as its primary mission ensuring a decent, safe, and sanitary home and suitable living environment for every American.<br><br> HUD 9 s mission includes making housing affordable for about 4 million low-income households by insuring loans for multifamily rental housing and by providing rental assistance. Its mission also includes helping to revitalize over 4,000 localities through community development programs. The department encourages homeownership by providing, through its Federal Housing Administration, mortgage insurance for about 7 million homeowners who otherwise might not have qualified for loans.<br><br> HUD is one of the nation 9 s largest financial institutions, responsible for managing about $508 billion in insured mortgages and $570 billion in guarantees of mortgage-backed securities. HUD 9 s budget in fiscal year 2001 was about $32 billion, including $360 million for information technology investments. Like many federal agencies, HUD relies on its information and financial management systems to help carry out its missions.<br><br> However, past reports by us and others showed that HUD was plagued by poorly integrated, ineffective, and generally unreliable information systems that did not satisfy management needs or provide adequate controls. 1 We designated HUD 9 s program areas as high risk in part because of widespread problems with information and financial management systems. In 1997, HUD began a management reform effort designed to transform its culture, manage its programs and staff more effectively, ultimately restore public trust in the department, and modernize it for the 21st century.<br><br> In January 2001, we reported that HUD had been making credible progress towards addressing its management reform goals. 2 However, we also reported that information and financial systems remained an area of management challenge because of unresolved issues. In addition, in its March 1, 2001, financial statement audit report, the Office of the HUD Inspector General listed two material internal control weaknesses related to systems.<br><br> The report stated that HUD needs to (1) complete improvements to its financial management systems to meet its program and financial management needs and (2) enhance the Federal Housing 1 GAO-01-248 , January 2001; GAO/OCG-99-8 , January 1999; GAO/HR-97-12 , February 1997; GAO/HR-95-11 , February 1995; GAO/AIMD-94-72R , January 27, 1994. 2 GAO-01-248 , January 2001. Chapter 1: Introduction HUD Relies on Information Systems Chapter 1: Introduction Page 8 GAO-01-962 HUD Information Systems Administration 9 s information systems to more effectively support its business practices.<br><br> HUD is aware that information systems remain an area of challenge. To help address some of the challenges, HUD plans to (1) institute more disciplined processes in its information technology capital investment planning reviews (quarterly reviews that assess how well projects are functioning and attempt to alert management to potential problems), (2) continue training its project managers in new project management techniques, (3) complete efforts to bring all software products under standard change control and configuration management 3 and designate a separate organization to address configuration management issues, (4) improve HUD 9 s end-to-end testing 4 capabilities, and (5) obtain and deploy new software for project-level status reporting. To manage specific software acquisitions, HUD uses integrated project teams.<br><br> These teams include representatives from the business organization(s) that will use the acquired software to do their work; information technology staff to oversee the contractor and review contractor work products; and contract and procurement staff to assist the team in contract management. Policy and procedures for software acquisition are the responsibility of HUD 9 s Chief Information Officer. 3 Configuration management is a discipline applying technical and administrative direction and surveillance to identify and document the functional and physical characteristics of hardware or software, control changes to these characteristics, record and report change processing, and verify compliance of hardware or software with specified requirements.<br><br> 4 End-to-end testing is done to verify that a defined set of interrelated systems, which collectively support an organizational core business area or function, interoperate as intended in an operational environment. This includes not only those systems owned and managed by the organization but also those external systems with which they interface. Chapter 1: Introduction Page 9 GAO-01-962 HUD Information Systems The software acquisition capability maturity model (SA-CMM), 5 developed by Carnegie Mellon 9 s Software Engineering Institute (SEI), is used to measure an organization 9 s capability to manage the acquisition of software.<br><br> SEI 9 s expertise in and methods for software process assessment are recognized and accepted worldwide throughout the industry. The model defines five levels of software acquisition maturity. Each level of maturity (except level 1) indicates process capability and identifies key process areas.<br><br> For a maturity level to be achieved, all key process areas related to that level must be implemented effectively. The first level of process capability is level 2 (referred to as the repeatable level), where basic management processes are established to track performance, cost, and schedule, and the necessary discipline is in place to repeat earlier successes on similar projects. Organizations that do not effectively implement all key process areas for the repeatable level are, by default, at level 1 or the initial level of maturity.<br><br> Level 1 processes can be described as immature, ad hoc, and sometimes chaotic; success in software acquisition for these organizations is usually dependent upon the ability and commitment of the staff involved. Figure 1 explains further the five-level software acquisition model. 5 Software Acquisition Capability Maturity Model (SA-CMM), Version 1.2, Software Engineering Institute, CMU/SEI-99-TR-002 (April 1999).<br><br> The Software Acquisition Capability Maturity Model Provides a Means of Assessing an Organization 9 s Ability to Manage Software Acquisitions Chapter 1: Introduction Page 10 GAO-01-962 HUD Information Systems Figure 1: SA-CMM Levels and Descriptions Table 2 identifies each of the seven key process areas associated with the repeatable level of maturity. The software acquisition process is characterized as ad hoc, and occasionally even chaotic. Few processes are defined and success depends on individual effort.<br><br> Level 1: Initial Basic project management processes are established to track performance, cost, and schedule. The necessary process discipline is in place to repeat earlier successes on projects in similar domains. Level 2: Repeatable The acquisition organization 9s software acquisition process is documented, standardized, and established as the standard software acquisition process.<br><br> All projects use an approved, tailored version of the organization 9s standard software acquisition process for acquiring their software products and services. Level 3: Defined Level 5: Optimizing Level 4: Managed Continuous process improvement is empowered by quantitative feedback from the process and from piloting innovative ideas and technologies. Detailed measures of quality of the software acquisition processes, products, and services are collected.<br><br> The software processes, products, and services are quantitatively understood and controlled. Disciplined process Standard consistent process Predictable process Continuously improving process Chapter 1: Introduction Page 11 GAO-01-962 HUD Information Systems Table 2: Key Process Areas for SA-CMM Level 2 SA-CMM level 2 key process areas Description Software acquisition planning Ensuring that reasonable planning for the software acquisition is conducted and that all elements of the project are included. Solicitation Ensuring that award is made to the contractor most capable of satisfying the specified requirements.<br><br> Requirements development and management Establishing a common and unambiguous definition of software acquisition requirements that is understood by the acquisition team, system users, and contractor(s). Project management Managing the activities of the project office and supporting contractor(s) to ensure a timely, efficient, and effective software acquisition. Contract tracking and oversight Ensuring that the software activities under contract are being performed in accordance with contract requirements and that products and services will satisfy contract requirements.<br><br> Evaluation Determining that the acquired software products and services satisfy contract requirements before acceptance. Transition to support Providing for the transition of the software products being acquired to their eventual support organization. As established by the model, each key process area contains five common features 4 commitment to perform, ability to perform, activities performed, measurement and analysis, and verifying implementation 4 that indicate whether the implementation and institutionalization of the key process area can be effective, repeatable, and lasting.<br><br> The common feature definitions are as follows: " Commitment to perform : This feature describes the actions that the organization takes to establish the process and ensure that it can endure. Key practices typically involve establishing organizational policies and sponsorship. " Ability to perform : This feature describes the preconditions that must exist in the project or organization to competently implement the software acquisition process.<br><br> Key practices typically include assigning responsibility and providing training. " Activities performed : This feature describes the roles and procedures necessary to implement a key process area. Key practices typically involve establishing plans and procedures, performing the work, tracking it, and taking appropriate management actions.<br><br> " Measurement and analysis : This feature describes the activities necessary to measure progress and analyze the measurements. Key practices typically involve defining the measurements to be taken and the Chapter 1: Introduction Page 12 GAO-01-962 HUD Information Systems analyses to be conducted to determine the status and effectiveness of the activities performed. " Verifying implementation : This feature describes the steps the organization must take to ensure that project activities are performed in accordance with established processes.<br><br> Key practices typically involve regular reviews by management. Each common feature consists of a number of key practices 4 specific activities such as developing an organizational policy for software acquisition, developing various plans for software acquisition activities, and tracking a contractor 9 s progress. When an organization is evaluated against the SA-CMM, comparisons of actual performance against a key practice can result in one of four possible outcomes or ratings: " Strength: The key practice involved was effectively implemented.<br><br> " Weakness: The key practice was ineffectively implemented or was not implemented. " Observation: The key practice was evaluated, but cannot be characterized as a strength because the (1) project team did not provide sufficient evidence to support a strength rating or (2) key practice was only partially performed. " Not rated: The key practice is not relevant to the project and was therefore not rated.<br><br> To achieve the repeatable level, HUD would have to demonstrate that the key practices related to that level were implemented effectively in its software acquisition projects. The Ranking Minority Member of the Subcommittee on Housing and Transportation, Senate Committee on Banking, Housing, and Urban Affairs, requested that we review HUD 9 s software acquisition capabilities. Our objective for this review was to determine the maturity of HUD 9 s software acquisition processes.<br><br> To determine HUD 9 s software acquisition process maturity, we applied the Software Engineering Institute 9 s SA-CMM using our SEI-trained analysts. We focused on the key process areas necessary to obtain a repeatable level of maturity, the second level of SEI 9 s five-level model. We met with project managers and project team members to determine whether and to Objective, Scope, and Methodology Chapter 1: Introduction Page 13 GAO-01-962 HUD Information Systems what extent they implemented each key practice and to obtain relevant documentation.<br><br> In accordance with the SEI model, for each key process area we reviewed, 6 we evaluated HUD 9 s institutional policies and practices and compared project-specific guidance and practices against the required key practices. More specifically, for each key practice we reviewed, we compared project-specific documentation and practices against the criteria in the software acquisition model. If the project met the criteria for the key practice reviewed, we rated it as a strength.<br><br> If the project did not meet the criteria for the key practice reviewed, we rated it as a weakness. If the evidence did not support a rating of a strength or a weakness, we rated it as an observation. If the key practice was not relevant to a project, we stated that the key practice was not rated.<br><br> We evaluated five HUD projects, each of which is described below. We asked HUD to select five system acquisition projects that were representative of HUD software acquisition practices and met the following criteria: all were (1) major efforts with large software acquisition components, (2) managed by integrated project teams, (3) at different stages of the life cycle, and (4) among the best-managed acquisitions. HUD selected the following projects: " Public and Indian Housing Information Center system (PIC) 4 an Internet- based system that collects information about the public housing inventory managed by HUD and automates the processes for allocating program funds to public housing authorities.<br><br> It is a central repository for information about public and Indian housing events and program areas as well as for the housing inventory. " Real Estate Management System (REMS) 4 a system that provides automated support to collect and maintain data on multifamily insured and assisted properties and their ownership/management entities and provides access to data collected by HUD on physical and financial assessments. 6 We evaluated HUD on four of the seven key process areas 4 requirements development and management, project management, contract tracking and oversight, and software evaluation.<br><br> We did not evaluate HUD on the three other key process areas 4 software acquisition planning, solicitation, and transition to support 4 because these were not recently performed for the selected projects. Chapter 1: Introduction Page 14 GAO-01-962 HUD Information Systems " Resident Assessment Subsystem (RASS) 4 a system that collects information on public housing from residents and manages the resident assessment process. " HUD Central Accounting and Program System (HUDCAPS) 4 HUD 9 s core accounting and general ledger system.<br><br> It serves as the standard accounting system for all program areas. " Empowerment Information System (EIS) 4 an enterprisewide data warehouse project that is to provide HUD users and business partners with access to reports and analytical information across a large variety of program data sources. We performed our work from March 2001 through August 2001 in accordance with generally accepted government auditing standards.<br><br> We provided a draft of our report to HUD for review, and the department 9 s comments are addressed in chapter 6 and reprinted as appendix I. Chapter 2: HUD 9 s Requirements Development and Management Processes Are Not Repeatable Page 15 GAO-01-962 HUD Information Systems HUD did not satisfy the criteria for the repeatable level of maturity in this key process area because of the number and severity of key practice weaknesses found. As a result of these weaknesses, HUD is at greater risk of acquiring software products that do not meet mission needs and that exceed cost, schedule, and performance goals.<br><br> The purpose of requirements development and management is to establish and maintain a common and unambiguous definition of software requirements 1 among the acquisition team, system users, and software development contractor. Within this key process area, the SA-CMM specifies key practices for each of the five common features that an organization must implement effectively to achieve the repeatable level of maturity. Generally, these practices include having a written organizational policy for establishing and managing requirements allocated to software, documenting plans for the development and management of requirements, having documented processes for requirements development, including elicitation, analysis, and verification, measuring and reporting on the status of activities to management, appraising the impact of system-level requirements changes on the software, and having a mechanism to ensure that contractor-delivered work products meet the specified requirements.<br><br> Of the 70 key practices for this key process area, HUD had 27 strengths, 31 weaknesses and 12 observations. For the " Commitment to perform feature, there were two strengths, five weaknesses, and three observations. " Ability to perform feature, there were nine strengths, five weaknesses, and one observation.<br><br> " Activities performed feature, there were ten strengths, 14 weaknesses, and six observations. " Measurement and analysis feature, there were one strength, three weaknesses, and one observation. " Verifying implementation feature, there were five strengths, four weaknesses, and one observation.<br><br> In addition, none of the projects had strengths in all the key practices. 1 Requirements describe the functions that the software will perform to meet user needs and provide support for business processes. Chapter 2: HUD 9 s Requirements Development and Management Processes Are Not Repeatable Chapter 2: HUD 9 s Requirements Development and Management Processes Are Not Repeatable Page 16 GAO-01-962 HUD Information Systems As a result of these weaknesses, HUD is exposed to increased risks that acquired software will not meet its needs and that projects will exceed cost, schedule, and performance goals.<br><br> HUD acknowledges the need to improve its requirements development and management processes and is committed to improving its software and systems acquisition capability. The department has prepared a statement of work to obtain contractor support to begin a software process improvement effort. This document specifies that the contractor would review HUD software acquisition processes, prioritize the key processes and practices to address, and develop a plan to improve HUD 9 s acquisition processes.<br><br> Details of our evaluation are provided in the following figure and tables. Figure 2 provides a comprehensive listing of the strengths, weaknesses, and observations for the requirements development and management key process area. The specific findings supporting the practice ratings cited in figure 2 are in tables 3 through 7.<br><br> Chapter 2: HUD 9 s Requirements Development and Management Processes Are Not Repeatable Page 17 GAO-01-962 HUD Information Systems Figure 2: Requirements Development and Management Summary Projects Common features Key practices PIC REMS RASS HUDCAPS EIS Commitment to perform 1 The acquisition organization has a written policy for establishing and managing the software-related contractual requirements. Weakness Weakness Observation Observation Weakness Commitment to perform 2 Responsibility has been designated for requirements development and management activities. Weakness Strength Strength Observation Weakness Ability to perform 1 A group is responsible for performing requirements development and management activities.<br><br> Weakness Strength Strength Observation Strength Ability to perform 2 Adequate resources are provided for requirements development and management activities. Weakness Weakness Strength Weakness Weakness Ability to perform 3 Individuals performing requirements development and management activities have experience or receive training. Strength Strength Strength Strength Strength Activity performed 1 The project team performs its activities in accordance with its documented requirements development and management plans.<br><br> Weakness Observation Strength Weakness Weakness Activity performed 2 The project team develops, baselines, and maintains software-related contractual requirements and places them under change control early in the project, but not later than the release of the solicitation package. Observation Weakness Strength Strength Observation Activity performed 3 The project team appraises system requirements change requests for their impact on the software being acquired. Weakness Weakness Strength Strength Weakness Activity performed 4 The project team appraises changes to the software- related contractual requirements for their impact on performance, architecture, supportability, system resource utilization, and contract schedule and cost.<br><br> Weakness Weakness Strength Weakness Weakness Activity performed 5 Bidirectional traceability between the contractual requirements and the contractor team 9 s software products and services is maintained throughout the effort. Observation Weakness Strength Strength Weakness Activity performed 6 The end user and other affected groups are involved in the development of all software-related contractual requirements and any subsequent change activity. Observation Strength Strength Observation Weakness Measurement and analysis 1 Measurements are made and used to determine the status of the requirements development and management activities and resultant products.<br><br> Weakness Weakness Observation Weakness Strength Verifying implementation 1 Requirements development and management activities are reviewed by the acquisition organization on a periodic basis. Strength Observation Weakness Strength Strength Verifying implementation 2 Requirements development and management activities are reviewed by the project manager on both a periodic and an event-driven basis. Weakness Strength Weakness Strength Weakness Key Strength Strength: Key practice effectively implemented.<br><br> Weakness Weakness: Key practice not effectively implemented. Observation Observation: Key practice evaluated; evidence not sufficient to rate as a strength, or practice only partially performed. Chapter 2: HUD 9 s Requirements Development and Management Processes Are Not Repeatable Page 18 GAO-01-962 HUD Information Systems Table 3: Requirements Development and Management Findings for the Public and Indian Housing Information Center System Public and Indian Housing Information Center system Common feature Key practice Finding Rating Commitment to perform 1 The acquisition organization has a written policy for establishing and managing the software-related contractual requirements.<br><br> HUD has no written policy for establishing and managing software-related contractual requirements. Weakness Commitment to perform 2 Responsibility has been designated for requirements development and management activities. Responsibility for requirements development and management has not been designated.<br><br> Weakness Ability to perform 1 A group is responsible for performing requirements development and management activities. No group is assigned responsibility for performing requirements development and management. Weakness Ability to perform 2 Adequate resources are provided for requirements development and management activities.<br><br> There is no mechanism to identify resource needs and ensure that they are provided to the project. Team members indicated that they do not have adequate resources for performing these activities. Weakness Ability to perform 3 Individuals performing requirements development and management activities have experience or receive training.<br><br> Project team members performing requirements development and management activities have experience. Strength Activity performed 1 The project team performs its activities in accordance with its documented requirements development and management plans. There is no requirements development and management plan, and so the activities are not performed in accordance with it.<br><br> Weakness Activity performed 2 The project team develops, baselines, and maintains software-related contractual requirements and places them under change control early in the project, but not later than the release of the solicitation package. The project team developed and baselined software-related requirements before the solicitation package was released. The project team provided no evidence of change control.<br><br> Observation Activity performed 3 The project team appraises system requirements change requests for their impact on the software being acquired. The project team does not assess system requirements change requests for their impact on the software being acquired. Weakness Activity performed 4 The project team appraises changes to the software-related contractual requirements for their impact on performance, architecture, supportability, system resource utilization, and contract schedule and cost.<br><br> The project team does not assess the impact of software-related contractual requirements changes on performance, architecture, supportability, and resource utilization. Cost and schedule impacts are assessed. Weakness Activity performed 5 Bidirectional traceability between the contractual requirements and the contractor team 9 s software products and services is maintained throughout the effort.<br><br> The project team maintains traceability between the contract deliverables and the contractor 9 s task/subtask specification numbering. Traceability to HUD 9 s original requirements, however, is not maintained. Observation Activity performed 6 The end user and other affected groups are involved in the development of all software-related contractual requirements and any subsequent change activity.<br><br> End users are involved in identifying software-related requirements for development but are not involved in subsequent change activities. Observation Chapter 2: HUD 9 s Requirements Development and Management Processes Are Not Repeatable Page 19 GAO-01-962 HUD Information Systems Public and Indian Housing Information Center system Common feature Key practice Finding Rating Measurement and analysis 1 Measurements are made and used to determine the status of the requirements development and management activities and resultant products. No measurements are made of HUD 9 s requirements development and management activities and resultant products.<br><br> Weakness Verifying implementation 1 Requirements development and management activities are reviewed by the acquisition organization on a periodic basis. Project status, including requirements development and management activities, is reviewed quarterly by acquisition management. Strength Verifying implementation 2 Requirements development and management activities are reviewed by the project manager on both a periodic and an event-driven basis.<br><br> The project manager is not briefed on project activities and the project manager 9 s reviews are not documented. Weakness Chapter 2: HUD 9 s Requirements Development and Management Processes Are Not Repeatable Page 20 GAO-01-962 HUD Information Systems Table 4: Requirements Development and Management Findings for the Real Estate Management System Real Estate Management System Common feature Key practice Finding Rating Commitment to perform 1 The acquisition organization has a written policy for establishing and managing the software-related contractual requirements. HUD has no written policy for establishing and managing software-related contractual requirements.<br><br> Weakness Commitment to perform 2 Responsibility has been designated for requirements development and management activities. Responsibility for requirements development and management activities is designated in the project plan. Strength Ability to perform 1 A group is responsible for performing requirements development and management activities.<br><br> End users and project staff are responsible for performing requirements development and management activities. Strength Ability to perform 2 Adequate resources are provided for requirements development and management activities. There is no mechanism to identify resource needs and ensure that they are provided to the project.<br><br> The project manager said that the team does not have adequate resources for performing these activities. Weakness Ability to perform 3 Individuals performing requirements development and management activities have experience or receive training. Project team members performing requirements development and management activities have experience.<br><br> Strength Activity performed 1 The project team performs its activities in accordance with its documented requirements development and management plans. There is no requirements development and management plan, and so the activities are not performed in accordance with it. The team does follow a process for requirements gathering.<br><br> Observation Activity performed 2 The project team develops, baselines, and maintains software-related contractual requirements and places them under change control early in the project, but not later than the release of the solicitation package. The project team did not baseline requirements before solicitation. Task orders do not show traceability to software- related requirements.<br><br> Weakness Activity performed 3 The project team appraises system requirements change requests for their impact on the software being acquired. The project team does not appraise system requirements change requests for their impact on the software being acquired. Weakness Activity performed 4 The project team appraises changes to the software-related contractual requirements for their impact on performance, architecture, support- ability, system resource utilization, and contract schedule and cost.<br><br> The project team does not assess the impact of software-related contractual requirements changes on performance, architecture, supportability, and resource utilization. Cost and schedule are assessed. Weakness Activity performed 5 Bidirectional traceability between the contractual requirements and the contractor team 9 s software products and services is maintained throughout the effort.<br><br> The project team does not maintain bidirectional traceability between the contractual requirements and contractor work products. Weakness Chapter 2: HUD 9 s Requirements Development and Management Processes Are Not Repeatable Page 21 GAO-01-962 HUD Information Systems Real Estate Management System Common feature Key practice Finding Rating Activity performed 6 The end user and other affected groups are involved in the development of all software-related contractual requirements and any subsequent change activity. End users are involved in the development of and changes to software-related contractual requirements.<br><br> Strength Measurement and analysis 1 Measurements are made and used to determine the status of the requirements development and management activities and resultant products. No measurements are made of HUD 9 s requirements development and management activities and resultant products. Weakness Verifying implementation 1 Requirements development and management activities are reviewed by the acquisition organization on a periodic basis.<br><br> Requirements development and management activities are not routinely reviewed by the acquisition organization. However, the project manager is a senior official of the acquisition organization. Observation Verifying implementation 2 Requirements development and management activities are reviewed by the project manager on both a periodic and an event-driven basis.<br><br> The project manager told us that he participates in biweekly status meetings that include a review of requirements development and management issues. The status meetings are documented. Strength Chapter 2: HUD 9 s Requirements Development and Management Processes Are Not Repeatable Page 22 GAO-01-962 HUD Information Systems Table 5: Requirements Development and Management Findings for the Resident Assessment Subsystem Resident Assessment Subsystem Common feature Key practice Finding Rating Commitment to perform 1 The acquisition organization has a written policy for establishing and managing the software-related contractual requirements.<br><br> HUD has no written policy for establishing and managing software-related contractual requirements. The project team used HUD 9 s software development methodology as its standard for establishing and managing software-related contractual requirements. Observation Commitment to perform 2 Responsibility has been designated for requirements development and management activities.<br><br> Responsibility for requirements development and management is designated in the project plan. Strength Ability to perform 1 A group is responsible for performing requirements development and management activities. The project team is responsible for performing requirements development and management activities.<br><br> Strength Ability to perform 2 Adequate resources are provided for requirements development and management activities. There is no mechanism to identify resource needs and ensure that they are provided to the project. However, the project manager stated that they do have adequate resources for performing requirements development and management activities.<br><br> Strength Ability to perform 3 Individuals performing requirements development and management activities have experience or receive training. Project team members performing requirements development and management activities have experience and have received training. Strength Activity performed 1 The project team performs its activities in accordance with its documented requirements development and management plans.<br><br> The project team performs its activities in accordance with its requirements development and management plan. Strength Activity performed 2 The project team develops, baselines, and maintains software- related contractual requirements and places them under change control early in the project, but not later than the release of the solicitation package. The project team developed and baselined contract requirements in its functional requirements document and requirements traceability matrix, and placed the requirements under change control.<br><br> Strength Activity performed 3 The project team appraises system requirements change requests for their impact on the software being acquired. The project team assesses system requirements change requests for their impact on the software being acquired. Strength Activity performed 4 The project team appraises changes to the software-related contractual requirements for their impact on performance, architecture, support- ability, system resource utilization, and contract schedule and cost.<br><br> The project team appraises changes to the software-related contractual requirements for impact on performance, architecture, support- ability, system resource utilization, and contract schedule and cost. Strength Chapter 2: HUD 9 s Requirements Development and Management Processes Are Not Repeatable Page 23 GAO-01-962 HUD Information Systems Resident Assessment Subsystem Common feature Key practice Finding Rating Activity performed 5 Bidirectional traceability between the contractual requirements and the contractor team 9 s software products and services is maintained throughout the effort. The project team maintains bidirectional traceability between contractual requirements and contractor work products.<br><br> Strength Activity performed 6 The end user and other affected groups are involved in the development of all software-related contractual requirements and any subsequent change activity. End users are involved in the development of contractual requirements, generation of system change requests, and testing. Strength Measurement and analysis 1 Measurements are made and used to determine the status of the requirements development and management activities and resultant products.<br><br> Masurements are only made of HUD 9 s requirements development and management products. Observation Verifying implementation 1 Requirements development and management activities are reviewed by the acquisition organization on a periodic basis. Requirements development and management activities are not routinely reviewed by the acquisition organization.<br><br> However, cost and schedule performance is reviewed periodically by the acquisition organization. Weakness Verifying implementation 2 Requirements development and management activities are reviewed by the project manager on both a periodic and an event-driven basis. The project manager is not briefed on requirements development and management activities, and the project manager 9 s reviews are not documented.<br><br> Weakness Chapter 2: HUD 9 s Requirements Development and Management Processes Are Not Repeatable Page 24 GAO-01-962 HUD Information Systems Table 6: Requirements Development and Management Findings for HUD 9s Central Accounting and Program System HUD 9s Central Accounting and Program System Common feature Key practice Finding Rating Commitment to perform 1 The acquisition organization has a written policy for establishing and managing the software-related contractual requirements. HUD has no written policy for establishing and managing the software-related contractual requirements. The project team used HUD 9 s software development methodology as its standard for establishing and managing software-related contractual requirements.<br><br> Observation Commitment to perform 2 Responsibility has been designated for requirements development and management activities. The project team has been designated responsible for requirements development and management, with no specific designation of what group or person within the team. Observation Ability to perform 1 A group is responsible for performing requirements development and management activities.<br><br> The project manager told us that the project team is responsible for requirements development and management activities. Observation Ability to perform 2 Adequate resources are provided for requirements development and management activities. There is no mechanism to identify resource needs and ensure that they are provided to the project.<br><br> The project team indicated that they do not have adequate resources for performing requirements development and management activities. Weakness Ability to perform 3 Individuals performing requirements development and management activities have experience or receive training. Project team members performing requirements development and management activities have experience and have received training.<br><br> Strength Activity performed 1 The project team performs its activities in accordance with its documented requirements development and management plans. There is no requirements development and management plan, and so the activities are not performed in accordance with it. Weakness Activity performed 2 The project team develops, baselines, and maintains software-related contractual requirements and places them under change control early in the project, but not later than the release of the solicitation package.<br><br> The project team developed and baselined software-related contractual requirements before the contractor began work, and has maintained the requirements under configuration board control. Strength Activity performed 3 The project team appraises system requirements change requests for their impact on the software being acquired. The project team assesses all system requirements change requests for their impact on the software being acquired.<br><br> Strength Activity performed 4 The project team appraises changes to the software-related contractual requirements for their impact on performance, architecture, support- ability, system resource utilization, and contract schedule and cost. The project team does not assess the impact of software-related contractual requirements changes on performance, architecture, supportability, and system resource utilization. Cost is appraised.<br><br> Weakness Chapter 2: HUD 9 s Requirements Development and Management Processes Are Not Repeatable Page 25 GAO-01-962 HUD Information Systems HUD 9 s Central Accounting and Program System Common feature Key practice Finding Rating Activity performed 5 Bidirectional traceability between the contractual requirements and the contractor team 9 s software products and services is maintained throughout the effort. The project team maintains bidirectional traceability between the contractual requirements and the contractor team 9 s software products and services. Strength Activity performed 6 The end user and other affected groups are involved in the development of all software-related contractual requirements and any subsequent change activity.<br><br> End users and an independent verification and validation team were involved in testing of changes. No evidence was provided to document user activity in the development of requirements. Observation Measurement and analysis 1 Measurements are made and used to determine the status of the requirements development and management activities and resultant products.<br><br> No measurements are made of HUD 9 s requirements development and management activities or resultant products. Weakness Verifying implementation 1 Requirements development and management activities are reviewed by the acquisition organization on a periodic basis. Acquisition management is regularly updated on project status, including requirements development and maintenance activities.<br><br> Strength Verifying implementation 2 Requirements development and management activities are reviewed by the project manager on both a periodic and an event-driven basis. Project manager has regular, documented meetings to review requirements development and management activities. Strength Chapter 2: HUD 9 s Requirements Development and Management Processes Are Not Repeatable Page 26 GAO-01-962 HUD Information Systems Table 7: Requirements Development and Management Findings for the Empowerment Information System Empowerment Information System Common feature Key practice Finding Rating Commitment to perform 1 The acquisition organization has a written policy for establishing and managing the software-related contractual requirements.<br><br> HUD has no written policy for establishing and managing software- related contractual requirements. Weakness Commitment to perform 2 Responsibility has been designated for requirements development and management activities. Responsibility for requirements development and management has not been designated.<br><br> Weakness Ability to perform 1 A group is responsible for performing requirements development and management activities. The project team is responsible for performing these activities. Strength Ability to perform 2 Adequate resources are provided for requirements development and management activities.<br><br> There is no mechanism to identify resource needs and ensure that they are provided to the project. The project team stated that they do not have adequate resources for performing requirements development and management activities. Weakness Ability to perform 3 Individuals performing requirements development and management activities have experience or receive training.<br><br> Project team members have training in IT project management, including requirements management. Strength Activity performed 1 The project team performs its activities in accordance with its documented requirements development and management plans. There is no requirements development and management plan, and so the activities are not performed in accordance with it.<br><br> Weakness Activity performed 2 The project team develops, baselines, and maintains software-related contractual requirements and places them under change control early in the project, but not later than the release of the solicitation package. The project team developed high-level requirements. Detailed requirements were developed by the contractor.<br><br> No evidence was provided regarding change control. Observation Activity performed 3 The project team appraises system requirements change requests for their impact on the software being acquired. The project team does not appraise system requirements changes for their impact on the software being acquired.<br><br> Weakness Activity performed 4 The project team appraises changes to the software-related contractual requirements for their impact on performance, architecture, supportability, system resource utilization, and contract schedule and cost. The project team does not assess the impact of software-related contractual requirements changes on performance, architecture, supportability, system resource utilization, and contract schedule and cost. Weakness Activity performed 5 Bidirectional traceability between the contractual requirements and the contractor team 9 s software products and services is maintained throughout the effort.<br><br> The project team does not maintain bidirectional traceability between the contractual requirements and contractor work products. Weakness Chapter 2: HUD 9 s Requirements Development and Management Processes Are Not Repeatable Page 27 GAO-01-962 HUD Information Systems Empowerment Information System Common feature Key practice Finding Rating Activity performed 6 The end user and other affected groups are involved in the development of all software-related contractual requirements and any subsequent change activity. No end users were involved in requirements development or subsequent change activities.<br><br> Weakness Measurement and analysis 1 Measurements are made and used to determine the status of the requirements development and management activities and resultant products. Measurements are made that identify the status of requirements development and management activities. Strength Verifying implementation 1 Requirements development and management activities are reviewed by the acquisition organization on a periodic basis.<br><br> Requirements development and management activities are reviewed periodically by the acquisition organization. Strength Verifying implementation 2 Requirements development and management activities are reviewed by the project manager on both a periodic and an event-driven basis. The project manager told us that he is not briefed on activities, and the project manager 9 s reviews are not documented.<br><br> Weakness Chapter 3: HUD 9 s Project Management Processes Are Not Repeatable Page 28 GAO-01-962 HUD Information Systems HUD did not meet the criteria for the repeatable level in the project management key process area because of the number and severity of key practice weaknesses found. These weaknesses increase the risk that the software acquisition project, the project team, and supporting contractors will not be adequately managed, resulting in software that does not meet mission needs or acquisitions that do not meet cost, schedule, and performance goals. The purpose of project management is to direct and oversee the activities of the project team and supporting contractors to ensure a timely, efficient, and effective software acquisition.<br><br> Within this key process area, the SA-CMM specifies key practices for each of the five common features that an organization must implement effectively to achieve the repeatable level of maturity. Generally, these practices include having a project team organized to accomplish the project 9 s objective; having a wr

less