Human Factors Survey of Aviation Technical Manuals Phase 1: Manual

To view this page ensure that Adobe Flash Player version 9.0.124 or greater is installed.

Please login or register to make a comment!

...Description...... more. less.

Report No. 9. Performing Organization Name and Address National Institute for Aviation Research 10. Work Unit No. (TRAIS) Wichita State University 1845 Fairmount Wichita, KS 67260 11. Contract or Grant No. 00-C-WSU-00-12 12. Sponsoring Agency Name and Address U.S. Department of Transportation Federal Aviation Administration 13. Type of Report and Period Covered Interim Report 07/01/00-01/11/01 Office of Aviation Research Washington, DC 20591 14. Sponsoring Agency Code AFS-300 15. Supplementary Notes The FAA William J. Hughes Technical Center Monitor was Cristina Tan. 16. Abstract This report contains the results from Phase 1 of a three-phase research effort. Phase 1 examines aviation industry procedures for developing maintenance technical data. Phase 2 will document user problems with maintenance technical data. Phase 3 will identify maintenance technical data development improvements by applying human factors principles. Five aircraft manufacturers were surveyed regarding company policy, communication, data tracking, user feedback, and error reduction efforts. The five industry participants represent both regional and large commercial transport manufacturers. Phase 1 survey results revealed th ree significant maintenance technical data issues: inconsistent development process guidelines, reactive rather than proactive response to user feedback, and inadequate assessment of errors involving usability as opposed to accuracy. Phase 1 results wil l later be compared to Phase 2 surveys of user problems with maintenance technical data to identify the impact of development procedures on the users perception of manual quality. 17. Key Words Error, Human Factors, Maintenance manuals 18. Distribution Statement This document is available to the public through the National Technical Information Service (NTIS), Springfield, Virginia 22161. 19. Security Classif. (of this report) Unclassified 20. Security Classif. (of this page) Unclassified 21. No. of Pages 34 22. Price Form DOT F1700.7 (8-72) Reproduction of completed page authorized iii TABLE OF CONTENTS Page EXECUTIVE SUMMARY v 1. INTRODUCTION 1 1.1 Background 1 1.2 Regulatory Requirements 4 2. DISCUSSION 5 2.1 Maintenance Technical Manual Considerations 5 2.1.1 Development 5 2.1.2 Time 6 2.1.3 Variations Between Manuals 7 2.1.4 Document Lag 8 2.1.5 Distribution 8 2.1.6 Multiple Manual Formats 8 2.1.7 Document Customization 9 2.1.8 Feedback 10 2.1.9 Aircraft Customization 11 2.1.10 Older Aircraft 11 2.2 Maintenance Technical Data Errors 12 3. EVALUATION 15 3.1 Survey Methods 15 3.2 Industry Analysis 17 3.2.1 Who Writes Manuals? 17 3.2.2 New Manual Development 17 3.2.3 Configuration Management 19 3.2.4 Error Tracking 20 3.2.5 Customer Feedback 21 3.2.6 Validation 22 3.2.7 Measures of Document Quality 23 3.3 Significant Issues 24 3.3.1 Inconsistent Guidelines for Control of the Development Process 24 3.3.2 Use of Feedback 25 3.3.3 Tracking Manual Quality 26 iv 3.4 Final Comments 26 4. REFERENCES 27 LIST OF TABLES Table Page 1 New Manual Development 17 2 Configuration Management 19 3 Error Tracking 20 4 Customer Feedback 21 5 Validation 23 6 Measures of Document Quality 24 v/vi EXECUTIVE SUMMARY Until recently, little attention has been paid to the procedures used to develop and revise aircraft maintenance technical data. Studies of maintenance errors have tended to focus on the actions of the mechanic, job culture, and work procedures. More recently, attempts have been made to document the source of maintenance errors and improve maintenance procedures. One of the identified contributing causes of errors is the documentation used to guide maintenance tasks. As a result, efforts have been made to establish guidelines for the design of maintenance job aids. A question that remains is how the procedures used by manufacturers to develop maintenance data may contribute to user error. In this document, the results of Phase 1 of a three-phase research effort to (1) examine the procedures used by industry to develop maintenance manuals, (2) document the problems encountered by users of these documents, and (3) identify ways in which human factors principles can be used to improve the development of these documents are reported. Phase 1 is a survey and report of the procedures used within the aviation industry to develop maintenance technical data. A cross-section of manufacturers was surveyed regarding company policy, communication, data tracking, user feedback, and error reduction efforts. 1 1. INTRODUCTION . 1.1 BACKGROUND . A little known fact is that aircraft manufacturers rank among the largest publishers in the world. With each aircraft they produce, they provide the technical documentation needed to maintain the aircraft in working order. Aircraft manufacturers provide a wide range of documentation including the maintenance manual itself, an illustrated parts catalog, wiring diagrams, structural repair manuals, and a host of other related documents. The development and revision of aircraft maintenance technical manuals is no small task. The amount of information is staggering, with manuals being measured more appropriately by the feet of shelf space they occupy rather than the number of pages they contain. For aircraft that have any degree of customization, the manufacturer may need to develop different, operator-specific manuals to include only the information relevant to the aircraft they maintain. The Boeing Company estimates that if all manual pages published in 1998 were stacked one on top of another, the resulting tower would reach approximately 103,000 feet in the air [1]. The size and complexity of a modern technical manual requires the integration of information from multiple sources, making it increasingly difficult to verify the accuracy and clarity of the information maintenance manuals contain. Each company has developed their own techniques and procedures to deal with these issues, but there is little documentation in the public domain or industry dialogue about the different procedures employed. While manufacturers may be hesitant to openly share this information, the potential benefits include the reduced cost of developing manuals and improved manual quality. Considering that the goal of a technical manual is to facilitate the safe and efficient maintenance of an aircraft, it is important to continually evaluate the degree to which the applied procedures meet these objectives. The goal of this effort is to document the unique ways companies have addressed the problems of manual development and to encourage a dialogue between the manufacturers, operators, and regulatory bodies within the industry. The task of developing manuals has become a more critical part of aircraft maintenance in recent years as the demands being placed on those manuals increase. The continued growth in worldwide air traffic has resulted in increased production of new aircraft and older aircraft remaining in service much longer than originally expected. While the number of aircraft has increased, the number of maintenance technicians has not; consequently there is greater pressure on aircraft maintenance in general. A 1993 Blue Ribbon Study, cPilots and Aircraft Maintenance Technicians for the 21st Century: An Assessment of Availability and Quality, d found that although there were enough aircraft mechanics at that time, the projected demand was increasing at a faster rate than the number of new mechanics. Amid the favorable economic conditions experienced globally in the years since that study was released, the situation has worsened more quickly than predicted and air traffic continues to increase. The growth of air carrier operations combined with a retiring maintenance workforce and a worker-friendly job market has resulted in a critical shortage of qualified aircraft mechanics. The Labor Department reports that approximately 140,000 mechanics work in the aviation industry today and 40,000 additional mechanics will be needed by 2008. In order to fill the positions necessary to support larger fleets, maintenance facilities are being forced to lower hiring requirements for maintenance 2 personnel. As an example, American Airlines has recently been forced to lower experience requirements from 4 to 2 years. At the same time, there appears to be a general lack of skilled applicants. In 1997, responding to concerns about the skill level of newly hired mechanics, United Airlines incorporated a basic skills test into their interview process for maintenance technicians. This skills test involves tasks required of any aircraft mechanic, including rivet installation and safety wiring of bolts. In an interview with USA Today , the United Airlines Director of Maintenance reported that from the end of 1999 to mid-2000, 1,600 potential employees were interviewed, and only 45% of those could pass the basic skills test [2]. United is one of only five airlines that include a skills test in their hiring process. The experience of United Airlines may indicate a need for long-term changes in the training of new mechanics, but the short-term reality is that the qualification of the maintenance workforce appears to be declining. Lesser-trained and lesser-educated maintenance personnel will rely much more heavily on the content of maintenance manuals. They will also lack the knowledge that comes with experience to determine when a manual may be in error. In some cases, in order to meet demand, work has been offloaded to foreign maintenance facilities that bring with them the added concern of the translatability of documents. Growing liability and regulatory pressures have also increased the scrutiny of technical manuals and, in some cases, have forced the inclusion of corporate lawyers into the manual development process. Because of these pressures, decisions regarding reading level, writing style, clarity, and the degree of detail to be used when writing maintenance manuals are even more important now than in the past. The way a particular manufacturer may handle these content decisions is often a matter of established company history. Manufacturers develop a writing style that is propagated from veteran writers in the company to the newly hired writers, and users come to expect a certain style from a particular manufacturer. Style consistency may have the benefit of a perceived familiarity across manufacturers models, but it may not be appropriate for the changing face of the maintenance workforce. In 1994, a review of major aircraft accidents found that approximately 12% could be attributed to maintenance [3]. According to the Boeing Company 9s summary of commercial airplane accidents worldwide for the years 1959-1999, the figure is a more conservative 5.9% [4]. Regardless of which figure one chooses to use, there has been an apparent increase in maintenance-related aircraft accidents in recent years. While aircraft accidents capture headlines, the truth is that few maintenance errors ever result in accidents. Analysis of maintenance errors indicates that the majority of incidents involve omitted or incorrect execution of tasks, incorrect installations, and the use of incorrect parts (Boeing research cited in reference 5). Such events can potentially lead to accidents if left uncorrected, but these are rare events, and to focus on accidents alone is to underestimate the actual number of errors and their impact on safety and operating costs. A Human Factors approach to reducing maintenance error requires that causal attribution be extended beyond just the offending mechanic. If the search for error sources is extended to environmental influences, causal attribution may include latent errors introduced by management 3 policy, organizational communication, or corporate culture of the maintenance facility. Another potential factor in maintenance error that is easily overlooked is the technical information used to guide maintenance operations. The potential contribution of technical documentation to maintenance error is not a new concern [6]. However, previous attempts at improving maintenance documents, such as Drury and Sarac 9s document design aid [7], have yet to be adopted by aircraft manufacturers on a large scale. Maintenance manuals can contribute to maintenance error if they contain misleading information, insufficient information, or unclear procedures. Not only must the information be technically sound, it must also be presented in an effective manner. A term common to the computer industry most applicable in this case is cusability. d The Institute of Electrical and Electronics Engineers (IEEE) defines usability as cthe ease with which a user can learn to operate, prepare inputs for, and interpret outputs of a system or component d [8]. Applied to aircraft manuals, usability includes the user experience of those manuals; how easy they are to use, how well they match the mechanic 9s representation of a task, and how useful the information is they contain. The user experience of the manuals will also impact the ways and degree to which mechanics will use them. In a study conducted for the Australian Transportation Safety Bureau, 67% report having been misled by maintenance documentation, 47% report having opted to perform a maintenance procedure in a way they felt was superior to that described by the manual, and 73% of mechanics surveyed reported failing to refer to maintenance documents either occasionally or often [9]. These findings raise concerns about the perceived usability of manuals and the quality of information exchange between mechanics and the writers of maintenance materials. The fact that mechanics report instances of failing to refer to maintenance manuals may be indicative of familiarity with the task or prior experience with manuals that were hard to use or not very helpful. If user comments are relied on as the final check of manual quality, the usage pattern becomes a critical issue. A lack of user complaints may be indicative of a well-crafted manual or one that users have simply stopped using and see no point in bringing these issues to the attention of the developer. The task of developing and revising maintenance manuals requires the coordination of multiple information sources across a number of departments within the management structure of the manufacturer. Engineering, technical support, customer service, and technical writing must integrate the most recent information from their respective sources to provide the technical base necessary to produce a technically sound document. Technical writers have the ultimate responsibility of verifying that they have the most recent and accurate information available on which to base technical manuals. The accuracy of the information contained in maintenance manuals is of paramount importance, and manufacturers have implemented multiple safeguards to protect against the inclusion of erroneous content. Document checklists, peer review, and software formatting have all been implemented to reduce the number of errors present in fielded maintenance manuals. Once released, each manufacturer has procedures for addressing problems that users encounter while using the manuals. The accuracy of maintenance manuals gets considerable attention from 4 manufacturers and operators, and the continued application of computer technology to the technical writing task promises greater ability to verify the accuracy of technical information. 1.2 REGULATORY REQUIREMENTS . The current procedure for developing maintenance manuals is an integral part of satisfying the Federal Aviation Administration (FAA) requirement for operators to develop and support an ongoing maintenance program. More specifically, manuals are designed to help operators meet the requirements outlined by the following Federal Aviation Regulations (FAR). " FAR Part 121.363 establishes the operators responsibility to maintain the airworthiness of its aircraft fleet. " FAR Part 121.367 requires each certificate holder to have programs for aircraft inspection, preventive maintenance, and oversight of alterations that ensures these tasks are performed in accordance with the certificate holders manual. The certificate holder is further required to ensure that these tasks are performed by competent personnel with adequate facilities and equipment. Finally, the certificate holder is ultimately responsible to ensure that each aircraft released to service is airworthy and properly maintained. " FAR Part 121.373 requires each certificate holder to establish and maintain a program to monitor the performance and effectiveness of its inspection and maintenance program. The certificate holder is responsible for correcting any identified deficiencies in those programs. " FAR Part 121.379 establishes the authority of an air carrier to perform aircraft maintenance in accordance with an FAA-accepted maintenance manual. The maintenance can be performed by the carrier itself or another approved facility. If maintenance requires a major repair or major alteration, the work must have been accomplished in accordance with FAA-approved technical data. Manufacturers are likewise required to provide the technical instruction necessary to support continued airworthiness of their aircraft. This obligation is outlined in FAR Part 25. " FAR Part 25.1529 requires instructions for continued airworthiness as part of type certification. The instructions may be incomplete at type certification if a program exists to ensure their completion prior to delivery of the first airplane or issuance of a standard certificate of airworthiness, whichever occurs later. " FAR Part 25: Appendix H elaborates on specifications for the technical information required by FAR Part 25.1529. The instructions must be written in English, providing for practical arrangement, and in a form appropriate for distribution. Instructions must include information about all equipment installed on the aircraft, including equipment made by third party manufacturers. Manual content requirements are outlined for system 5 descriptions, maintenance and inspection procedures, required schedules, and information about system tests and service points. While the FAA has established a need for airframe maintenance manuals in a required maintenance program, exact regulatory requirements for those manuals are not outlined. The manuals must be accepted as part of the maintenance program, but the manual data itself is not approved. Only portions of the maintenance technical information require direct regulatory approval (e.g., Structural Repair Manual) while the majority of maintenance information does not. Considering the massive amount of maintenance information and the potential degree of variation in manuals from one operator to another, direct regulatory approval of all procedures is not desirable. From the regulatory perspective, the intent of a maintenance procedure is more important than the particulars of its execution. However, the lack of detailed standards underscores the responsibility placed on the manufacturer and, to a lesser extent, the operator to develop and maintain the quality of information contained in manuals. 2. DISCUSSION . 2.1 MAINTENANCE TECHNICAL MANUAL CONSIDERATIONS . 2.1.1 Development . New manuals begin development during the early stages of the aircraft design process. During the design and development stages of a new aircraft program for FAR Part 121 operators, the manufacturer produces a Maintenance Planning Document (MPD) outlining the inspection and maintenance schedules that will be required by the airframe and system components. The development of the MPD is guided, in part, by the Maintenance Review Board (MRB) document developed by the FAA and industry to aid in establishing initial inspection and maintenance schedules. The process is further aided by logic developed through the Maintenance Steering Group (MSG), an industry collaboration of manufacturers, airlines, and regulatory authorities. The MSG allows for the adaptation of the MRB process to meet the unique requirements of a particular aircraft while conforming to the original intent of FAR inspection and repair regulation. The MSG logic has gone through several iterations and is now referred to as MSG-3. The entire MRB process is required for the aircraft to receive a Type Certificate and serves as an initial check of the integrity and accuracy of maintenance information. With the required inspection and maintenance schedules established, the manufacturer must begin the task of outlining those procedures. In addition to the required scheduled maintenance, procedures must also be developed for the removal, installation, and repair or replacement of all components installed in the aircraft. It is in the outlining and description of these procedures that technical writers have the greatest freedom and it is in this area that manuals vary the most. A guiding force in the style and appearance of technical manuals is the Air Transport Association (ATA), through the document guidelines they develop. The ATA specification 100, and the new i spec 2200 that replaces it, provides direction for the format and layout of technical manuals. Although not regulatory nor mandatory in nature, manufacturers, operators, and regulatory agencies have come together through the ATA to standardize the format of technical manuals. 6 This standardization extends from chapter organization to headings and fonts. The goal is to make navigation and use of manuals as consistent as possible across the industry. Once again, although there is considerable structure imposed by the ATA specifications, decisions about the content are left to the discretion of the technical publications group. For example, manuals can differ greatly in the use of constrained language (i.e., restricted vocabulary set and/or sentence structure). Technical writers have used constrained languages in an attempt to improve usability by addressing the effects of differences in reading level and language fluency among maintenance personnel. Some manufacturers use the United States Air Force dictionary that limits the vocabulary to words that should be understood by a person with an eighth grade reading level. Other manufacturers use more restrictive standards such as simplified English, to control both vocabulary and sentence structure. The simplified English standard has writing rules and a list of acceptable words, noun phrases, and verbs. The use of simplified English is meant to eliminate slang and create concise sentences. Because maintenance documentation is authored in English and normally not translated into the native language of an international operator, simplified English may make it easier for non-native English speakers to use technical manuals. Given all of these factors, two manuals, both conforming to ATA specifications, may differ greatly in clarity, reading level, degree of detail, and writing style. 2.1.2 Time . The underlying time and budget pressures that drive the manufacturers of aircraft need to be kept in mind when examining the scope of the task facing the writers of technical maintenance information. Safety is of ultimate concern to anyone in the aviation industry, but there are often trade-offs between ideal safety practices and the real-world compromises adopted. One of the single greatest factors in determining the amount of money made on an aircraft is time. An airplane sitting on the production line is not making money. The longer it takes to deliver an aircraft, the longer the delay until it begins earning revenue. Operators pressure the manufacturer to meet an established contractual delivery schedule because, in most cases, the new airplane has already been scheduled for revenue service. This economic reality is crucial to the full appreciation of the environment surrounding the development of technical documents. The time in production determines the number of aircraft that can be produced in a year and may be the deciding factor in a customer 9s choice of a manufacturer. In theory, delivery could be delayed to ensure that maintenance manuals are completed. In reality, the technical publications department within manufacturers is rarely given authority commensurate with engineering, production, or flight test departments. Delivery will not occur until all final assembly on all systems have been installed and tested, but it is unlikely to ever be delayed for the technical documentation that refers to those systems. When budgetary concerns arise, the technical publications department is often faced with funding cuts and time limitations. In most cases, the technical publications department is aware of the problems they face but lack the perceived importance within the company hierarchy to command the time and resources to implement changes. 7 2.1.3 Variations Between Manuals . During the production life of an aircraft model, engineering and equipment improvements continue to be made. Vendors supplying parts for that aircraft are also constantly working to improve their products. These improvements are incorporated into production aircraft as they become available. Because of the evolution of the aircraft design, different serializations of the same aircraft model may include different parts and, therefore, require different maintenance procedures. For any given aircraft model, there is also a variety of optional equipment that can be installed. For smaller, regional aircraft, configuration differences may be limited to interiors and avionics options. For large transport aircraft, configuration options can be extended to accommodate any desire of the customer including engines, environmental systems, and in-flight entertainment equipment. The result is that any two examples of a particular aircraft model may vary significantly. The mechanic responsible for maintaining those aircraft must be provided with the necessary information to make the distinctions between the aircraft and adjust accordingly to the task of repairing each of them. To make this as easy as possible, the manual must contain all of the information relevant to the aircraft the mechanic is working on, while at the same time not require sorting through a lot of nonapplicable information. For smaller aircraft, or those with fewer customization options, it may be acceptable to include all information into every manual and give the user the necessary data to determine what applies to their particular aircraft. For more complex aircraft, or those with a large number of customization options, it becomes necessary for the manufacturer to tailor a customer 9s manual to the particular configuration of their aircraft. When a large transport aircraft has a maintenance manual numbering in the tens of thousands of pages, unnecessary information is not tolerated by the operator. For these aircraft, manufacturers are forced to provide manuals customized to each operator. The way this is handled depends on the manufacturer. The task of the manual technical writer is to obtain all of the information necessary to incorporate both the constantly changing engineering information and any necessary customization information into the finished manual. Several sources within the organizational structure of the manufacturer need to coordinate their respective data in order for this information to make it into the manual. Significant design changes affecting the form, fit, or function of the aircraft must be communicated to all affected internal organizations. When notified of a significant design change, technical writers must modify the applicable maintenance data and incorporate that information into a manual revision. The burden is on the technical writer to ensure that the data presented in the manual reflects the actual configuration of the airplane. Individual writers are assigned to writing tasks based on their expertise with aircraft subsystems. A writer that is assigned to avionics, for example, must not only be knowledgeable about each of the avionics systems available, but also how those systems interact with other subsystems. When writing the documentation for an aircraft that has system A, they are actually writing the documentation for the hundreds of aircraft with that same system. The document must not only reflect the data applicable to that system, but it must also address any potential interaction between combinations of installed subsystems. 8 2.1.4 Document Lag . As new data becomes available, the writers must review that data and make any required additions or changes to the manual. If multiple changes are necessary, they are worked in order of importance and safety relevance. Source data continues to evolve throughout the writing process, but at some point in time, a decision must be made to finish authoring activity for the manual. Additional time is then needed to compile all of the information, format and/or print it, and send it to the operators. Once the decision point is reached, the document is said to be clocked d and no changes will be made. The clockup d date is usually 3 to 4 weeks before the intended release of the manual or revision. Revision schedules are either driven by the amount of accumulated content or a calendar cycle. At the point of lockup, the process of developing the next revision begins, starting with any data not included in the previous revision. Even though writing has ceased for that release of the manual, engineering continues to progress during the lockup period. The airplane manufacturer tries to synchronize the release of maintenance data with the delivery of a new airplane, but in some cases, design engineering data or supplier data is released after the lockup period for the current revision. Consequently, this late engineering or supplier data can cause a cdocument lag d in which the released maintenance data does not match the delivered airplane configuration. Maintenance data not included in the current revision of the manual will be prioritized for inclusion into a later revision. If safety-sensitive data or maintenance significant data (causing an operator economic or technical difficulty) emerges between the release of the current and next planned revision, manufacturers have established a process to provide a temporary revision until the next release becomes available. 2.1.5 Distribution . Once a manual or manual revision is completed, it must be converted to the deliverable format requested by the operator and distributed. For domestic operators, the distribution is straightforward. Postal delivery or parcel service can have a hard copy of the manual to a maintenance facility in a few days or less. From that point, it is the responsibility of the respective maintenance personnel to assure the new information is added to the manual and disseminated to mechanics. For international operators, the distribution process can be slowed weeks or even months depending on the efficiency of customs procedures it needs to pass through. Non-English speaking facilities may slow down the process further by translating the information into their native tongue. 2.1.6 Multiple Manual Formats . Although the size of the published manual gives a good indication of the magnitude of the task required to develop one, in practice, the manuals are frequently distributed to users in other media formats. Microfiche or microfilm cassettes are common media options for distributing manuals without the physical space requirements of paper. Mechanics can view film versions of the manual in the appropriate viewer and print the pages pertaining to the job they are assigned. 9 Although they are smaller and easier to store, film versions of the manual are identical to the paper manual and differ only in the presentation media. Although the manual can be distributed on whatever media preferred by the operator, the paper copy of the manual remains the master reference. Digital data formats are slowly replacing other media formats. Large operators prefer the ease and flexibility of digital manuals. Large amounts of electronic data can be easily transferred via network or distributed on CD-ROM. Computerized maintenance data can be easily manipulated by the operator and integrated into company manuals. If desired, electronic maintenance data can then be downloaded to a notebook or hand-held computer for use by a field mechanic at the aircraft. Digital data does not suffer from delays associated with printing and distribution of paper manuals and manufacturers could conceivably distribute daily updates via the Internet. Computerized data allows for the inclusion of multimedia enhancements not available in paper- based manuals. In spite of its potential advantages, the implementation of computerized data is inconsistent across the industry. To avoid potential problems arising from different revision schedules for paper and electronic data, electronic data is synchronized to the slower paper publication schedule, eliminating the potential distribution time savings. As a result, the full benefits of the electronic medium are currently not being realized. Aside from the problems of digital versus print data, supporting multiple data mediums requires additional effort to verify that document formatting is compatible with each type of media supported by the individual manufacturer. Each aircraft operator may request the manual in a different media to match existing hardware base (e.g., microfiche viewer, tape, CD-ROMs). Interactions between new and legacy systems (i.e., data maintained on older computer databases) can cause unforeseen irregularities in the appearance of the finished document. Finding and correcting these problems may at times require manually reviewing each new page. 2.1.7 Document Customization . Depending upon the size of the operator, maintenance manual data may be modified for the purpose of integrating into company manuals. In some cases, these operators receive the manual data in an electronic format (tape, CD-ROM, native SGML code) and then modify it to adhere with company procedures. Alternatively, the operator may provide details to the manual developer and have them produce the manual in a format consistent with approved company manuals. Customization in this case may include company procedures for handling parts, completing paperwork, or reporting problems. In other cases, customization may extend to the way procedures are carried out. When an operator specifies a procedure that is different than that outlined by the manufacturer, the manufacturer will examine it to determine whether it is consistent with what it considers to be safe procedures. If the operator requests procedures to be entered that have not been approved by the manufacturer, those procedures may be included in the manual with a notation indicating that it is customer-originated data. 10 Operator customization of the manual can create problems for both the manufacturer and the user of that manual. From the mechanics perspective, manufacturer-generated data and company- generated data are often indistinguishable. Likewise, it is sometimes difficult for the manufacturer to have a clear idea of the quality of their procedures when they have been modified by the operator. Another example of this mismatch is the generation of the work task cards or job aids generated from the manual for use by mechanics. Many manufacturers provide job aids automatically generated directly from the manual, however many operators develop their own version of job aids. User experience and feedback may be quite different depending on the source of this data and it may not be immediately apparent which party is responsible for an identified problem. 2.1.8 Feedback . Once a manual or revision is released, further changes may be required to address user feedback. The user is the ultimate check of the quality of maintenance information. If the user thinks the manual contains erroneous data, unclear directions, or difficult procedures, they can report such problems back to the customer service department of the manufacturer. In the case of larger operators, concerns raised by mechanics are typically channeled through the operators engineering department before they are passed on to the manufacturer. The integration of user feedback is primarily a reactive process, in which the data is released, and remains unchanged unless problems are reported. Except in the case of safety of flight, responses to user feedback may differ based on the unique requirements of an operator. In the case of customized manuals, if one operator wishes to change the way a particular procedure is performed, they will make a request and only their manual will be changed. The requests of an operator will reflect the efforts of their own engineering departments and are typically not shared with potential competitors. Manuals for a new aircraft may start out very similar, but over time the unique feedback generated by the mechanics and engineering departments of each operator cause them to diverge. Many operators invest large amounts of time and resources into customizing their manuals and do not wish that information to be shared with potential competitors, so this process is repeated with each operator, requiring the manual developer to continually address similar issues. As a result, manufacturers are not only providing manuals to support each aircraft they produce, but multiple variations of that manual to address the unique desires of each individual operator. Another downside of this practice is that potential improvements in the manual are not being disseminated throughout the industry. The requirement for unique user feedback is primarily true of operators of large aircraft, but regional operators may make similar requests. In an attempt to prevent the repeated occurrence of similar errors in the manual development process, some manufacturers have developed methods for receiving, categorizing, and cataloging identified problems. Once recorded, problems can be tracked to identify trends in errors that point to potential shortcomings in a manufacturer 9s manual development process. Whether or not a database of errors is maintained, continual monitoring of customer feedback is a critical part of providing a document that meets user expectations. 11 2.1.9 Aircraft Customization . Once an aircraft leaves the factory, it will continue to be modified throughout its operating lifespan. Operators make changes and updates to avionics, interiors, and passenger comfort equipment. In some cases, changes can be more drastic, such as structural modifications or cargo conversions. There is considerable variation in the degree to which maintenance information is updated to reflect those changes. If the aircraft was returned to the manufacturer or a factory- authorized maintenance center for modification, a fully integrated manual update is often included in the total cost of the job. For in-house changes, or work done by third-party modification centers, very little maintenance information may be available. If the operator wishes to have these changes integrated into their existing manuals they can, but because the rework was not due to a factory-initiated change, the operator must shoulder the cost. Many times the operator elects not to pay to have the information incorporated in the original manual. In such a case, the operator may be given a generic manual supplement pertaining to the modification. This supplement may contain detailed maintenance information or be limited to a basic technical description. The manufacturer may be impacted by the continued customization in two ways. First, if the operator wishes to integrate the new information in their manual, the manufacturer must communicate with the modification center to gather all the necessary information pertaining to the operation and maintenance of the new system. This can often be very difficult, depending on the quality of the engineering provided by the modification center. Secondly, if the operator chooses not to involve the manufacturer, the manufacturer may still be impacted because they are not aware of the actual configuration of the aircraft. Consequently, future attempts by the manufacturer to improve the aircraft or manual will not consider the potential interaction between factory-originated changes and modifications performed by a third party. When revising manuals, the author will write procedures based on the known aircraft configuration, usually the configuration at time of delivery. If a writer changes a procedure based on original equipment manufacturer (OEM) procedural or engineering changes, it may be inappropriate for the unique configuration of the customized aircraft. 2.1.10 Older Aircraft . The high cost of providing maintenance manual updates and revisions has led most manufacturers to implement some form of revision subscription service. In order to continue receiving updates to maintenance manuals, operators are usually required to pay for a subscription to that manual. A subscription entitles the operator to receive all revisions and updates as they are released. Most manufacturers also include a regular newsletter that includes information about pending revisions and model information. Operators that opt not to pay for a subscription receive only information considered to directly impact the safety of flight. By far, the greatest numbers of maintenance manual changes come early in the life of a new aircraft. The time pressures of delivery often result in the latest engineering changes being left out of early versions of the manual. The largest numbers of problems with the manual are usually identified soon after its release, and those problems are addressed in early revisions. At some point, the number of changes begins to decline and level off. It is at this point that an 12 operator may feel that the cost of maintaining a manual subscription is not warranted by the small amount of change in each revision and may choose to suspend the manual subscription. Unless the operator chooses to later pay to have the maintenance manual updated, the manual will reflect the condition of the aircraft at the time of suspension regardless of future changes to parts information, vendors, or recommended procedures. With the age of some aircraft still in service, it is not unheard of to find operators with manual subscriptions that have been suspended for 20 or 30 years. In most cases, these aircraft are being flown by foreign airlines or have been leased to smaller operators. Because these manuals have been suspended, it is unclear whether these operators are using old manuals or newer documents purchased from a third-party manual developer. From the perspective of the manufacturer, older aircraft present a particularly costly challenge. Once an aircraft model is released, the maintenance information is open to scrutiny and must continue to be supported until the last aircraft is retired. Unlike most departments within a manufacturer, technical writers must remain knowledgeable about the design and function of systems in both production and out-of-production aircraft. For an established aircraft manufacturer, this may require them to support manuals for aircraft that have been out of production for decades. 2.2 MAINTENANCE TECHNICAL DATA ERRORS . When asked about ways of improving technical manuals most writers/developers emphasize the development of procedures to eliminate the accidental inclusion of incorrect technical information. Although accidents attributed to correctly following maintenance procedures that turned out to be erroneous may capture public interest, other more frequent and potentially more significant sources of errors receive comparatively little attention. It is important to note the difference between the academic and popular definitions of error. In academia, error is understood to encompass a wide variety of forms, including lack of clarity and the omission of information. However, when technical writers are questioned about errors, their definition of an error is typically limited to the inclusion of incorrect technical information in the manual. Quality control procedures tend to focus on identifying incorrect information, formatting irregularities, errors in grammar, and aircraft configuration errors. Using this narrow definition of error may lead one to overlook other important factors that impact the user 9s perception of the usability of a manual. Although they may garner considerable attention when they occur, the cases of blatantly incorrect information being incorporated in a manual are relatively small when one considers the size of the document and the amount of detailed information it contains. The possibility of a procedure being written that is unclear, difficult to follow, or fails to represent a mechanics mental model of the task is more likely and has the potential to be of equal or greater consequence than incorrect information. Poor usability of documents introduces a systemic potential for error due to the unpredictability of how a mechanic interprets the manual and how closely this interpretation adheres to the intent of the writer. 13 If an incorrect procedure is outlined, the error will presumably be identified and corrected in the first few attempts at doing that task. The effect of a poorly written procedure however may be much more subtle; for example, a mechanic may think they are performing a task correctly when, in fact, they are not interpreting the procedure in the way intended by the writer. In this case, an unclear procedure may lead to similar mistakes being repeated each time the offending procedure is attempted. In cases where difficulties with a procedure may be more salient, the user may be more apt to abandon the documented procedure and rely on their own judgment to perform the task. Because the procedure may be technically sound, a problem resulting from confusion with or failure to use the manual is likely to be attributed to incompetence on the part of the mechanic. Ultimately, a mechanic may be reprimanded or fired for committing actions they thought were correct in light of documentation that was difficult to follow or understand. When attempting to assess the degree of error present in maintenance manuals, it is important to have a clear understanding of what constitutes error. Reason 9s model of active and latent error [10 and 11] has been given considerable mention in the maintenance error research literature [7 and 12]. The term active error is used to describe an erroneous action or violation committed by an individual, while latent error refers to environmental factors that may contribute to error. The oft-cited cSwiss Cheese d model of error illustrates the way latent errors caused by working conditions, management policy, and organizational communication can contribute to a situation that results in an active error on the part of a mechanic. Manuals can indeed be a source of latent maintenance error, but this model is not particularly well suited to the study of errors in maintenance manuals. Maintenance error investigations focus on the failure event or inappropriate action of the maintenance personnel. Attempts at reducing active errors require an understanding of all possible errors, so that either the individual or the system can be changed to protect against those possibilities. Because the number of possible errors is potentially infinite, likely errors are usually determined by examining previous error occurrences [12]. Once an error is identified, investigators must work backward to examine all of the potential contributing effects that lead to its occurrence. They must determine how far to track the source of error and estimate a degree of influence for each contributing factor [13]. Errors in maintenance manuals are different from the erroneous action of an individual and attempts at identifying, cataloging, tracking, and reducing errors in technical information need to take a different approach. As Rasmussen points out [14], the goal of generating a taxonomy for classifying, analyzing, and addressing existing errors in a system, based on previous failures, is best suited to systems that remain reasonably static for long periods of time. Tracking the reliability of a system requires an accumulation of data over time, and if a change is introduced into the system without being controlled, the accuracy of the resulting reliability measure will be suspect. The procedures used to develop maintenance documentation have been relatively stable for a long time. However, these procedures have recently experienced substantial changes resulting from the application of new technology and changes in the demographics of the user population. From the earliest aircraft until the advent of computers, manuals were assembled by hand through cutting and pasting of text and hand-drafted illustrations. Even early applications of 14 computer technology to the development process operated merely as a more efficient version of the cut and paste, paper document assembly. Technology is changing so rapidly that in many cases legacy information is not migrated to new systems before those systems are replaced by the next version. Consequently, the way manuals are developed today is not the same as they were last year or will be a year from now. Traditional attempts at cataloging and reducing process errors may have worked well with the transfer of blueprint information to paper documents, or even early computer-based authoring systems, but are inadequate to address the problems caused by rapidly evolving technology and a changing user workforce. Because the process of developing manuals is now so dynamic, any attempt to develop a reliability measure must take into account both the changes in the development process and the target audience. Attempts to measure reliability using static measures such as recording the number of typographical errors or incorrect part numbers are necessary but no longer sufficient. Simple reliability measures may provide information about the integrity of the process but it will not indicate how well the manual meets the needs of the user [14]. The user population is not static, and as the development process continues to evolve, the manual writers must repeatedly verify that it continues to meet the needs of the user population. The information contained in the manual must be correct but to focus on technical accuracy alone is to inappropriately simplify the issue. Focusing attention on matching the task to the user would go beyond process error to include the cognitive and psychological processes guiding the task and user. Reason outlines a process for developing a framework of error that includes ccontextual error sources d introduced by the task and situation and cbasic error tendencies d of the individual [10]. As Reason points out, while there is no universally applicable classification for error that will meet all needs, a general framework of error can be defined from repeated user testing that can predict the majority of problems. The technical writers responsible for maintenance manuals are professionals who take seriously the potential safety impact of their work. It is fair to assume that their intention is to produce the best manuals possible, but in spite of these intentions, errors occur. This is important to note at the outset, as error cannot be separated from intention. The notion of intention is based on two things: (1) an expressed goal to be attained and (2) a clear procedure for how to reach that goal. Volition is fundamental to the definition of error, and therefore, the term error can only be applied to intentional actions. Intention, in this case, includes intention in action as well as prior intention. Errors then fall into one of two broad categories: (1) a failure of actions to go as intended (Norman 9s slips and lapses or Reason 9s execution failures) or (2) a failure to choose the appropriate actions to achieve the desired outcome (Norman 9s slips or Reason 9s mistakes failures) [15 and 16]. In either case, the intended outcome is not reached. While the terms used in the error research literature generally refer to an individual, they could effectively be applied to the processes of industry. In the process of developing maintenance manuals, there can be failures of execution (printing mistakes, formatting errors, etc.) or failures of planning (difficult procedures, overlooked information, etc.). Failures of execution are reduced through proofreading techniques, technical monitoring, and software document checkers. Recently, efforts intended to reduce process error in manufacturing have been applied to the 15 development of technical documents. For example, ISO 9000 certification was originally conceived as a method of improving the consistency and quality of manufacturing processes but is now being applied to the development of data as well [17]. The reduction of failures of execution requires tightly controlled procedures for data transfer and error checking. If procedures are in place to eliminate error in the transfer of data from its point of origin to its entry in the manual, the task is reduced to one of verifying that the procedures were indeed followed. If execution errors do occur, it then becomes a matter of determining whether the procedures were adequate or whether they were violated. The error source can be quickly identified and eliminated. Execution failures are the most visible type of error and the easiest to control. ISO 9000 certification seeks to eliminate execution error by documenting all procedures and then enforcing adherence to those procedures by all personnel. In its simplest form, ISO 9000 certification can be reduced to documenting what is done and doing what is documented. If problems arise, then the procedures must be adjusted to correct the problem. Just as latent error is the more difficult type of maintenance error to address, planning failures are the more difficult type of error to identify and eliminate in document creation. In the case of a planning failure, an action is executed as intended, but when the action is followed to its outcome, it fails to produce the desired result. As it applies to maintenance manuals, a planning failure would mean that a procedure is technically sound but is misunderstood by the mechanic or fails to match the way the job is actually performed. Format checkers and peer reviews may not identify planning failures because the information may be accurate and appear to follow a logical sequence but cannot be performed by the mechanic (due to physical constraints, available tools, etc.) or may be misinterpreted by the mechanic. Planning errors can be further separated by their ultimate outcome. Although the development process may suffer from planning and execution errors, a flawed procedure might have little negative impact due to some level of forgiveness or error tolerance within the system. In the case of maintenance manuals, this tolerance is afforded by the experience, skills, and knowledge of the mechanics. Experience allows a highly skilled mechanic to overcome minor problems using their expert knowledge to identify what the writer intended or to find a way to perform the described task. If the user lacks this expert knowledge, the tolerance for error is decreased and smaller errors have a greater potential for negative impact on the system. In short, any attempt to improve the way in which manuals are developed requires a thorough understanding of the processes used by the manufacturers and the problems encountered by the users. In Phase 1, the focus is on the procedures used by the manufacturers to develop aircraft maintenance manuals. 3. EVALUATION . 3.1 SURVEY METHODS . This phase sought to gain a working knowledge of the issues surrounding the development, revision, and distribution of aircraft maintenance technical manuals and the current industry 16 procedures that address those issues. This information was gathered through the cooperation of multiple aircraft manufacturers and their personnel, including: " Technical writers " Customer service representatives " Engineers " Illustrators " Department managers " Manufacturer representatives Information was collected through informal interviews and directed discussions. The topics covered in these interviews included: " The process of developing and revising maintenance manuals. " The systems and procedures used to coordinate information from numerous sources within the organizational structure of the manufacturer. " The solicitation and inclusion of user feedback into the development of technical manuals. " The means used to identify, track, and reduce error in fielded technical manuals. In addition to these direct contacts, participation in industry conferences provided an opportunity to discuss additional industry perspectives. Site visits were conducted at each of the participating organizations. These visits consisted of a series of meetings over the course of several days. To the extent possible, site visits included demonstrations of procedures and technology used in the technical publication process. Visits were normally arranged through a technical publications manager. In addition to the hosting manager, interviews included employees from a variety of areas in the technical manual process. Researchers were provided with copies of procedure guidelines, organizational charts, and related documents as appropriate. The industry sample included five aircraft technical manual producers. The five organizations included manufacturers and modifiers of FAR Part 25 aircraft, representing both regional and large commercial transport models. For the purposes of anonymity, the participating organizations will be referred to throughout this report as companies V, W, X, Y, and Z. Because of the limited information available from manufacturers concerning the types and number of identified errors, data supplied by third-party maintenance facilities was used to supplement the review of problems identified in fielded manuals. 17 3.2 INDUSTRY ANALYSIS . 3.2.1 Who Writes Manuals ? The manufacturers differ considerably with regard to what they consider appropriate qualifications for a technical writer. For example, company X hires mostly engineers while companies V, W, and Z hire a composite of certified mechanics, writers, and technicians, while company Y hires equal numbers of engineers and former mechanics. The make-up of the technical writing staff is potentially important, as the knowledge base of the writer may not match that of the user. In some cases, an engineer 9s cognitive representation of a mechanical system may be very different from that of a mechanic. 3.2.2 New Manual Development . The procedures used by the surveyed companies to develop manuals differed primarily with regard to the systems used to communicate between engineers, writers, and operators. This section is summarized in table 1. TABLE 1. NEW MANUAL DEVELOPMENT Company V " All communication and transfer of engineering information is handled through a centralized computer database. " Engineering and technical publications are separate departments. Company W " Engineers and technical writers are collocated for new projects. " Computers are used for authoring, but integrated team meetings are used to facilitate communication. Company X " Engineers and technical writers are collocated for new projects. " Multiple computer systems are used for authoring. " The majority of information transfer is handled through centralized computer systems. " Integrated team meetings are also used to facilitate communication. Company Y " Multiple computer systems are used for authoring. " Information transfer is handled through centralized computer systems and face-to-face meetings. Company Z " Computers are used for authoring. " Memos, face-to-face, and electronic communications are used to transfer data. For each of the reviewed manufacturers, the process of writing a new manual begins during the early planning stages of aircraft development. As the preliminary details of the aircraft systems are established, descriptions of aircraft systems are developed for use in the manual. To the extent possible, a framework is then developed with the anticipated information required to maintain those systems. This framework is based on established MSG logic and any similarity with existing models. From this point on, any differences between the organizations in the initial 18 development of manuals reflect their unique approaches toward communication between various groups including writers, engineers, and customer service. At company W, the initial development of the aircraft concept involves technical manual writers. When a new project is first proposed, a senior writer is assigned to lead manual development and is involved in the decisions regarding budget, systems, and component vendors. The senior writer has the opportunity to set requirements for the data that vendors and suppliers must provide. The senior writer is also part of an integrated product team that tracks deadlines, changes to engineering, or potential problems. The product group is a means for immediately communicating any pertinent information to all affected parties. Communication between engineers and writers is facilitated by first assigning a writing team to the new project and then having them physically move their offices to collocate with the development engineers. In this way, technical writers are aware of proposed engineering changes and can anticipate how those changes may impact their schedules. A more subtle effect of this arrangement is that technical writers are aware of the issues driving engineer

less