SMS GAP ANALYSIS FORM
1.
SAFETY POLICY AND OBJECTIVES 1.1 – MANAGEMENT COMMITMENT AND RESPONSIBILITY 1.2 – SAFETY ACCOUNTABILITIES OF MANAGERS 1.3 – APPOINTMENT OF KEY SAFETY PERSONNEL 1.4 – SMS IMPLEMENTATION PLAN 1.5 – COORDINATION OF THE EMERGENCY RESPONSE PLAN 1.6 – DOCUMENTATION
2.
SAFETY RISK MANAGEMENT 2.1 – HAZARD IDENTIFICATION PROCESSES 2.2 – RISK ASSESSMENT AND MITIGATION PROCESSES 2.3 – INTERNAL SAFETY INVESTIGATIONS
3.
SAFETY ASSURANCE 3.1 – SAFETY PERFORMANCE MONITORING AND MEASUREMENT 3.2 – THE MANAGEMENT OF CHANGE 3.3 – CONTINUOUS IMPROVEMENT OF THE SAFETY SYSTEM
4.
SAFETY PROMOTION 4.1 – TRAINING AND EDUCATION 4.2 – SAFETY COMMUNICATION
November 2006 1
Gap Analysis Model Form Element Number
ICAO SMS Framework
Response If yes, state where (Yes/No) the requirement is
Describe actions needed to establish SMS compliant element
addressed.
Person responsible for action
Estimated completion Date
SAFETY POLICY AND OBJECTIVES
1.1
Is a safety management system with defined components established, maintained and adhered to? Evidence of Completion:
No
Is the safety management system appropriate to the size and complexity of the organization? Evidence of Completion:
No
Is there a safety policy in place?
Yes
Have safety objectives been established?
No
November 2006
A SMS document has been drafted but not yet approved. Many parts of the system infrastructure are in place but adherence to requirements is not yet adequate. The system that is in place is not yet mature and may be overly complex given the size of the overall operation.
Complete all required documents. Establish logical procedure framework. Establish improved accessibility of all procedures for affected departments and their employees
Sun Li
9/30/2010
Simplify/consolidate procedures. Identify opportunities to reduce requirements in procedures that may not be absolutely necessary based on real risks.
Sun Li
9/30/2010
Policy 4 outlines the basic expectations and RAA’s Specific objectives have not been identified by LT, have not been communicated and Metrics have not been selected and published.
No further action needed Andrew Ho (in consultation with LT)
6/30/2010
Establish and communicate safety objectives and metrics that measure progress to achieving each objective.
2
Are safety objectives publicized and distributed?
No
Is there a formal process to develop a coherent set of safety goals?
No
Is there a formal process to develop and maintain a set of safety performance indicators and safety performance markers?
No
Has the organization based its safety management system on the safety policy?
Partial
Is the safety policy approved by the accountable executive?
Yes
November 2006
Cannot be communicated if not yet established by LT There is no process for planning by LT, nor process for identifying goals based on risk and objectives Only early states for establishing outcome performance measures for occ. Safety, none for aviation safety or environment. No planning and review a part of Quality and Safety meeting Policy and procedures are generally aligned, however little thought has gone into the strategy for reliable deployment and ongoing checking by LT. current process is mostly ad hoc Policy 4 coordinated with LT and approved by CEO
Establish communication schedule and plan for objectives. Create SMS badge card that lists objectives Establish SMS goals and include goals on SMS badge card to ensure communication to employees
Chen Sheng
7/30/2010
Andrew Ho (in consultation with LT)
6/30/2010
Formally establish metrics for goals and objectives. Communicate progress to goals and objectives during all hands meetings. Also post metrics performance on employee bulletin boards.
Sun Li
6/30/2010
Establish policy and procedure framework that is in harmony of the SMS framework.
Business Management
8/30/2010
No further action required
3
Is the safety policy promoted by the accountable executive?
Yes
Is the safety policy reviewed periodically?
Yes
Is there a policy in place that ensures that employees are free to report safety deficiencies, hazards or occurrences without being subject to unjust discipline? Does the accountable executive have responsibility for ensuring that the safety management system is properly implemented and performing to requirements in all areas of the organization? Does the accountable executive have control of the financial and human resources required for the proper execution of their SMS responsibilities?
No
November 2006
Partially
Yes
CEO is engaged in ongoing review of safety program elements, issues and corrective actions. Accountability for LT could be improved. There is a schedule for review of all safety documents including policies and procedures Policy is in draft but not yet deployed or communicated
No further action required
Publish Employee Concerns procedure and deploy process
Sun Li
5/15/2010
CEO is engaged through weekly meetings and ongoing reviews. However, without objectives and metrics it is difficult for him to determine true performance. Overall budget is within the control of the CEO, could be improved by establishing specific safety budget lines in appropriate departments.
See response to requirements for objectives and goals
Andrew Ho
8/15/2010
No further action required
May wish to establish a detailed budget for Safety/EHS so that there are discrete budget lines to meet all anticipated spending needs.
4
Has a qualified person been appointed to oversee the operation of the SMS?
No
Assignment is informal at this time.
Hire Aviation Safety Engineer to oversee SMS project
Does the person overseeing the operation of the SMS fulfill the required job functions and responsibilities?
No
Without formal assignment with RAA’s and BG&O’s job functions remain ambiguous
Establish clear RAA’s in SMS procedure under responsibilities section
Are the safety authorities, responsibilities and accountabilities of personnel at all levels of the organization defined and documented?
Partially
Continued flux in many organizations so lines of responsibilities and reporting are ambiguous
Establish and publish a stable organization and reporting structure. RAA’s must be integrated into procedures in the responsibilities section
Do all personnel understand their authorities, responsibilities and accountabilities in regards to all safety management processes, decisions and actions?
No
No formal program kickoff and training has yet occurred
Establish training curriculum and information plan for all employees and ensure their roles, authority, responsibilities and accountabilities are communicated and understood.
Does the organization have an emergency response procedure appropriate to the size, nature and complexity of the organization?
Partially
A procedure has been published but training, drilling, and communication have not yet occurred
Establish training curriculum and drill schedule.
November 2006
5
Have the emergency response procedures been documented, implemented and assigned to a responsible manager?
Partially
Have the emergency response procedures been periodically reviewed as part of the management review of the SMS, and after key personnel and organizational change?
No
Does the organization have a process to distribute the emergency response procedures and to communicate the content to all personnel?
No
Has the organization conducted drills and exercises with all key personnel at specified intervals?
No
November 2006
A responsibility matrix has been created and some communication has occurred, however more training and communication are needed There is no process in place to assure proper maintenance of assignments, no specific person has been assigned to establish the process and maintain it. There is a lack of clarity in the responsibility for the publishing and communication of new and updated command media. No evidence of drill schedule, no table top exercises have been performed, not all drills for all possible scenarios have been planned, scheduled or performed
See above
Establish a documented, reliable process that ensures that key role players are maintained even when organization changes and personnel changes occur
Establish rational procedure frame work and ensure that RAA is establish for its maintenance and appropriate department assumes responsibility.
Establish detailed drill schedule and plans for the performance of drills including after action review process.
6
Has a documented procedure been established and maintained for identifying applicable regulatory requirements?
No
Are regulations, standards and exemptions periodically reviewed to ensure that the most current information is available?
Yes
Is there consolidated documentation that describes the SMS and the interrelationships between all its components? Does this information reside or is it incorporated into approved documentation, such as Company Operations Manual, Maintenance Control/Policy Manual, Airport Operations Manual, as applicable, and where these approved documents are not required by regulation, the organization includes the information in a separate, controlled document?
No
November 2006
Yes
No clear strategy for establishment of procedures, the framework for procedures appears to have some misalignment. All procedures are reviewed to a schedule. There is no specific process established to monitor regulatory changes that might trigger a needed procedure change There is no mapping of policies, processes and procedures for all SMS elements
Regulatory review must be integrated into the policy and procedure process to ensure that all procedures incorporated regulatory and legislative changes when procedures undergo their scheduled updates. No additional action needed
Create a map of all SMS elements showing their relationship to various documentation including policies, procedures, curricula, manuals and online resources. Publish this map on Boeing Shanghai website.
There are such manuals available.
7
Does the organization have a records system that ensures the generation and retention of all records necessary to document and support operational requirements, and is in accordance with applicable regulatory requirements and industry best practices? Does the system provide the control processes necessary to ensure appropriate identification, legibility, storage, protection, archiving, retrieval, retention time, and disposition of records?
No
Current records retention and storage is not well established nor is there a clear process owner. Records management is very poor.
Establish a written procedure and master retention schedule of records. Establish an improved records indexing and storage process.
No
No retention matrix exists, the importance of records management has not been communicated, no system for archiving and indexing has been established.
See above
Does the organization have a reactive process or system that provides for the capture of internal information including incidents, accidents and other data relevant to safety and risk management?
Yes
The MET process captures on a reactive basis any safety issues and tracks them through completion. The incident reporting process has been established by procedure, compliance is not completely reliable.
November 2006
8
Is the reactive reporting process simple, accessible and commensurate with the size of the organization?
Yes
Are reactive reports reviewed at the appropriate level of management?
Yes
Is there a feed back process to notify contributors that their reports have been received and to share the results of the analysis? Is there a process in place to monitor and analyze trends? Are corrective and preventive actions generated in response to event analysis?
Yes
November 2006
The MET system is easy to access and use. The incident reporting process is a manual paper process that is appropriate for the size of the operation and the frequency of incidents. MET items are reviewed by LT monthly. Some improvement is needed for review by Operations The MET system automatically distributes initial information and updates
No further action required
No
This is in work
Establish trend analysis graphs for monthly review during Quality/Safety meeting.
Yes
MET system requires CAPA, however quality of analysis needs improvement
No further action required
No further action required
No further action required
9
Does the organization have a process or system that provides for the capture of internal information including hazard identification, occurrences and other data relevant to safety?
Partial
MET captures audit and incident information, but there is no means for capturing employee concerns. There is a process in draft but not yet deployed
Approve employee concerns procedure and deploy process. Establish analysis charts showing trends that require additional action and use this data to create MET items.
Is the proactive reporting process simple, accessible and commensurate with the size of the organization?
No
Employee concern report process is in draft but not deployed yet
See above
Is there a structured process for the assessment of risk associated with identified hazards, expressed in terms of severity, and probability of occurrence?
No
No systematic
Establish a schedule process for performing risk analyses that includes methods, data sources and action thresholds.
Are there criteria for evaluating risk and the acceptable level of risk the organization is willing to accept?
No
Current process is completely ad hoc
See above, establish criteria for risk action threshold based on cost, morale, reputation, compliance and business impacts.
Does the organization have risk management strategies that include corrective/ preventive action plans to prevent recurrence of reported occurrences and deficiencies?
No
There is no existing proactive risk management process for use of MET
Create connection between risk management process and MET. Document connection in MET manual and other appropriate procedures.
November 2006
10
Are regular and periodic, planned reviews conducted regarding company safety performance, internal audit results, hazard and occurrence investigations, hazard and occurrence analysis results, internal/external feedback analysis/results, status of corrective actions, followup actions from previous management reviews, changes that could affect safety, recommendations for improvement and sharing of best practices across the organization? Is there a process to evaluate the effectiveness of corrective actions?
Yes
Audit results, action items, issues, corrective actions and improvement opportunities are reviewed regularly by LT during Quality/Safety meetings
No further action required
Yes
No further action required
Are proactive reports reviewed at the appropriate level of management?
No
Is there a feedback process to notify contributors that their reports have been received and to share the results of the analysis?
Yes
Corrective action effectiveness is a requirement for closure of MET items Employee concern reports will be reviewed during Quality/Safety Meetings once the process is implemented Draft procedure for employee concerns includes provisions for communicating disposition of reports
November 2006
Implement Employee Concerns Report process
No further action required
11
Is there a process in place to monitor and analyze trends?
No
No process for trend analysis of employee concerns has been documented No procedure exists for performance of assessments and audits, nor have a set of audit protocols yet been created.
Establish trend analysis similar to trend analyses of MET items
Has the organization planned self-evaluation processes, such as regularly scheduled reviews, evaluations, surveys, operational audits, assessments, etc.? Are corrective and preventive actions generated in response to risk analysis?
No
No
No risk analysis process has been established
Include requirements for MET items to be generated when actions are required as a result of negative trends identified from risk analyses
Is a process in place for analyzing changes to operations or key personnel for risks?
No
Establish and document an appropriate change management process that includes impact evaluation and mitigation for significant changes to staffing, organization structure, production process changes or changes to facilities.
Are there procedures in place for the conduct of investigations?
Yes
Do measures exist that ensure all reported occurrences and deficiencies are investigated?
Yes
There is no formal process for determining operational impact when work process changes, personnel changes or facility/equipment changes occur The procedure incident notification and investigation has been deployed including MEDA requirements See above
November 2006
Draft and approve program assessment process procedure. Create a complete set of audit protocols and post the protocols on the Boeing Shanghai website
Additional training may be required to ensure that managers and leads are proficient in the use of investigation methods and tools
No further action required
12
Is there a process to ensure that occurrences and deficiencies reported are analyzed to identify all associated hazards
No
Are corrective and preventative actions generated in response to event investigation and risk analysis?
Yes
Does the organization have a process for evaluating the effectiveness of the corrective/ preventive measures that have been developed?
No
Are corrective/ preventive actions, including timelines, documented?
Yes
Does the organization conduct reviews and audits of its processes, its procedures, analyses, inspections and training?
Yes
November 2006
There are requirements to do root cause but no standard tools or training have been created The standard process for corrective action of safety issues is MET, it has a very good process for capturing and reporting corrective actions There is no systematic method for evaluating corrective actions to determine effectiveness and for assuring a complete and permanent fix. MET captures that information, however, use of the system is still undisciplined However, this process is not yet systematic nor is there a complete set of audit protocols aligned to all procedures
See above action for training in conduct of investigations
No further action required
Establish quality criteria for evaluating corrective action effectiveness related to incident investigations
No further action required
See action concerning establishment of assessment process procedure and audit protocols
13
Does the organization have a system to monitor the internal reporting process and the associated corrective actions? Is there an operationally independent audit function with the authority required to carry out an effective internal evaluation program?
Yes
Corrective actions in MET are monitored by the LT as part of the Quality/Safety reviews The audit function is in the Quality /Safety organization which is independent of production and Facilities. Quality reports directly to the CEO. Coverage is based on risk. Audits include LM, HM, Backshop and Facilities
No further action required
Does the audit system cover all functions, activities and organizations within the company? Are there defined audit scope, criteria, frequency and methods?
Yes
No
The process is not yet systematic and well defined
See action above related to establishing an assessment process procedure
Are there selection/training process to ensure the objectivity and competence of auditors as well as the impartiality of the audit process?
No
Establish auditor selection and training criteria in the proposed assessment process procedure
Is there a procedure for reporting audit results and maintaining records?
No
No selection and training process has been established. There is no method to assess auditor competence. Currently safety staff function as the auditors. See answer to records retention question
November 2006
Yes
No further action required
No further action required
See above, reporting of results to be included in the procedure
14
Is there a procedure outlining requirements for timely corrective and preventive action in response to audit results?
No
Is there a procedure to record verification of action(s) taken and the reporting of verification results?
Yes
Does the organization perform periodic Management reviews of safety critical functions and relevant safety or quality issues that arise from the internal evaluation program?
No
Are there communication processes in place within the organization that permit the safety management system to function effectively?
Yes
November 2006
No time limits or aspirational goals have been established for timeliness of corrective action. Many corrective actions languish unless specifically highlighted during the Quality/Safety Review MET has data fields for verification of actions for CAPA items. However, no specific requirement to attach proof of corrective actions to MET. There is no analytical risk assessment model established that would permit such a management review.
See above, include timeliness of corrective actions in the assessment process procedure
Safety bulletins in the newsletter, Quality/Safety Meetings, MET, All Hands meetings, Webpages, internal servers, emails.
No further action required
Include in MET manual directions for attachment of verification records to MET items
Incorporate periodic management reviews for safety critical items into the risk management process.
15
Are communication processes (written, meetings, electronic, etc.) commensurate with the size and scope of the organization?
Yes
Is information established and maintained in a suitable medium that provides direction in related documents? Is there a process for the dissemination of safety information throughout the organization and a means of monitoring the effectiveness of this process?
Yes
Is there a process in place to monitor and analyze trends? Are corrective and preventive actions generated in response to event analysis?
November 2006
Communication is relatively simple because facility is in one location and all employees are under one roof, easily accessible. Most information is in an electronic format, readily accessible by all employees. Dissemination processes are based on the audience. Effectiveness is not currently measured.
No further action required
No
No metrics or sample methods have been established
Establish metrics that provide visibility of the effectiveness of communication related to safety
No
Data analysis has not yet started, once analysis methods are selected, processes need to be established to make proper use of the data.
See earlier answer above concerning trend analyses. Ensure inclusion of communication effectiveness in these analyses.
Partial
Improve structure and access to procedures to simplify employee access
Establish review process to measure effectiveness of information dissemination and flowdown.
16
Is there a documented process to identify training requirements so that personnel are competent to perform their duties?
No
Is there a process that measures the effectiveness of training?
No
.Is the organization’s safety training incorporated into indoctrination training upon employment?
No
Is there emergency response and response training for affected personnel?
Yes
PTP is in work, but does not address safety, no single training matrix for safety related training has been established. Most safety training does not require an exam There is no formal, standard orientation checklist or materials. Only quick review of safety manual. No shop specific safety orientation. An emergency response course exists for production and production support employees.
Establish safety training matrix related to job titles and departments. Establish a regular review schedule to ensure that employees are up to date in their attendance of classes. Establish course effectiveness criteria and incorporate into the site training process. May include requirements for pre and post testing or a process for a test out option. Improve current process for initial safety orientation for all new employees.
Perform needs assessment for training of other employees and establish appropriate course curriculum based on needs.
-----------------------------
November 2006
17