CIMC´2000

Audit of ICUs by professionals

A French experience

Groupe de Recherche en Audit Hospitalier (GRAH)

Text written by B Guidet for the GRAH.

Members of the GRAH administration committee :

Charles Bernard, MD, private clinic physician

Gerard Bleichner, MD, medical intensivist

Richard Boiteau, MD, medical intensivist (treasurer)

Bertrand Dureuil, MD, anaesthesiologist

Alexandra Fourcade, MD, methodologist

Bertrand Guidet, MD, medical intensivist

Laurand Holzapfel, MD medical intensivist

Marie-Claude Jars-Guincestre, MD, methodologist (secretary)

Francis Leclerc, MD, intensive care paediatrician

Caroline Leroy, ICU nurse

Jean-Louis Pourriat, MD, anaesthesiologist

Jean-Claude Raphael, MD, medical intensivist

Alain Tenaillon, MD, medical intensivist (president)

Maurice Toulallan, hospital director

Correspondance : Pr Bertrand Guidet, service de Réanimation Médicale, Hôpital Saint Antoine, 75571 Paris Cedex 12, France; e.mail : bertrand.guidet@sat.ap-hop-paris.fr

Introduction :

The GRAH is a non-profit organization whose aim is to improve intensive care units.

Its goal can be summarized by the Australian Council Care standards: “ how can we do to do better what were are doing now? “ .

The GRAH organizes audits of intensive care units which have three main differences from a private audit:

- The audit is only performed if there is a formal agreement between the ICU director and the hospital director who accept the procedures recommended by the GRAH. An audit will never be organized “against” one of the two party.

- No administrative actions are taken according to the audit results without the approval of the audited ICU.

- The audit is performed by specialists in the field of ICU.

Thus, the methodology of GRAH’s audit is different from the one used for accreditation or certification. Its aim is to introduce the continuous quality improvement spirit in the ICU.

The motivation of the ICU director or administrative director could be for example to set up a project for a new ICU structure, merger of units, nomination of a new ICU director, external assessment of the quality program introduced in the unit.

1 – Composition of an audit team

The audit team is made up of six people plus a member of the GRAH’s board :

- an intensive care medicine professor who is in charge of the organization of the group and the audit report,

- an intensive care medicine physician,

- a hospital director,

- a methodologist,

- an ICU head nurse

- an ICU nurse.

- the member of the GRAH’s board who makes sure that the methodology is respected. He or she is also in charge of the logistic aspect of the audit and the preliminary meeting with ICU director and hospital director.

For every audit process, each member of the team has specific tasks according to the different dimensions covered by the audit.

- administrative data : hospital director assisted by a physician.

- unit organization : hospital director and ICU physician.

- architecture and organization of the space in the unit : ICU physician and nurse

- human resources : hospital director and methodologist

- medical care and protocols : ICU physicians and methodologist

- nursing activity : ICU nurse.

- hygiene and prevention of nosocomial infection : a physician and a nurse.

- teaching and medical research : intensive care medicine professor.

During the on site visit, the auditors are responsible for the above part. They have to check the auto-questionnaire responses filled in by the audited ICU preliminary to the on site visit. Each auditor is in charge of writing his part for the final report.

2 – Methodology of the audit

2-1- Preliminary contact between GRAH and ICU.

The request for an audit is sent to the GRAH’s President. This letter should emphasize the motivation for the audit and the expectations of the unit together with those of the administration of the hospital.

After acceptation of the audit by the GRAH’s board, documents which summarize the audit methodology are sent to the unit and the unit team is designated.
The list of auditors is submitted to the ICU director who has the choice to refuse auditors.

A formal meeting is usually organized with members of the GRAH’s board and the ICU and hospital administration in order to present the methodology and the GRAH organization.

Financial agreement is made between the GRAH and the hospital direction. The budget covers the cost of transportation, accommodations, and hours of secretary. Since it is a non-profit organization, there is no honorarium for the auditors and the budget is well below the usual cost of a private audit.

2. 2. – Documents that have to be filled in before the on site visit

The ICU has about 3 months to collect and gather all audit information required.

2.2.1. Questionnaires

Three types of questionnaires have to be distributed before the on site visit.
- Satisfaction questionnaires for all the ICU staff. These questionnaires are anonymous.

- Satisfaction questionnaires distributed to the families for five consecutive weeks.

- Questionnaires for all medical or medico-technical department directors of the hospital. These questionnaires deal with the quality of the relationship between the ICU and the other departments of the hospital and also the subjective quality assessment of the ICU.

For all the questionnaires, the results are compared with the mean results of the previously audited ICU (mean, max, median) and also with the auditor’s findings during the on site visit.

The ratio received / expected questionnaires is calculated for each staff category. This ratio provides valuable information (i.e. if only 50 % of the nurses have returned their questionnaires or if very few satisfaction questionnaires filled out by patients’ relatives are available).

2.2.2. – The reference document

This is the main document of the audit. It was constructed with the help of experts in the different fields of intensive care and updated regularly. It includes official French government texts, guidelines and recommendations originating from scientific societies dealing with intensive care medicine (UK, USA, Canadian, Australian). It does not pretend to cover every aspect of ICU. It is not intended to compare the audited ICU to an ideal ICU but rather to identify the strong and weak points of the unit in order to help the unit improve its performance, thus, it is not a certification document. This document is influenced by French culture and should be adapted to other countries. The relative importance of the different items of the reference document, to the analysis of ICU performance, have not been validated thus these figures should be used with caution.

The reference document contains one thousand two hundred items classified into ten different categories :

- Administrative data including financial information.

- Unit organization

- ICU description

- ICU architecture

- Equipment and disposable materiel.

- Human resources.

- Treatment and care

- Medical care includes the analysis of hospital records, drug prescription, quality of care assessment, and patient’s information.

- Nursing care include the analysis of nursing charts, protocols, quality assessment.

- Hygiene.

- Ethical issues, patients rights.

- Teaching

- Research.

2.3 – Preliminary meetings of the audit team

The audit team has to be familiar with the methodology in order to insure uniformity throughout all the audits and to avoid the usual pitfalls (i.e. to forget to check important items of the reference document, to waste time chatting during interviews, to make early conclusion without agreement of the whole audit team...)
During this meeting, the results of the different questionnaires and the unit responses to the reference document are discussed. This analysis eases the choice of the items that should be checked or specifically addressed during the on site visit.

A detailed letter is sent to the ICU director in order to request from him documents that should be available during the on site visit and also to organize interviews. The list of people that has to be met includes at least the hospital director, the president of the medical community, the nursing director and also all individuals who have potential conflict with the ICU. The interviews should be prepared with the help of standardized documents and specific questions in relation with the information gathered through the different questionnaires.

The on site visit has to be prepared with a clear definition of each area that should be covered by each auditor.

The quality of the preliminary meeting, attended by all auditors and leaded by a member of GRAH’s board is paramount to the success of the audit.

Each auditor fills in a document stating that the audit results will be kept confidential.

2.4. The on site visit

The audit lasts two or three days. This short period of time warrants a precise schedule with observation periods, interviews and debriefings. The whole audit team has to meet twice a day in order to outline the problems, to confront the different experiences and reflections. The information filled in the reference document (1200 items) is checked and agreements or disagreements between the unit and the auditors are noted. The information is directly entered in a lap-top computer. An ICU operates 24 hours a day, so a night visit is always organized with the interviews of the nurses and the on call physician.

The general organization of the on site visit is as followed :

- A short welcome meeting enabling everybody to be introduced and the methodology to be summarized.

- The team of auditors visits the unit, keeping in mind the specific areas that have to be covered by the each auditor. The auditors wear a badge with their name and title. The audit should not interfere with patients’ care and the work within the unit. The auditors must be careful in observing in order to gather objective information that will back up their recommendations (i.e. it should not be stated that the hygiene is insufficient in this unit but rather give examples such as dirty linens are kept close to clean linens).

- The interviews are performed as planned during the preliminary meeting. The ICU director and head nurse should be interviewed early in the audit process. Each interview should not last more than 20 minutes and should be conducted with two auditors in order to avoid personnel bias. A summary of the different interviews is annexed in the report.

- The medical records of ten patients hospitalized the previous year and those patients presently hospitalized are analyzed using scoring system.

- Special attention should be paid if a new patient is admitted or discharged during the on site visit, and during shift changes (communication, collaboration, documentation)

- An informal diner is organized with members of the unit and the auditors. This friendly diner allows open discussions and helps to understand the audit process which could be viewed as a sort of “inquisition”.

- There is no conclusion given at the end of the on site visit. This should be explained to the unit. The final conclusions, remarks and improvement proposals are in the final report that has to be validated by the different parties. So, the final meeting between the auditors and the unit leaders is rather short and not very informative.

2.5. – Audit report

The audit report is impatiently awaited by the unit and the hospital administration and because the auditors are overworked, the first draft of the report should be written very soon after the on site visit. The final report should be ready two months after the on site visit. This emphasizes the crucial importance of defining the tasks of every auditor.

The recommendations for writing the report are :

- The document is easy to read with short sentences

- The document does not include personal comments, but only facts.

- The numbers (i.e. number of admission, mortality,....) must be compared to a reference (mean of the previously audited ICU, regional, national data base). This benchmark is important since its allow to identify the weak and strong points of the unit.

- The document should not be censored since it is a working document whose aim is to help the unit to improve its performance.

The report content is organized into different chapters :

- Introduction that includes the presentation of the audit goal, the audited unit, the members of the audit team, the period and site of the audit.

- Summary of the methodology used with the different people that were interviewed and the schedule of the audit.

- Summary of the audit outlining the strong and weak points and introducing the recommendation sheets (RS)(see infra)

- Detailed analysis of each chapter of the reference document. This is the core of the report. It should contain documented information gathered during the visit, the interviews, ..... The weak and strong points, together with recommendations can be written using different characters in order to make reading of the report easier.

- Results of the questionnaires :

The three different types of questionnaires are analyzed (staff, families and other department directors). The idea is to have access to different points of view. The questionnaires are anonymous except for ICU director, head nurse and department director. The analysis of the questionnaires should keep the answers confidential. The mean and ranges for each question are compared to the mean and ranges of the previously audited ICU.

- Recommendation sheets

These recommendations cover important dysfunction with potential impact on performance and quality of care. Each of these recommendation sheets use the same structure : what is the problem ?, what are the facts ?, what are the consequences ?, what are the causes ? , what are the short or long term solutions ? This part is the most appreciated since it is very practical and enables the unit to plan modifications or improvements. A new audit, which is scheduled 2 or 3 years later should focus primarily on these recommendation sheets.

- Conclusion : it is a brief summary of the audit process and a proposal of priorities that should be addressed urgently.

- Annexes contain:

- The entire reference document with the auditors’ responses. The results are expressed as “adequacy coefficients” for each chapter which is the ratio of the yes responses to the total of responses. A ratio above 7 is considered as an acceptable ratio and a ratio below 5 as a major weakness of the unit (see results).

- Questionnaires

- List of the subjects that differ between the unit and the audit team

The draft of the report is initially a “kind of patchwork” since all the auditors write their own part. It is under the audit coordinator (ICU professor) responsibility to insure that the manuscript is conform. It is then validated by the audit team during a meeting.

The preliminary report is then sent to the ICU director for criticism. He may require some modifications or corrections. If the two parties don’t agree, then another meeting could be set up in order to settle the dispute. In case there is still disagreement, the report content is not modified but the unit comments are added to the report.

The final report is then sent to the unit, the hospital director, the audit team members and the GRAH for archives.

2.6. Advice to the auditors

- the auditors should avoid comparison between the audited unit and their own unit (i.e. in my unit we proceed differently...)

- no comments during the on site visit (i.e. these drugs are out of date)

- auditors must not interfere with the ongoing work

- every observation should be noted in order to back up the facts in the final report (i.e. hygiene procedure are not respected : a physician did not wash his hands after a septic procedure)

- every item of the reference document should be checked

- no personal comments are made during the interviews (i.e. during the meeting with the hospital director : there are too many nurses in this unit)

- confidentiality is important in order to insure a confidence and mutual respect. It is the GRAH’s responsibility to prevent information from leaking outside the unit.

- The GRAH’s audits are aimed at helping the ICU and should not be presented as an administrative audit.

3- Results

The results of 26 different ICU audits are analyzed and presented.

3.1 – Family questionnaires (figure 1)

- Ratio received questionnaires / expected questionnaires : 35 %

- Positive points :

- admission procedures and initial information (information was clear, complete and was given soon after admission)

- identification who was who

- Weak points :

- the information to explain the different treatments given during the ICU stay.

- visiting period was too short

3.2 – Physician questionnaires (figure 2)

- Positive points :

- organization, collaboration

- equipment, disposables, access to drugs

- quality of medical care

- Weak points :

- architecture and space organization

- ethical and psychological issues

- working conditions

- inter physician relationships

3.3 – Non physician questionnaires

- Positive points :

- Organization, planification

- Equipment, disposables, access to drugs

- Quality of nursing care and the nursing chart

- Weak points :

- Architecture and space organization

- Ethical and psychological issues

- Teaching and training

- Relationship with the physicians

The physicians gave higher notes for most of the items than the nurses did (figures 2 & 3).

This is particularly true for the psychological and ethical issues and also for the ICU climate and relationship between physicians and nurses.

3.4.- Questionnaires for the heads of clinical, medico-technical and administrative departments of the hospital.

The questions cover two main dimensions : subjective assessment of quality of the care and quality of the relation between the ICU and the other department of the hospital.

Table 1 : Quality of the care

	Mean	Max	Min
Medical department	8.18	9	7.6
Surgical department	8.49	9.67	7.33
Global	8.33	9.14	7.50
Standard deviation	1.26	1.55	0.74

Table 2 : Quality of the relationship between the ICU and the other departments

	Mean	Max	Min
Medical department	8.04	9	7.73
Surgical department	8.58	10	7.83
Medico-technique	8.11	9.5	5.6
administration	7.71	10	0
Global	8.29	8.93	8
Standard deviation	1.7	1.74	1.1

The ICU is considered by the other hospital departments as very autonomous, requiring very few advice from others with very little collaboration for research. So the quality of the care provided in the ICU is almost always recognized but failure to communicate well with the other departments is also reported.

3.5. – Analysis of the reference documents with the calculation of an adequacy ratio for each chapter (figure 4).

The figures used to calculate the different ratios are those collected by the auditors and not the responses of the unit. The correspondence between the unit rating and auditors rating is 0.79 + 0.04. As shown in the figure, whatever the type, size or location of the audited ICUs, the weak and strong points are similar:

- Weak points :

- No medical project, no written internal functioning rules

- No nursing activity report

- No formal meetings between nurses and physicians to discuss the budget or the activity of the unit

- Patient recruitment hasn’t been thought about before hand

- No motivation to deal with security problems (fire, electricity,...)

- Very few systematic evaluation procedures

- No specific area for material decontamination, no specific place to discuss with the families.

- Head nurse has problems in organization and job clarification

- Very little nursing research

- Organization to insure the quality of care in particular at night

- Very few physiotherapists

- Failure to deal openly and “as a team” with the ethical issues

Strong points :

- Equipment

- Drugs

- Secretary

- Medical care , records, charts, prescriptions

- Quality of nursing care

- Involvement of the personnel in the unit

It is interesting to compare the unit (physician and nurses) and audit appreciation of different items (figure 5).

The rating of the auditors is always below the unit’s one. This is particularly true concerning ethics and organization. This confirms that only an external audit can allow for an objective assessment of an ICU

4 - Conclusion

The professional approach of GRAH’s audit should be viewed as a “win – win” situation since the audit benefits the audited unit but also the auditors. The organization and active participation in audits is viewed as very beneficial for the auditors. The multidisciplinary approach with administrators, nurses and physicians in the audit team and also in the GRAH’s board is appreciated by the units and by the hospital administration. However, some limitations and difficulties should be pointed out :

- The budget necessary to cover all the expenses is increasing and justifies another type of financial organization

- The participation in the audit is time consuming

- The relation of the GRAH to the scientific societies dealing with intensive care medicine in France (SRLF and SfAR) should be clarified.

- The audit methodology requires a lot of documents, questionnaires and probably some information is redundant. A shorter and simpler version should be available in order to generalize the audit process to more ICUs.

- The ICU should be compared (and not ranked) to a reference. This implies to define clearly the different types of ICUs in order to allow for a fair comparison of similar units.

- Most of the audited units are reluctant to accept a second audit. This may be related to their failure to put into effect the recommendation sheets and the difficulties to improve the organization. Nevertheless, after a certain period of time, the audit is viewed as very helpful for the audited ICU.