Using
Large Scale Databases to Measure Out comes in Critical Care
Peter Pronovost, MD, PhD*
Derek C. Angus, MD, MPH
* Assistant
Professor of Anesthesiology and Critical Care Medicine, Surgery, and Health
Policy and Management. Johns Hopkins
University, 600 N Wolfe Street; Meyer 295; Baltimore, MD
Health
Delivery and Systems Evaluation Team (HeDSET); Department of Anesthesiology and
Critical Care Medicine; University of Pittsburgh; Pittsburgh, PA
Running head: Data Sources for Critical Care Outcomes Research
Keywords:
Outcomes
research, critical care, observational research, database management,
randomized clinical trials.
Address for
correspondence:
Derek C. Angus,
MB, ChB, MPH
Room 606B Scaife Hall
Critical Care Medicine
University of Pittsburgh
200 Lothrop Street
Pittsburgh, PA 15213.
Tel: (412) 647 8110
Fax: (412) 647 3791
Email: angus@smtp.anes.upmc.edu
Introduction
For a long
time, critical care research has focused on mechanism of action studies in the
laboratory and efficacy studies inn small groups of patients to explore the
effects of specific therapies. With the
reorganization in health care financing and delivery, there is a need to
understand what works in medicine, and researchers are now asking such
questions as, does the way intensive care is delivered affect outcome (open
versus closed), does a full-time intensivist versus a generalist improve
outcome, do different populations access intensive care services differently,
does insurance status affect access to critical care services, and are the
organizational characteristics of ICUs associated with outcomes.{Pronovost, Jencks, et al. 1999 ID: 3283}{Angus, Linde-Zwirble, et al. 1996 ID: 934}{Knaus, Wagner, et al. 1993 ID: 1926} In order to explore these types of questions, it
is usually inappropriate to conduct a prospective study. Rather, we need to
conduct observational outcome studies on existing databases that describe the
real world.
Although the
major intent of this paper is to describe the availability and potential use of
large-scale databases, we believe it is essential to first understand the basic
principles involved in the conduct and interpretation of observational outcomes
studies. Therefore, in this paper we briefly overview the design of
observational outcomes studies as applied to critical care medicine followed by
suggested criteria for evaluating data sources and in-depth review of the
available data sources from which these observational studies can be
conducted. For further discussion of
the role of observational outcomes research in critical care, we refer the
reader to the report of the recent American Thoracic Society Workshop on
outcomes research in critical care.{Rubenfeld, Angus, et al. 1999 ID:
3284}
Observational outcomes studies in critical care
By some
estimates, critical care accounts for up to 1% of the gross domestic product of
the United States.{Jacobs & Noseworthy 1990 ID: 73}
Despite this large expenditure, there is little evidence to support much
of the daily practice of critical care.{Cook, Sibbald, et al. 1996 ID: 3201}
The rapid growth in health care costs, attributed in large part to the
increasing use of expensive technology, has forced clinicians, payers, and
society alike to consider the value of medical care.{Schwartz 1987 ID: 3202}
We can evaluate the value, or balance between cost and quality, of
medical care through outcomes studies that attempt to define what works and
what does not and that are being used to formulate practice guidelines, to
assess quality of care, and to inform health policy decisions.
Outcomes
research is designed to define what works in the real world, as opposed to
the contrived world of a randomized clinical trial. To address its questions, outcomes research relies on a variety
of disciplines including clinical epidemiology, informatics, anthropology,
economics, health services research, health policy, and biostatistics.{Curtis, Rubenfeld, et al. 1998 ID:
2811}
While it is difficult to offer an exact definition of outcomes research,
it is helpful to distinguish efficacy studies (randomized clinical trials) and
effectiveness studies (outcomes research) (Table 1). To answer their questions, outcomes researchers use a variety
of study designs and techniques (figure 1).
Although randomized clinical trials have the strongest validity to
answer is A better than B, they are often not possible to conduct due to
ethical or logistic reasons, are expensive, and do not focus on patient
centered outcomes. Observational outcomes
research is designed to evaluate interventions, providers, organizations, and
health systems.
Providers,
patients, and payers use outcomes research.
Providers use outcomes research to create practice guidelines and to
evaluate quality of care. Patients use
outcomes research to aid in treatment and health care purchasing
decisions. Payers use outcomes
research to create national guidelines, to identify ineffective care, and to
make purchasing and coverage decisions.
Outcomes research has only recently been accepted by academia, embraced
by managed care organizations, and demanded by patients.
Outcomes
research can focus on a variety of outcomes that include mortality, morbidity,
complications, symptom relief, quality of care, health status, functional
status, health related quality of life, patient satisfaction, and cost of
care. To evaluate these outcomes, the
researcher can use the following tools: formal literature review and evidence
synthesis, secondary dataset analysis, quality of life assessment, patient
satisfaction surveys, risk adjustment, patient preference assessment, and cost
effectiveness and decision analysis. Observational outcomes studies include
prospective and retrospective cohort studies and case-control studies. Several critical care observational studies
have been conducted.{Lemeshow & Le Gall 1994 ID: 796}{Knaus, Zimmerman, et al. 1981 ID:
113}{Berenson 1984 ID: 83}
Observational outcomes research can be used to
identify associations between various exposures and outcomes. Exposure is defined broadly in outcomes
research and can include medications, medical procedures, socio-demographic
status, organization of health care delivery, access to health care, region of
country, or provider. Associations
between exposures and outcomes are particularly important when there is
evidence to believe a causal relationship exists. Some observed associations are due to a confounding variable that
can obscure a true causal association or create the appearance of a causal
association when none exists. For
example, uninsured patients may have higher ICU mortality because they present
with more severe disease. Identifying
and minimizing the effect of these confounding variables constitutes a large
part of the design and analysis in observational outcomes research.
The need to
control confounding variables in observational outcomes research is in
counter-distinction to randomized clinical trials which are much less affected
by confounding variables because randomization reduces treatment allocation
bias and reduces selection bias by randomly distributing both known and unknown
confounding variables between the exposed and unexposed groups. Since observational outcomes studies do not
use randomization, they are subject to bias and the researchers must make
explicit efforts to minimize bias.
Critical care presents several unique
challenges for outcomes research. To
formulate a research question, an investigator must be able to define a
disease, treatment, patient population, or provider to study. Operationalizing these variables for
critical care outcomes research is complex.
Compare the investigator studying outcomes after myocardial infarction
to one studying the same research question in sepsis. Myocardial infarction is a common disease that is readily
diagnosed with several laboratory tests and thus is easy to define a population
of patients with the disease. Patients
with myocardial infarction receive treatments that are relatively specific for
the diagnosis and are generally cared for by cardiologists, internists or
family physicians. Sepsis, on the other
hand, is defined by clinical criteria which continue to evolve and that does
not have an accepted diagnostic test or unique treatment, and thus it is
difficult to define a diseased population.{Bone 1992 ID: 3101}{Bone 1991 ID: 3203}{Vincent 1997 ID: 1658}
Critically
ill septic patients may receive their care from a variety of physicians
including surgeons, anesthesiologists, family physicians, general internists, pulmonologists,
and pediatricians. Critical care is a
challenge to the outcomes researcher precisely because the key variables of
disease, patient population, therapy, and provider are difficult to define and
these challenges increase the risk for bias.
Can we believe the results
of outcomes research?
Observational
outcomes research, regardless of the study design, is subject to a variety of
biases and, because of this, many people question the validity of the results.{Moses 1995 ID: 3212}{Byar 1991 ID: 3213}
Biases can
be introduced in the way patient samples are selected, in the way the outcome,
exposure, and confounders are defined or measured, and in the analysis.{Sackett 1979 ID: 3214}
Readers are cautioned not to combine all such concerns into an
assessment of whether data collection was prospective or retrospective; neither
approach is necessarily superior. Instead, the reader should focus attention on
the specific nature of the bias, its direction, and magnitude. Biases do not always invalidate a
study. Rather, the reader should
evaluate how the researchers dealt with the biases. However, observational research methods do not account for
unknown confounding variables (only randomization accounts for unknown
confounders), and the reader of an observational outcomes study must address
the possibility that the observed effect (or lack of effect) is due to an
unknown confounder. Clinical
experience, sensitivity analysis, and confidence intervals all help the reader
assess the potential for an unknown confounder to bias the study results.
Criteria for evaluating datasets
Outcomes
research relies on data to answer research questions. There are encounter, enrollment, medical records, and survey data
to choose from. Below we have outlined
the criteria for evaluating datasets that are commonly used for outcomes
research.
Feasibility
A data source
may appear ideal in terms of the patients studied and the data collected. However, the data may be expensive or
impractical to acquire. For example,
Public Use Files for Medicare data can cost as much as $18,000 per year of
data.{Iezzoni 1990 ID: 3204}
Acquiring data from insurers or other proprietary data sources can be a
laborious task, complicated by considerable paperwork and by concerns relating
to ownership. Additionally, the ability
to access electronically stored data, such as warehoused laboratory data, may
be limited for technological or political reasons.
Accuracy
The way in
which data are collected and coded can introduce biases that affect the results
of particular questions. Most secondary
data sets were designed for billing purposes and these financial incentives may
increase their validity but impair their value.{Jollis, Ancukiewicz, et al. 1993 ID:
625}{Iezzoni, Foley, et al. 1992 ID: 2113}{Mitchell 1998 ID: 3285} Datasets coded by medical records
departments suffer from the vagueness of the International Classification of
Diseases (ICD-9-CM) system and the errors inherent in the abstraction process.{Romano, Roos, et al. 1994 ID: 3205}
Although it would appear that medical records are the most accurate data
source, they are not infallible.
Physicians vary in their thoroughness and diagnostic accuracy, and
medical records may not contain accurate data about processes or outcomes or
care. Such potentials for bias in a
given dataset do not preclude its use, but rather, require that the user and
reader be aware of the potential limitations.
Appropriateness
Datasets
might not be appropriate for the research question for several reasons. They may lack sufficient detail to provide
adequate severity of illness, comorbidity, or diagnosis data on patients. Studies that seek to follow patients across
multiple hospitalizations or to incorporate data from several datasets will
require unique patient identification information that may be lacking in a
particular dataset. Older datasets may
be inappropriate for researchers questioning a current practice. No single dataset will address all research
questions. As an example, the APACHE
III{Knaus, Wagner, et al. 1991 ID: 36} and MPM II{Lemeshow, Teres, et al. 1993 ID: 159} development
datasets, while rich in clinical detail, do not contain variables for answering
quality of life questions.
Data sources for observational
outcomes research in critical care
Data Sources
Just as the
methods for outcomes research differ from clinical trials, the data sources
also differ. Often times the data
source is labeled, either by the authors or the readers, as "administrative,"
a term used frequently in order to describe datasets collected for purposes
other than research. This is an
inappropriately broad term in that it groups data sources that vary widely in
form, function, and content. To address
this, we present the taxonomy recently proposed by the ATS that classifies
secondary datasets based on their administrative purpose, such as encounter,
enrollment, or clinical trial. This
taxonomy is summarized in Table 2 with examples and described in detail
below. The different databases present
varying challenges in maintaining patient confidentiality. Although this is an area of current debate,
we will not address it in this essay.
Encounter
Data
Encounter
datasets represent databases whose purpose is to maintain a record of health
care encounters, typically maintained by payers, to track reimbursement. Data are often grouped by ambulatory care
visit or hospital admission. However,
these data can be summed to generate data on a region and if unique patient
identifiers are available, can also be linked to create a longitudinal history
for individual patients.
There
is considerable variation in the quality and detail of these datasets. For example, datasets may allow different
numbers of diagnoses or procedure codes, and may vary in detail regarding ICU
utilization.{Iezzoni 1990 ID: 3204}
When data are collected for billing purposes, financial incentives
driving the data collection can impair its value but may increase validity
because coding is tied to reimbursement.{Jollis, Ancukiewicz, et al. 1993 ID:
625}{Iezzoni, Foley, et al. 1992 ID: 2113}{Mitchell 1998 ID: 3285}
Datasets
coded by medical records departments suffer from the vagueness of the ICD-9-CM
system and the errors inherent in the abstraction process.{Romano, Roos, et al. 1994 ID: 3205}
Analyses based on variables available in encounter data may miss
important relationships present in detailed clinical data.{Poses, Smith, et al. 1995 ID: 3206}
Authors presenting results from such studies should therefore clearly
specify the characteristics of the dataset and its limitations.
There are
three major sources of encounter data: federal government, state government,
and private insurance companies.
Federal Government
Medicare data
In administering Medicare, the Health Care
Financing Administration (HCFA) processes enormous amounts of beneficiary,
billing, and institutional data. Most
of the data are publicly available through HCFAs Office of Statistics and Data
Management, Bureau of Data Management and Strategy. Since 1991, all institutional providers and physician claims are
entered into the National Claims History (NCH) 100% Nearline File that contains
all claims submitted by Medicare beneficiaries. These files are enormous with over 2.5 billion records{Health Care Financing Administration 1995 ID:
3215} and require extensive skill to work
with. Researchers can purchase
confidential data extracted from the NCH files and configured into analytic
files.
A common
extracted file contains information on inpatient stays and is called the
Medicare Provider Analysis Review (MEDPAR).
The MEDPAR database contains records for 100% of the Medicare
beneficiaries who receive inpatient services.
The data elements in MEDPAR are from the UB 92 hospital claim form and
include demographic data, ICD-9-CM codes for the primary diagnosis and up to 5
secondary diagnoses, ICD-9-CM codes for the principal procedure and up to 5
secondary procedures and dates, specialty bed days including ICU days, charges
for several cost centers, reimbursement data, and data on the type of ICU and
ICU days. Most researchers purchase a
5% sample of this file that contains encrypted codes for physicians and
hospitals.
State hospital discharge
databases
Thirty-six states collect
data from hospitals on patients discharged from nonfederal hospitals.{Iezzoni 1997 ID: 892}
Although most states gather these data to help contain hospital costs,
states vary in the agencies that administrate these databases and on whether
data are required on all discharged patients or only certain patients such as
Medicare and Medicaid. The data
elements in a typical state database include demographic data, ICD-9-CM codes
for principal and secondary diagnosis (although the number of secondary
diagnoses varies), ICD-9-CM codes for primary and secondary procedures without
dates, ICU days, the hospital ID, and an encrypted patient and physician ID.
Although some states do not allow researchers access to this data, most allow
researchers to purchase the data without unique patient identifiers. The lack of a unique patient identifier
limits the ability to link patients over multiple admissions, and thus the
episode of care is limited to the hospitalization. On request, some states grant special permission to link patient
records from multiple hospitalizations, although this is becoming increasingly
difficult. Due to differences in coding
and data definitions, comparisons across states is problematic.{Iezzoni 1997 ID: 892}
Medicaid data file
Medicaid is a
combined state and federal government program and the eligibility, coverage,
and databases vary across states.{Ku, Ellwood, et al. 1990 ID: 3095}
Medicaid claims databases can be purchased but are rarely used for
outcomes research because they are large, patients move on and off Medicaid
eligibility, and there is state-to-state variability in the data
collected. Additionally, risk
adjustment may be problematic because most risk adjustment models were
developed in more affluent populations and might not apply to vulnerable
populations such as Medicaid patients.{Iezzoni 1997 ID: 892}
Private insurance claims
Many health
insurance companies have useful administrative databases.{Garnick, Hendricks, et al. 1996 ID:
3092}
Like other encounter datasets, these data are designed for business
purposes, not health services research, and may be difficult to use. However, there are companies that specialize
in obtaining data sets from managed care organizations, combining them, and
selling them to researchers. Studies
using these data sets are further complicated because data elements and
definitions vary between organizations, and the patients in these data sets are
generally younger and healthier than those in Medicare or Medicaid data
sets. Additionally, private insurance
claim data usually only include claims paid by the insurance company and thus
may not reflect the complete episode of care or the extent to which services
were utilized.
Enrollment
Data
In addition to information
about patients encounters with the medical system, one might also wish to know
about the denominator population from which the encounter numerator is
drawn. This allows the investigator to
draw inferences about population-based rates of morbidity and utilization of
health services for various diseases and to investigate changes in these rates
and utilization over time. In
calculating rates and utilization, researchers should be careful to use a
denominator that reflects the same sampling frame as the numerator. Hence, a study that uses Medicare data for
the numerator should use Medicare enrolled persons as the denominator.{Romano & Harold 1992 ID: 3098}
However, private insurance claim data are usually very good at defining
the denominator (insured population) that allows calculation of rates.
Census data
Census data
can provide age, race, and gender specific data as well as some information on
socio-economic variables at the national and regional level. Census data can be used to provide
denominator data (population at risk) or to help adjust for patient
characteristics (age, race or socioeconomic status.)
Annual zip code enrollment
file
HCFA
has data on Medicare enrollees organized by state, county, and zip code. This dataset can provide denominator data
for Medicare claims analysis.
Electronic
Clinical Data
The
electronic medical record promises to be a boon to outcomes research should a
usable format gain widespread acceptance.
There are several electronic medical records designed specifically for
use in the ICU.{Clemmer & Gardner 1991 ID: 3207}
Whether or not a formal electronic medical record exists, virtually all
clinical information in the modern hospital, including laboratory data,
pharmacy data, and many diagnostic test results are converted to an electronic
format. The extent to which data from
these separate systems can be collated into a usable resource and subsequently
analyzed for research purposes has not been explored. Barriers to the effective
use of hospital databases include their sheer volume, the lack of standardized
data collection and computer formats, lack of established quality control
protocols, and protection of confidentiality and proprietary interests.
Although electronic medical records will provide more detailed information
about care processes and outcomes, they often do not completely or accurately
reflect care rendered to the patient.{Starfield, Steinwachs, et al. 1979
ID: 3097}{Kosecoff, Fink, et al. 1987 ID: 3096}
Data Registries
Data
registries are databases focused on a particular disease or intervention, for
example, the acute respiratory distress syndrome (ARDS) or cardiovascular
diseases.{Milberg, Davis, et al. 1995 ID: 1904}{Califf & Mark 1994 ID: 3208}
Data
registries have the advantage of collecting a set of pre-defined data on a
large number of patients with homogenous characteristics. Frequently, data registries rely on
collection from a single or several academic institutions and therefore may not
generalize to other clinical settings. Because they only include patients with
the disease or treatment of interest, many important questions about incidence,
variability in management, and comparative outcome cannot be answered. A new model for a data registry has been
developed for select hemotological diseases in the Northern Region of the
United Kingdom. The population-adjusted
clinical epidemiology (PACE) model emphasizes well-controlled studies to obtain
valid numerators, adjusts the estimates for the relevant denominator population,
and uses these findings to inform clinical practice on a continuous
process. Although methodologically
elegant, PACE can be applied only where data are available about each patient,
where data can be trended, where utilization and case management data can be
linked to outcomes, and where a formal auditing system assures complete,
accurate, and timely data. Such systems
are rare.
Performance Data
In the
current era of health care reform, several organizations and agencies have
begun to collect data explicitly for the purpose of assessing institutional and
provider performance. Some of these
performance data sources are constructed de
novo, such as the Society of Cardiovascular Surgery or Greater Cleveland
Health Quality Coalition, while others, such as the Pennsylvania Health Care
Cost Containment Council coronary bypass surgery and acute myocardial
infarction reports, involve the augmentation of existing data with
clinical-based severity information.{Pennsylvania Health Care Cost
Containment Council 1996 ID: 2382}{Sirio, Angus, et al. 1994 ID: 730}{PA Health Care Cost Containment
Council 1995 ID: 2381}
There has
been considerable debate over whether the available data are sufficient to
allow for the discrimination between better and worse providers.{Iezzoni 1997 ID: 2385}
These datasets are of great potential value in that they are collected
across a number of institutions and usually contain data significantly richer
in clinical detail than other encounter datasets and thus allows researchers to
evaluate the association between care processes and outcomes.
Survey Data
Traditionally,
assessments of critical care and critical care therapies have been based on
clinical measures and objective outcomes such as mortality. Increasingly, however, attention is being
focused on the importance of patient-centered outcomes such as patient
preferences, functional status, health related quality of life, and
satisfaction with care.{Spoeri & Ullman 1997 ID: 3209}
While most surveys have not collected ICU-specific information to date,
it is likely that this information may appear in the future and be a valuable
data source for outcomes research in critical care.
Clinical Trial Data
Randomized
controlled trials (RCTs) contain a considerable amount of detailed information
on the demographics, severity of illness, and outcomes of critically ill
patients that has been collected prospectively under rigorous conditions. In addition to meta-analysis, which combines
results from several RCTs to estimate a treatment effect, it may be possible to
perform a secondary cohort analysis on the patients in the clinical
trials. This approach has been used in
clinical trials of acute myocardial infarction and pulmonary embolism to
explore the effect of different risk factors on outcome.{Mak, Moliterno, et al. 1997 ID: 3210}{Carson, Kelley, et al. 1992 ID: 3211}
While these
datasets seem promising, the same factors that limit the assessment of
effectiveness by RCTs may make extrapolation from their patient data to broader
populations problematic. For example,
it was recently demonstrated that the mortality rate in asymptomatic patients
having carotid endarterectomy is 10 fold higher in Medicare patients than the
mortality rate observed in the Asymptomatic Carotid Atherosclerosis Study.{Wennberg, Roos, et al. 1987 ID: 887} Consequently, readers and researchers should
consider carefully the applicability of these data sources for the potential
questions asked.
Prospective Cohort Data
There are
several datasets that were collected with the specific intention to be
representative of ICU populations.{Knaus, Wagner, et al. 1991 ID: 36}{Lemeshow, Teres, et al. 1993 ID: 159}
These databases are valuable sources of rich clinical detail in broad,
generally representative, critically ill populations. The principal limitations governing their use include limited
ability for investigators to access the data and the selected data elements
collected. For example, the APACHE III
database collected daily information only through day 7 and thus is not a good
source for understanding the characteristics of resource use in patients with
long ICU length of stay. Nevertheless,
the level of clinical detail, attention to ICU specific diagnoses and
variables, and large size make these datasets the current standard against which
other data sources ought to be compared for many critical care observational
outcomes research questions.
National
Minimal ICU Datasets
In part as a
response to the limitations of prospective cohort data, several countries and
national societies have developed or proposed the widespread collection of a
core set of data on all ICUs and ICU patients.{Nelson 1997 ID: 2383}
One example of a national dataset is the Society for Critical Medicines
Project IMPACT that is collecting standardized data elements in several ICUs
across the country. These projects are
ambitious and some have been plagued by a lack of funding and slow initiation. Nevertheless, since their purpose is to
understand national ICU delivery, including an understanding of the
demographics and outcomes of critically ill patients, the cost and processes of
care, and the extent to which practice variation occurs, such datasets will be
very useful resources.
Future Implications
The reorganization of health care financing and delivery, rapidly advancing technology, evidence based medicine, and an informed and empowered consumer base have all contributed to the prominent role of outcomes research in medical decision making. In the future, detailed data on medical practice and outcomes from a broad range of providers will shed new light on current practice and suggest optimal practice. In addition to patient level data, study level datasets will continue to grow and can be used to understand the effects of therapies and to help define what works and what does not in the real world. Other future advances will likely include longitudinal electronic patient records that will allow better assessment of the long-term effects of critical care, the timing of critical care within an illness, and the impact of critical illness on patients and families. Additionally, consumerism will continue to grow, and we will likely need to produce performance reports for ICUs that may drive policy and purchasing decisions. Although this is an exciting future, we must remember the strengths and limitations of both outcomes research and the large databases from which the evaluations are based.
Figure Legend
Figure 1: Study designs in outcomes research. Study designs are either experimental, where treatment allocation
is determined by chance, or non-experimental, where treatment allocation is
determined by something other than chance.
Data synthesis methods involve both data synthesis studies and
observational studies.
Table 1. A comparison of features
between traditional clinical research and outcomes research.
|
Randomized Clinical Trials |
Outcomes Research |
|
Efficacy |
Effectiveness |
|
Mechanisms of disease |
Impact of disease on the patient |
|
Many entry criteria |
|
|
Minimized selection bias |
Selection bias; needs risk adjustment |
|
The effect of biochemical and physiologic factors on bio-physiologic outcomes |
The effect of socioeconomic factors on patient centered outcomes |
|
Disease centered |
Patient and community centered |
|
Standardized protocol |
Protocol reflects practice |
|
Inventing technology |
Assessing technology |
|
Evaluates drugs and devices |
Evaluates interventions, providers, organizations, and health systems |
|
|
|
