Stephen D. Persell MD, MPH1
David W. Baker MD, MPH, FACP2
Kevin B. Weiss MD, MPH, FACP3
1Assistant Professor, of Medicine, Northwestern University Feinberg School of Medicine
2Associate Professor of Medicine, and Chief, Division of General Internal Medicine, and Co-Director, Institute for Healthcare Studies, Northwestern University Feinberg School of Medicine
3Professor of Medicine, and Director, Institute for Healthcare Studies, Northwestern University Feinberg School of Medicine
Performance in health care is the degree to which desirable objectives are accomplished. Performance measurement can inform quality-improvement activities and allow health care consumers and commercial health care purchasers to hold physicians and health care organizations accountable for the services they provide. Over the past decade, the methodology supporting performance measurement has matured. With this maturity has come an increasing array of performance measures covering a range of care settings and specialties; the increasing information has given rise to an expanded interest in the application of performance measurement. The Centers for Medicare and Medicaid Services (CMS) and large commercial health care purchasers now pay close attention to publicly released performance data. At present, the vast majority of patients in the United States who are enrolled in managed care organizations or Medicare receive health care from health plans, hospitals, nursing homes, or ambulatory care centers for which publicly reported performance data are available; the trend toward the public release of performance information is likely to accelerate in coming years. Physicians may increasingly find themselves the targets of efforts to profile the care provided by group practices or individual physicians. It is likely that the trend toward performance measurement will have increasingly noticeable effects on the ways in which health care is delivered and physicians are compensated.1
Why Measure Performance?
Ideally, individual health care professionals would be able to deliver high-quality care as a matter of course, and performance measurement would be unnecessary. Unfortunately, the best practices often are not followed, and patients frequently do not receive indicated services for acute problems, chronic illnesses, and preventive health care.2 Measurement of clinical performance is needed to assess the quality of care and to compare what is achieved with what is desired. This information can then serve the related but distinct purposes of internal and external quality review. First, measurement can enable internal quality improvement—in other words, deficiencies in health care can be identified and physicians and health care organizations can implement changes to improve quality and track their progress.3 Second, the public release of performance information can increase accountability of health care providers by (1) allowing health care purchasers and consumers to make informed health care choices and (2) permitting regulators responsible for licensure or accreditation to evaluate the quality of care offered by specific health care organizations. Both purposes figure prominently in the Institute of Medicine's Strategic Framework for a national quality measurement and reporting system.4
In the past, performance measurement was often conducted through the use of implicit review to assess the quality of care provided. In this method, a physician reviewed patient records and judged whether appropriate care had been provided. This approach to performance measurement is poorly reproducible because implicit review involves a level of subjectivity, and physician reviewers frequently do not agree.5 Currently, the most widely used quality measures define explicit criteria against which performance is judged. Although no performance measure is ideal, well-developed measures share three characteristics6: (1) they are based on strong clinical evidence; (2) they depict uncontroversial clinical practices that have broad consensus among physicians; and (3) they incorporate agreed-upon standards for determining satisfactory performance.
For performance measurement to be undertaken, it must be feasible to collect the necessary data. Methods of data collection that are too burdensome limit a measure's utility. When used for comparisons of health care providers, performance measures should be attributable to the physician or organization being assessed. Furthermore, sufficient numbers of patients should be assessed to allow for meaningful statistical comparisons; if required, statistical adjustments should be made to account for confounding variables in the collected data.6Several measures are used to assess the quality of health care; these include measures of clinical performance, measures of patient experience, and measures of efficiency.
MEASURES OF CLINICAL PERFORMANCE
Measures of clinical performance can assess health care structures, processes, and outcomes.7 Structural measures are not measures of clinical performance per se; rather, they describe characteristics of physicians, hospitals, or other health care organizations. Structural characteristics are selected for assessment because they are perceived to be associated with favorable clinical outcomes. Examples of structural measures include the ratio of nurses to patients in the hospital,8 whether a board-certified critical care physician is available in the intensive care unit of a hospital 24 hours a day,9 whether a hemodialysis center is operated as a not-for-profit facility,10 and whether a hospital has a computerized drug-order entry system.11,12
Process measures assess specific components of the encounters between physicians and patients: Was a screening test for colon cancer obtained? Was a beta blocker prescribed to a patient with a myocardial infarction? Did a patient who was hospitalized for a mental illness receive an outpatient appointment promptly after discharge?13
Outcome measures are direct assessments of patients' health status. Examples include whether a nursing-home patient has a pressure ulcer and whether death occurs within 30 days after coronary artery bypass graft surgery.14,15
Structure and process measures have some advantages over outcome measures. Because they are to a greater degree under the control of a single health care organization or individual physician, structure and process measures are less likely than outcome measures to require adjustment for confounding differences between patient groups—a statistical correction referred to as case-mix adjustment, or risk adjustment [see Case-Mix Adjustment, below]. In addition, structural measures often require very little data collection, as compared with other measures.
Conversely, a limitation of both structure and process measures is that they need to be causally related to desirable health outcomes to be valid measures of health care quality. If structure or process measures are not directly related to desirable health outcomes, efforts made to improve these measures may merely increase the costs of care without improving patients' health status. An advantage of outcome measures is that they directly measure the ultimate objectives of health care—clinical outcomes; however, they are often dependent on the characteristics and actions of individual patients. Observed differences in outcomes may be largely driven by factors that are not under the control of physicians or health care organizations. Patients who receive excellent care may still have bad outcomes, and physicians caring for patients of lower socioeconomic status or educational attainment, who are known to be at higher risk for disease severity and who lack access to care because of mechanisms such as cost sharing of copayments and high deductibles, may falsely appear to be providing inferior care.16,17,18,19
PATIENT EXPERIENCE AND SATISFACTION MEASURES
Although expert-derived clinical measures may best assess the technical aspects of care, consumers of health care are in the best position to evaluate their own experience and level of satisfaction with the services they receive. Furthermore, care that is timely and well received by patients may lead to better health outcomes. Examples of patient experience measures include the ease with which medical advice can be obtained by phone, the number of times a patient must wait more than 30 minutes past an appointment time to see a physician, and the rating of a personal physician on a scale of 0 to 10.20 The Consumer Assessment of Health Plans Survey (CAHPS) is a widely used series of surveys to assess patients' experience of health care.21 CAHPS captures patient satisfaction with the delivery of care occurring during office visits, the level of assistance obtained from health-plan customer service, the perceived accessibility and timeliness of medical and reimbursement services, and the clarity and timeliness of health advice provided by the physician. CAHPS is one form of evaluation used by the National Committee for Quality Assurance (NCQA) to evaluate health plans.14
Efficiency measures focus on the costs of delivering health care. When combined with measures of clinical performance, efficiency measures assess the value of health care (i.e., the quality of care delivered per unit cost). The methodology for assessing efficiency is less well developed than that of quality measurement.22 Common units of efficiency measurement (e.g., cost of health plan per member per month and cost per episode of care) do not capture the quality of the care received for that cost.
From Quality Improvement to Accountability
Performance measurement can support quality improvement in several ways. Measuring performance and relaying this information to the physicians who were assessed can produce beneficial, albeit modest, improvements in quality.23 Within health care organizations, reliable measurements of quality are necessary to assess the impact of quality improvement initiatives such as clinical-reminder systems, disease-management programs, clinical-decision support systems, patient-directed programs, and multimodality interventions. Because they are in a position to implement system-level changes, health care organizations may be in a better position than individual physicians to act on performance data and institute quality improvement.24,25
Using performance measurement to hold physicians and organizations accountable for the care they provide is a very different undertaking from that of quality improvement. Although publicly released reports on quality performance may prompt physicians to improve performance, they are intended for an audience external to the health care team.26 The goal of measurement for accountability is to enable comparisons of different health care organizations. For this reason, the measures used in assessing accountability must be standardized across sites. In the United States, private organizations such as the NCQA, the Quality Consortium of the American Medical Association, the Joint Commission on Accreditation of Healthcare Organizations, and the National Quality Forum (NQF) have taken leading roles in the development of standardized performance measures and the dissemination of performance results. Government agencies, such as CMS, are also a major source of performance data. Consumers and commercial health care purchasers can find comparative information on health plans, hospitals, nursing homes, and hemodialysis centers on the Internet or through health insurers.13,14,15
The use of publicly released performance data is not limited to consumers or purchasers of health care. Increasingly, physicians may find their remuneration influenced by their measured performance. The Institute of Medicine has issued a call for government health care payers to reward high-quality care with increased payments.1 Nongovernmental organizations also are adopting pay-for-performance (PFP or P4P) plans. For example, the Integrated Healthcare Association, a California consortium of medical groups and health plans, distributed $50 million in bonuses to health care providers on the basis of performance in the first year of the consortium's pay-for-performance initiative.1,27 In the United Kingdom, the government has instituted an ambitious pay-for-performance initiative in which family practitioners will receive from the government an additional £1 billion ($1.8 billion) more than they ordinarily would have received; this represents a 20% increase over the previous year's family-practice budget. The additional funding will be distributed to family practitioners on the basis of a combination of three factors: their performance on a variety of clinical measures, organizational indicators, and patient surveys.28 Both positive and negative consequences of pay-for-performance have been predicted,28 but the full impact of the program remains to be seen.
Aside from influencing payments, performance measurement can also be incorporated into the accreditation process. For example, the American Board of Internal Medicine has incorporated a module on self-assessment of the quality of care provided by the physician as an option by which to meet the 2006 practiceassessment requirement for recertification in internal medicine.29
As the use of performance measurement increases, continued awareness of its limitations is vital. There remain both methodological and practical limitations to successful performance measurement.
FACTORS AFFECTING DATA UTILITY
Collecting the data required for performance measurement can be labor-intensive and expensive. Measures that rely on data collected for other purposes (e.g., administrative data) are not burdensome to implement but may not be as reliable as data collected explicitly for quality measurement. Expanded use of clinical computer systems and electronic health records is one way to facilitate the collection of clinical data of high quality,28 but the implementation of new clinical computer systems is costly.
Use of publicly released data to inform consumer choice or to increase provider accountability also has its drawbacks. One significant problem is that publicly available data may include falsified results. To help maintain the accuracy of data reported to such organizations as the NCQA, auditing is required. Some falsification, however, may be undetectable by audits (e.g., a physician's falsely recording a blood pressure measurement that is below a quality goal). The potential exists for physicians to adjust their performance measures by the selection or dismissal of patients. For example, physicians may dismiss nonadherent or outlier patients (i.e., patients whose data lie far outside the central statistical mass) to improve their measured performance.30 Physicians who will be judged by clinical outcomes such as mortality following coronary artery bypass graft sur gery may avoid taking on patients at high risk for bad outcomes. In one instance, following the release of publicly reported cardiac surgery mortality, cardiologists reported that it became harder to find surgeons willing to operate on high-risk patients who needed surgery.31
Another potential problem in performance measurement is insufficient sample size. For any single measure, there may be too few patients for statistically meaningful comparisons, either between providers or between the performances of a single provider over time. An insufficient sample size is frequently a factor when individual physicians are profiled. When statistically meaningful differences cannot be detected because of limited sample size, those who report data should not suggest that providers with statistically insignificant differences in quality scores differ from one another. Combining several years of data is one way to compensate for insufficient sample size; however, this reporting method makes the reported results less timely and may obscure improvements in quality that occur in the short term.
The differences in patient characteristics (referred to as case mix) can introduce systematic biases into quality measurements; case-mix adjustment to correct these biases is an important methodological consideration in the utilization of performance data. Apparent differences in the quality of care received by patients cared for by different physicians may disappear when differences in patients' socioeconomic status or education are taken into account.16,17,18,19 Case-mix adjustment is especially important for measures of clinical outcome or cost efficiency, but it may alter the interpretation of data pertaining to process measures.
Statistical methods can be used to improve the validity of statistical inference when sample sizes are small or when the case mix differs across providers; however, it is not clear whether the methodology to correct for these variables will be generally adopted or, if adopted, applied in uniform ways.32
Public reporting of performance data may be of limited use to health care purchasers and consumers if poor performers can choose not to report their data. It has been noted that some health plans that participated in the NCQA's Health Plan Employer Data and Information Set (HEDIS) withdrew from HEDIS when they performed poorly.33
Inadequate Use of Data
Even when public reports of health care quality are available, purchasers and consumers may not put them to good use.34 Consumers often do not access the performance data that are available and may have difficulty making sense of the information when they find it.25,35,36
METHODS TO IMPROVE PERFORMANCE MEASURES
Although all these methodological issues and limitations are formidable, the organizations developing and promoting such measurement seek to identify and promote only those measures meeting clear criteria. An example of this move toward standardization is the recent development by the NQF of a set of voluntary consensus criteria by which to endorse physician-focused ambulatory care performance measures. The NQF's consensus standards include criteria to judge the importance, scientific acceptability, usability, and feasibility of proposed performance measures.37
Issues for the Physician
Although encumbered by methodological and practical issues, performance measurement is a reality of modern medical care. Experts caution physicians that they must be leaders in quality measurement and performance improvement or be vulnerable to challenge by economic or political stakeholders and to the potential loss of patient confidence.38 Because it is likely that we physicians will increasingly be held accountable for the quality of care we provide, it is to our advantage to have a working knowledge of current performance measurement activities in our specialty [see Sidebar Internet Resources for Performance Measures]. By understanding the types of measures by which we will be judged and knowing how the data are collected, we may be able to improve our performance—as well as the documentation of the data needed for proper measurement of that performance—without greatly increasing our work.
Internet Resources for Performance Measures
Agency for Healthcare Research and Quality (AHRQ)
A governmental agency and information clearinghouse on issues of quality, safety, efficiency, and effectiveness of health care
Ambulatory Care Quality Alliance (AQA)
Initially convened by the American Academy of Family Physicians, American College of Physicians, America's Health Insurance Plans, and AHRQ, the Alliance consists of a large body of stakeholders, including physicians, consumers, health care purchasers, and health plans
Centers for Medicare and Medicaid Services (CMS)
Provides hospital quality information to consumers and others in initiatives designed to improve hospital care in the United States
National Committee for Quality Assurance (NCQA)
An independent nonprofit organization that provides information on the quality of managed care plans in the United States
National Quality Forum (NQF)
A private, not-for-profit membership organization created to develop and implement a national strategy for measuring and reporting health care quality
Measures and Tools
National Quality Measures Clearinghouse
Public repository for evidence-based quality measures and measure sets
Measuring Healthcare Quality
Includes National Health Care Quality reports, AHRQ quality indicators, and ambulatory care clinical-performance measures
National Voluntary Consensus Standards for Ambulatory Care
Appendix A contains physician standards for ambulatory care pertaining to asthma and respiratory conditions, heart disease, hypertension, prenatal care, and prevention
Case studies that examine performance measures in such topics as chronic illness, women's health, preventive care, and behavioral health
Quality Improvement Initiative Tools
Templates of the tools used by health plans to help implement their health care improvement initiatives
Physician Quality Reports
Includes measures used to qualify physicians for the Diabetes Physician Recognition Program, the Heart/Stroke Physician Recognition Program, and Physician Practice Connections
Hospital Quality Measures
Presents hospital quality measures, inpatient quality indicators, and prevention quality indicators
When possible, individual physicians should assess the quality of the care they provide and correct their own deficiencies. They should consider making systematic changes that facilitate quality measurement and internal quality improvement, such as making use of disease registries, flow sheets, or computer systems, to track the quality of chronic disease management and preventive care. It is unlikely that the costs of implementing these changes will be recovered in the short term; however, if pay-for-performance becomes widespread, physicians and health care organizations that have not prepared for it by routinely measuring their own quality of care will find themselves at a distinct disadvantage in comparison with their competitors who have.
Editors: Dale, David C.; Federman, Daniel D.