ACP medicine, 3rd Edition

Clinical Essentials

Performance Measurement in Clinical Practice

Stephen D. Persell MD, MPH1

David W. Baker MD, MPH, FACP2

Kevin B. Weiss MD, MPH, FACP3

1Assistant Professor, of Medicine, Northwestern University Feinberg School of Medicine

2Associate Professor of Medicine, and Chief, Division of General Internal Medicine, and Co-Director, Institute for Healthcare Studies, Northwestern University Feinberg School of Medicine

3Professor of Medicine, and Director, Institute for Healthcare Studies, Northwestern University Feinberg School of Medicine

September 2005

Performance in health care is the degree to which desirable objectives are accomplished. Performance measurement can inform quality-improvement activities and allow health care consumers and commercial health care purchasers to hold physicians and health care organizations accountable for the services they provide. Over the past decade, the methodology supporting performance measurement has matured. With this maturity has come an increasing array of performance measures covering a range of care settings and specialties; the increasing information has given rise to an expanded interest in the application of performance measurement. The Centers for Medicare and Medicaid Services (CMS) and large commercial health care purchasers now pay close attention to publicly released performance data. At present, the vast majority of patients in the United States who are enrolled in managed care organizations or Medicare receive health care from health plans, hospitals, nursing homes, or ambulatory care centers for which publicly reported performance data are available; the trend toward the public release of performance information is likely to accelerate in coming years. Physicians may increasingly find themselves the targets of efforts to profile the care provided by group practices or individual physicians. It is likely that the trend toward performance measurement will have increasingly noticeable effects on the ways in which health care is delivered and physicians are compensated.1

Why Measure Performance?

Ideally, individual health care professionals would be able to deliver high-quality care as a matter of course, and performance measurement would be unnecessary. Unfortunately, the best practices often are not followed, and patients frequently do not receive indicated services for acute problems, chronic illnesses, and preventive health care.2 Measurement of clinical performance is needed to assess the quality of care and to compare what is achieved with what is desired. This information can then serve the related but distinct purposes of internal and external quality review. First, measurement can enable internal quality improvement—in other words, deficiencies in health care can be identified and physicians and health care organizations can implement changes to improve quality and track their progress.3 Second, the public release of performance information can increase accountability of health care providers by (1) allowing health care purchasers and consumers to make informed health care choices and (2) permitting regulators responsible for licensure or accreditation to evaluate the quality of care offered by specific health care organizations. Both purposes figure prominently in the Institute of Medicine's Strategic Framework for a national quality measurement and reporting system.4

Performance Measures

In the past, performance measurement was often conducted through the use of implicit review to assess the quality of care provided. In this method, a physician reviewed patient records and judged whether appropriate care had been provided. This approach to performance measurement is poorly reproducible because implicit review involves a level of subjectivity, and physician reviewers frequently do not agree.5 Currently, the most widely used quality measures define explicit criteria against which performance is judged. Although no performance measure is ideal, well-developed measures share three characteristics6: (1) they are based on strong clinical evidence; (2) they depict uncontroversial clinical practices that have broad consensus among physicians; and (3) they incorporate agreed-upon standards for determining satisfactory performance.

For performance measurement to be undertaken, it must be feasible to collect the necessary data. Methods of data collection that are too burdensome limit a measure's utility. When used for comparisons of health care providers, performance measures should be attributable to the physician or organization being assessed. Furthermore, sufficient numbers of patients should be assessed to allow for meaningful statistical comparisons; if required, statistical adjustments should be made to account for confounding variables in the collected data.6Several measures are used to assess the quality of health care; these include measures of clinical performance, measures of patient experience, and measures of efficiency.

MEASURES OF CLINICAL PERFORMANCE

Measures of clinical performance can assess health care structures, processes, and outcomes.7 Structural measures are not measures of clinical performance per se; rather, they describe characteristics of physicians, hospitals, or other health care organizations. Structural characteristics are selected for assessment because they are perceived to be associated with favorable clinical outcomes. Examples of structural measures include the ratio of nurses to patients in the hospital,8 whether a board-certified critical care physician is available in the intensive care unit of a hospital 24 hours a day,9 whether a hemodialysis center is operated as a not-for-profit facility,10 and whether a hospital has a computerized drug-order entry system.11,12

Process measures assess specific components of the encounters between physicians and patients: Was a screening test for colon cancer obtained? Was a beta blocker prescribed to a patient with a myocardial infarction? Did a patient who was hospitalized for a mental illness receive an outpatient appointment promptly after discharge?13

Outcome measures are direct assessments of patients' health status. Examples include whether a nursing-home patient has a pressure ulcer and whether death occurs within 30 days after coronary artery bypass graft surgery.14,15

Structure and process measures have some advantages over outcome measures. Because they are to a greater degree under the control of a single health care organization or individual physician, structure and process measures are less likely than outcome measures to require adjustment for confounding differences between patient groups—a statistical correction referred to as case-mix adjustment, or risk adjustment [see Case-Mix Adjustment, below]. In addition, structural measures often require very little data collection, as compared with other measures.

Conversely, a limitation of both structure and process measures is that they need to be causally related to desirable health outcomes to be valid measures of health care quality. If structure or process measures are not directly related to desirable health outcomes, efforts made to improve these measures may merely increase the costs of care without improving patients' health status. An advantage of outcome measures is that they directly measure the ultimate objectives of health care—clinical outcomes; however, they are often dependent on the characteristics and actions of individual patients. Observed differences in outcomes may be largely driven by factors that are not under the control of physicians or health care organizations. Patients who receive excellent care may still have bad outcomes, and physicians caring for patients of lower socioeconomic status or educational attainment, who are known to be at higher risk for disease severity and who lack access to care because of mechanisms such as cost sharing of copayments and high deductibles, may falsely appear to be providing inferior care.16,17,18,19

PATIENT EXPERIENCE AND SATISFACTION MEASURES

Although expert-derived clinical measures may best assess the technical aspects of care, consumers of health care are in the best position to evaluate their own experience and level of satisfaction with the services they receive. Furthermore, care that is timely and well received by patients may lead to better health outcomes. Examples of patient experience measures include the ease with which medical advice can be obtained by phone, the number of times a patient must wait more than 30 minutes past an appointment time to see a physician, and the rating of a personal physician on a scale of 0 to 10.20 The Consumer Assessment of Health Plans Survey (CAHPS) is a widely used series of surveys to assess patients' experience of health care.21 CAHPS captures patient satisfaction with the delivery of care occurring during office visits, the level of assistance obtained from health-plan customer service, the perceived accessibility and timeliness of medical and reimbursement services, and the clarity and timeliness of health advice provided by the physician. CAHPS is one form of evaluation used by the National Committee for Quality Assurance (NCQA) to evaluate health plans.14

EFFICIENCY MEASURES

Efficiency measures focus on the costs of delivering health care. When combined with measures of clinical performance, efficiency measures assess the value of health care (i.e., the quality of care delivered per unit cost). The methodology for assessing efficiency is less well developed than that of quality measurement.22 Common units of efficiency measurement (e.g., cost of health plan per member per month and cost per episode of care) do not capture the quality of the care received for that cost.

From Quality Improvement to Accountability

Performance measurement can support quality improvement in several ways. Measuring performance and relaying this information to the physicians who were assessed can produce beneficial, albeit modest, improvements in quality.23 Within health care organizations, reliable measurements of quality are necessary to assess the impact of quality improvement initiatives such as clinical-reminder systems, disease-management programs, clinical-decision support systems, patient-directed programs, and multimodality interventions. Because they are in a position to implement system-level changes, health care organizations may be in a better position than individual physicians to act on performance data and institute quality improvement.24,25

Using performance measurement to hold physicians and organizations accountable for the care they provide is a very different undertaking from that of quality improvement. Although publicly released reports on quality performance may prompt physicians to improve performance, they are intended for an audience external to the health care team.26 The goal of measurement for accountability is to enable comparisons of different health care organizations. For this reason, the measures used in assessing accountability must be standardized across sites. In the United States, private organizations such as the NCQA, the Quality Consortium of the American Medical Association, the Joint Commission on Accreditation of Healthcare Organizations, and the National Quality Forum (NQF) have taken leading roles in the development of standardized performance measures and the dissemination of performance results. Government agencies, such as CMS, are also a major source of performance data. Consumers and commercial health care purchasers can find comparative information on health plans, hospitals, nursing homes, and hemodialysis centers on the Internet or through health insurers.13,14,15

The use of publicly released performance data is not limited to consumers or purchasers of health care. Increasingly, physicians may find their remuneration influenced by their measured performance. The Institute of Medicine has issued a call for government health care payers to reward high-quality care with increased payments.1 Nongovernmental organizations also are adopting pay-for-performance (PFP or P4P) plans. For example, the Integrated Healthcare Association, a California consortium of medical groups and health plans, distributed $50 million in bonuses to health care providers on the basis of performance in the first year of the consortium's pay-for-performance initiative.1,27 In the United Kingdom, the government has instituted an ambitious pay-for-performance initiative in which family practitioners will receive from the government an additional £1 billion ($1.8 billion) more than they ordinarily would have received; this represents a 20% increase over the previous year's family-practice budget. The additional funding will be distributed to family practitioners on the basis of a combination of three factors: their performance on a variety of clinical measures, organizational indicators, and patient surveys.28 Both positive and negative consequences of pay-for-performance have been predicted,28 but the full impact of the program remains to be seen.

Aside from influencing payments, performance measurement can also be incorporated into the accreditation process. For example, the American Board of Internal Medicine has incorporated a module on self-assessment of the quality of care provided by the physician as an option by which to meet the 2006 practiceassessment requirement for recertification in internal medicine.29

Methodological Issues

As the use of performance measurement increases, continued awareness of its limitations is vital. There remain both methodological and practical limitations to successful performance measurement.

FACTORS AFFECTING DATA UTILITY

Data Collection

Collecting the data required for performance measurement can be labor-intensive and expensive. Measures that rely on data collected for other purposes (e.g., administrative data) are not burdensome to implement but may not be as reliable as data collected explicitly for quality measurement. Expanded use of clinical computer systems and electronic health records is one way to facilitate the collection of clinical data of high quality,28 but the implementation of new clinical computer systems is costly.

Use of publicly released data to inform consumer choice or to increase provider accountability also has its drawbacks. One significant problem is that publicly available data may include falsified results. To help maintain the accuracy of data reported to such organizations as the NCQA, auditing is required. Some falsification, however, may be undetectable by audits (e.g., a physician's falsely recording a blood pressure measurement that is below a quality goal). The potential exists for physicians to adjust their performance measures by the selection or dismissal of patients. For example, physicians may dismiss nonadherent or outlier patients (i.e., patients whose data lie far outside the central statistical mass) to improve their measured performance.30 Physicians who will be judged by clinical outcomes such as mortality following coronary artery bypass graft sur gery may avoid taking on patients at high risk for bad outcomes. In one instance, following the release of publicly reported cardiac surgery mortality, cardiologists reported that it became harder to find surgeons willing to operate on high-risk patients who needed surgery.31

Sample Size

Another potential problem in performance measurement is insufficient sample size. For any single measure, there may be too few patients for statistically meaningful comparisons, either between providers or between the performances of a single provider over time. An insufficient sample size is frequently a factor when individual physicians are profiled. When statistically meaningful differences cannot be detected because of limited sample size, those who report data should not suggest that providers with statistically insignificant differences in quality scores differ from one another. Combining several years of data is one way to compensate for insufficient sample size; however, this reporting method makes the reported results less timely and may obscure improvements in quality that occur in the short term.

Case-Mix Adjustment

The differences in patient characteristics (referred to as case mix) can introduce systematic biases into quality measurements; case-mix adjustment to correct these biases is an important methodological consideration in the utilization of performance data. Apparent differences in the quality of care received by patients cared for by different physicians may disappear when differences in patients' socioeconomic status or education are taken into account.16,17,18,19 Case-mix adjustment is especially important for measures of clinical outcome or cost efficiency, but it may alter the interpretation of data pertaining to process measures.

Statistical methods can be used to improve the validity of statistical inference when sample sizes are small or when the case mix differs across providers; however, it is not clear whether the methodology to correct for these variables will be generally adopted or, if adopted, applied in uniform ways.32

Optional Reporting

Public reporting of performance data may be of limited use to health care purchasers and consumers if poor performers can choose not to report their data. It has been noted that some health plans that participated in the NCQA's Health Plan Employer Data and Information Set (HEDIS) withdrew from HEDIS when they performed poorly.33

Inadequate Use of Data

Even when public reports of health care quality are available, purchasers and consumers may not put them to good use.34 Consumers often do not access the performance data that are available and may have difficulty making sense of the information when they find it.25,35,36

METHODS TO IMPROVE PERFORMANCE MEASURES

Although all these methodological issues and limitations are formidable, the organizations developing and promoting such measurement seek to identify and promote only those measures meeting clear criteria. An example of this move toward standardization is the recent development by the NQF of a set of voluntary consensus criteria by which to endorse physician-focused ambulatory care performance measures. The NQF's consensus standards include criteria to judge the importance, scientific acceptability, usability, and feasibility of proposed performance measures.37

Issues for the Physician

Although encumbered by methodological and practical issues, performance measurement is a reality of modern medical care. Experts caution physicians that they must be leaders in quality measurement and performance improvement or be vulnerable to challenge by economic or political stakeholders and to the potential loss of patient confidence.38 Because it is likely that we physicians will increasingly be held accountable for the quality of care we provide, it is to our advantage to have a working knowledge of current performance measurement activities in our specialty [see Sidebar Internet Resources for Performance Measures]. By understanding the types of measures by which we will be judged and knowing how the data are collected, we may be able to improve our performance—as well as the documentation of the data needed for proper measurement of that performance—without greatly increasing our work.

Internet Resources for Performance Measures

General Information

Agency for Healthcare Research and Quality (AHRQ)

http://www.ahrq.gov/clinic/epcix.htm

A governmental agency and information clearinghouse on issues of quality, safety, efficiency, and effectiveness of health care

Ambulatory Care Quality Alliance (AQA)

http://www.ahrq.gov/qual/aqaback.htm

Initially convened by the American Academy of Family Physicians, American College of Physicians, America's Health Insurance Plans, and AHRQ, the Alliance consists of a large body of stakeholders, including physicians, consumers, health care purchasers, and health plans

Centers for Medicare and Medicaid Services (CMS)

http://www.cms.hhs.gov/quality/hospital

Provides hospital quality information to consumers and others in initiatives designed to improve hospital care in the United States

National Committee for Quality Assurance (NCQA)

http://www.ncqa.org

An independent nonprofit organization that provides information on the quality of managed care plans in the United States

National Quality Forum (NQF)

http://www.qualityforum.org

A private, not-for-profit membership organization created to develop and implement a national strategy for measuring and reporting health care quality

Measures and Tools

National Quality Measures Clearinghouse

http://www.qualitymeasures.ahrq.gov

Public repository for evidence-based quality measures and measure sets

Measuring Healthcare Quality

http://www.ahrq.gov/qual/measurix.htm

Includes National Health Care Quality reports, AHRQ quality indicators, and ambulatory care clinical-performance measures

National Voluntary Consensus Standards for Ambulatory Care

http://www.qualityforum.org/txWEBambreport04-29-05.pdf

Appendix A contains physician standards for ambulatory care pertaining to asthma and respiratory conditions, heart disease, hypertension, prenatal care, and prevention

Quality Profiles

http://www.qualityprofiles.org/quality_profiles/index.asp

Case studies that examine performance measures in such topics as chronic illness, women's health, preventive care, and behavioral health

Quality Improvement Initiative Tools

http://www.qualityprofiles.org/qia_Tools/index.asp

Templates of the tools used by health plans to help implement their health care improvement initiatives

Physician Quality Reports

http://www.ncqa.org/PhysicianQualityReports.htm

Includes measures used to qualify physicians for the Diabetes Physician Recognition Program, the Heart/Stroke Physician Recognition Program, and Physician Practice Connections

Hospital Quality Measures

http://www.qualityindicators.ahrq.gov

Presents hospital quality measures, inpatient quality indicators, and prevention quality indicators

When possible, individual physicians should assess the quality of the care they provide and correct their own deficiencies. They should consider making systematic changes that facilitate quality measurement and internal quality improvement, such as making use of disease registries, flow sheets, or computer systems, to track the quality of chronic disease management and preventive care. It is unlikely that the costs of implementing these changes will be recovered in the short term; however, if pay-for-performance becomes widespread, physicians and health care organizations that have not prepared for it by routinely measuring their own quality of care will find themselves at a distinct disadvantage in comparison with their competitors who have.

References

  1. Epstein AM, Lee TH, Hamel MB: Paying physicians for high-quality care. N Engl J Med 350:406, 2004
  2. McGlynn EA, Asch SM, Adams J, et al: The quality of health care delivered to adults in the United States. N Engl J Med 348:2635, 2003
  3. Kiefe CI, Allison JJ, Williams OD, et al: Improving quality improvement using achievable benchmarks for physician feedback: a randomized controlled trial. JAMA 285:2871, 2001
  4. Crossing the Quality Chasm: A New Health System for the 21st Century. Committee on Quality Health Care in America. Institute of Medicine. National Academy Press, Washington, D.C., 2001
  5. Hayward RA, McMahon LF Jr, Bernard AM. Evaluating the care of general medicine inpatients: how good is implicit review? Ann Intern Med 118:550, 1993
  6. Landon BE, Normand SL, Blumenthal D, et al: Physician clinical performance assessment: prospects and barriers. JAMA 290:1183, 2003
  7. Brook RH, McGlynn EA, Cleary PD: Quality of health care. Part 2: Measuring quality of care. N Engl J Med 335:966, 1996
  8. Needleman J, Buerhaus P, Mattke S, et al: Nurse-staffing levels and the quality of care in hospitals. N Engl J Med 346:1715, 2002
  9. Pronovost PJ, Angus DC, Dorman T, et al: Physician staffing patterns and clinical outcomes in critically ill patients: a systematic review. JAMA 288:2151, 2002
  10. Garg PP, Frick KD, Diener-West M, et al: Effect of the ownership of dialysis facilities on patients' survival and referral for transplantation. N Engl J Med 341:1653, 1999
  11. Bates DW, Leape LL, Cullen DJ, et al: Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. JAMA 280:1311, 1998
  12. Computer Physician Order Entry Fact Sheet The Leapfrog Group for Patient Safety. Washington, D.C., 2004http://www.leapfroggroup.org/media/file/Leapfrog-Computer_Physician_Order_Entry_Fact_Sheet.pdf
  13. Follow-up after hospitalization for mental illness. State of Health Care Quality Report, 2003. National Committee for Quality Assurance. Washington, D.C., 2003 http://www.ncqa.org/sohc2003/follow_up_after_hospitalization.htm
  14. Nursing Home Compare: Medicare, the official U.S. government site for people with Medicare, 2005http://www.medicare.gov/NHCompare
  15. Hospital report cards: mortality and complication based outcomes, 2005. Hospital methodologies. Healthgrades, Golden, Coloradohttp://www.healthgrades.com
  16. Zaslavsky AM, Hochheimer JN, Schneider EC, et al: Impact of sociodemographic case mix on the HEDIS measures of health plan quality. Med Care 38:981, 2000
  17. Greenfield S, Kaplan SH, Kahn R, et al: Profiling care provided by different groups of physicians: effects of patient case-mix (bias) and physician-level clustering on quality assessment results. Ann Intern Med 136:111, 2002
  18. Fiscella K, Franks P: Influence of patient education on profiles of physician practices. Ann Intern Med 131:745, 1999
  19. Franks P, Fiscella K: Effect of patient socioeconomic status on physician profiles for prevention, disease management, and diagnostic testing costs. Med Care 40:717, 2002
  20. Zaslavsky AM, Beaulieu ND, Landon BE, et al: Dimensions of consumer-assessed quality of Medicare managed-care health plans. Med Care 38:162, 2000
  21. Crofton C, Lubalin JS, Darby C: Consumer Assessment of Health Plans Study (CAHPS). Foreword. Med Care 37(3suppl):MS1 9, 1999
  22. Provider Efficiency White Paper: Measuring Provider Efficiency. Version 1.0. Bridges to Excellence and the Leapfrog Group for Patient Safety. Washington, D.C., 2004 http://www.bridgestoexcellence.org/bte/white_paper_release.htm
  23. Thomson O'Brien MA, Oxman AD, Davis DA, et al: Audit and feedback: effects on professional practice and health care outcomes. Cochrane Database Syst Rev (2):CD000259, 2000
  24. Berwick DM, James B, Coye MJ: Connections between quality measurement and improvement. Med Care 41(1 suppl):I30, 2003
  25. Marshall MN, Shekelle PG, Leatherman S, et al: The public release of performance data: what do we expect to gain? A review of the evidence. JAMA 283:1866, 2000
  26. Solberg LI, Mosser G, McDonald S: The three faces of performance measurement: improvement, accountability, and research. Jt Comm J Qual Improv 23:135, 1997
  27. Pay-for-performance takes off in California. ACP Observer Online. January–February, 2005http://www.acponline.org/journals/news/jan05/pfp.htm
  28. Roland M: Linking physicians' pay to the quality of care: a major experiment in the United Kingdom. N Engl J Med 351:1448, 2004
  29. Self-evaluation of practice performance. Maintenance of Certification. American Board of Internal Medicine. Philadelphia, 2005http://www.abim.org/moc/sempbpi.shtm
  30. Hofer TP, Hayward RA, Greenfield S, et al: The unreliability of individual physician “report cards” for assessing the costs and quality of care of a chronic disease. JAMA 281:2098, 1999
  31. Schneider EC, Epstein AM: Influence of cardiac-surgery performance reports on referral practices and access to care: a survey of cardiovascular specialists. N Engl J Med 335:251, 1996
  32. Zaslavsky AM: Statistical issues in reporting quality data: small samples and casemix variation. Int J Qual Health Care 13:481, 2001
  33. McCormick D, Himmelstein DU, Woolhandler S, et al: Relationship between low quality-of-care scores and HMOs' subsequent public disclosure of quality-of-care scores. JAMA 288:1484, 2002
  34. Hibbard JH, Jewett JJ, Legnini MW, et al: Choosing a health plan: do large employers use the data? Health Aff (Millwood) 16:172, 1997
  35. Schneider EC, Epstein AM: Use of public performance reports: a survey of patients undergoing cardiac surgery. JAMA 279:1638, 1998
  36. Hibbard JH, Jewett JJ: Will quality report cards help consumers? Health Aff (Millwood) 16:218, 1997
  37. Pre-voting review for “National Voluntary Consensus Standards for Ambulatory Care: An Initial Physician-Focused Measure Set.” The National Quality Forum. Washington, D.C., 2005 http://www.qualityforum.org/txWEBambreport04-29-05.pdf
  38. Blumenthal D: Quality of care: what is it? Part 1. N Engl J Med 335:891, 1996

Editors: Dale, David C.; Federman, Daniel D.