The Trouble With Ranking National Health-Care Systems

By Carl Bialik

My print column this week examines a 2000 World Health Organization ranking of national health-care systems that continues to be cited in today’s U.S. health-care debate. I argue that the ranking is outdated and flawed.

I’m not the first to criticize the ranking. The editor-in-chief of the report, who said he wasn’t fully aware of the ranking methodology until after it was published, denounced the rankings in a journal article in 2003, drawing a rebuttal from the creators of the rankings. Glen Whitman, an associate professor of economics at California State University at Northridge, also criticized the rankings in a paper for the conservative Cato Institute.

I asked Whitman how he would have constructed a ranking. He, like many researchers, said he wouldn’t have. “I probably would not have tried, because I think the project is inherently problematic,” Whitman said. “The problem is that the quality of health care has lots of different dimensions: success in treating condition A, success in treating condition B, etc.; speed; convenience; cost; privacy; and so on. People with different values will place different weights on these factors. So any attempt to create a universal index of quality necessarily relies on value-laden assumptions.”

Another criticism noted that the rankings didn’t control for many variables it could have, such as geography (tropical climate, and distance from coast, can affect health). Controlling for such variables materially affected many country’s rankings — but not the U.S.’s. And that points out an important point that many of the WHO ranking’s critics acknowledge: Though the rankings may have been flawed, a ranking of 37th for the U.S. seems reasonable, and the U.S. may have slipped since 2000. For instance, data from the United Nations Population Division show the U.S. falling further behind its peers in life expectancy and preventing mortality of children under age five.

“When it’s good, it’s very, very good, and when it’s bad, it’s horrid,” Mark Pearson, head of health for the Organisation for Economic Co-operation and Development, the 30-member, Paris-based organization of major economies, said of the U.S. health-care system.
Some researchers have attempted to get a firmer handle on the U.S.’s proper health-care ranking by attempting to remove the effects of factors outside the health-care system’s control, such as violent crime. One such effort, which I recounted here two years ago, found that removing such factors greatly improved the U.S.’s standing in life expectancy. However, critics assailed the report for its indirect approach to the problem. One of the authors of the report, John E. Schneider, general manager of U.S. Health Economics for international health-care consultancy Oxford Outcomes, said that its intent was to show that small differences in life expectancy at birth, such as two years, “are too small to be meaningful due to differences in measurement and populations across countries.”

Researchers at the London School of Hygiene and Tropical Medicine have been tracking trends in deaths from causes deemed treatable or preventable by the health-care system, such as bacterial infections, treatable cancers and diabetes. The U.S.’s performance has slipped, falling to last among 19 wealthy nations. “Diabetes is highly correlated with obesity, and it’s not clear how ‘amenable’ obesity is to the health system,” Schneider said, in response to that study. “Can the health system make you get of your [posterior] and turn off the TV, or make you order one Big Mac instead of two? Probably not. … It’s the right general direction to go, but our old point is still relevant because this article demonstrates how methodologically challenging these kinds of analyses can be.”

The debate depends partly on how widely the health-care system is defined. “The health-care system ought to take responsibility,” for disease rates, says Ellen Nolte, director of health and healthcare for RAND Europe and co-author of the mortality-rate study. “It really is a case of integrating prevention and treatment rather than leaving the former to other sectors and only the latter to health care.”

An alternate analysis, focusing on prostate cancer and breast cancer, found signs of improvement in treatment and death prevention for those diseases in the U.S. and concluded that the country’s poor life-expectancy ranking “is not likely to be a result of a poorly functioning health-care system.” Samuel H. Preston, a sociologist at the University of Pennsylvania, pointed out that Nolte’s study focused on mortality by age 75, whereas the U.S. is strongest at later ages. Nolte counters that the Preston study examines only a select set of diseases, not the broader spectrum of her study.

Another problem with comparing countries by disease outcomes is that definitions and measurement “can vary within a country and across countries, even when the true prevalence is identical,” according to Alan Garber, an economist and professor of medicine at Stanford University. “There have been many efforts to standardize disease reporting across nations but it’s a tough challenge.”

Still other studies, though, also rank the U.S. low compared to its peers. “It still looks as if the U.S. is lagging behind,” said Jon Skinner, an economist at Dartmouth.

Further reading: The New York Times recently analyzed one of the newer comparative health-care studies. Here’s the WHO’s press release about its initial report, and a recent Politifact post evaluating the U.S.’s supposed No. 37 ranking.