Return to Top
THE CAPACITY OF STATE DATA TO SUPPORT HEALTH REFORM
Lauren Burnbauer, Ph.D.
Karyen Chu , M.P. P.
Revised July 1996
Jim Knickman, Beth Stevens, and Joel Cantor of the Robert Wood Johnson Foundation conceived the survey on which this article is based and supported its development. Ira Kaufman encouraged the survey designers to include factual items. Randy Desonia, Steve Long, Mark Epstein, and Larry Patton reviewed and commented on a draft of the instrument.
Many MPR staff contributed to this article. Joy Gianolio was responsible for survey development during the design phase. Richard Strouse provided advice on survey structure and design. Linda Mendenko was responsible for sample development. Poria DeFilippes provided programming support. Aleda Freeman developed the graphics. Harold Beebout provided comments on earlier drafts. Daryl Hall edited the manuscript. Ann Niles provided secretarial support.
This article reports on the results of a recent survey of all 50 states sponsored by the Robert Wood Johnson Foundation. A major objective of the survey was to identify the extent to which state policymakers see their data systems as being able to provide the kinds of information health reform is likely to require, such as insurance coverage, health care expenditures, and health plan performance. The results reported here show that systems are seriously limited in their ability to provide these kinds of data. These limitations appear to occur, at least in part, because health reform requires states to assume responsibilities that depart from their traditional focus. Our results suggest that states are aware of many limitations and are taking steps to improve their data. States perceive that the three most serious barriers to improvement are (1) funding; (2) comparability across data sets; and (3) the willingness of the private sector to submit necessary data. Federal support is likely to be critical in removing these barriers, but so too is the concerted effort of the many different state officials and the support of the private sector. For federal policymakers, an issue related toimproving state data systems will be to determine which data are needed across all states, and for these, which should be collected centrally under federal sponsorship, and which should be collected by each state or locality. Our results also imply that important data on baseline performance of the health system are lacking at the state level. Unless this gap is addressed rapidly, it will serve as a major constraint on the ability to monitor and evaluate both health reform efforts and other system changes with differential structures or effects across states.
States have led the nation in the effort to reform the health care system, and they are likely to have an active and leadership role in implementing any federal reform'. States already have responsibilities for key functions embedded in reform efforts (e.g., insurance regulation and licensure of health plans and providers). Furthermore, building on their responsibilities for Medicaid and public health, many states already have also initiated reform efforts. To continue their health reform initiatives and responsibilities, states must have certain data and analysis. For example, data on health expenditures are key to monitoring the ability to meet a cost- containment target and data on health plan performance are important to competitive strategies of system redesign. We assess in state databases can delay the enactment or implementation of health reform efforts. With the attention given to the limitations in the policy relevance of health data within states, it is clear that identifying how and where to enhance state-based data systems is a critical issue.
We address the issue of improving state data systems by presenting results from a Robert Wood Johnson Foundation-funded telephone survey of key senior analytical staff in all the states. Conducted by Mathematica Policy Research in winter 1994, the survey provides timely information on state's perceptions of the quality of their data. The survey was also intended to identify certain key data systems and activities established by states to address critical issues in reform. These results are part of a larger study on the extent to which states' health data can support state health policy; these results are reported elsewhere.
The results in this article are based on a telephone survey of key senior state officials in the legislative and executive branches of all 50 states and the District of Columbia. The interviews were conducted between January and March 1994 by executive interviewers in the Survey Center at Mathematica Policy Research.
The survey was designed to obtain information from eight to nine individuals in each state. The sample frame was structured such that we could identify and survey the most appropriate senior person responsible for health policy analysis and for providing advice on each of eight functions.
We devoted considerable attention to defining and locating appropriate respondents. While some lists of state officials exist (e.g., state Medicaid directors), they are spotty and not sufficiently targeted for this study. We therefore identified appropriate respondents through an interactive process using one or more key informants in each state. Definitions were designed to prompt the selection of the most senior person who is both policy advisor and analyst in each area.
This article is based on answers to questions on health reform asked of three of the eight types of survey respondents: the governor's aide, the legislative analysts, and the lead staffer in a healthreform entity. The last was defined as the most senior staff member (other than the chair) of any state health commission on reform or similar body who is most knowledgeable about health reform, involved in analysis, and helps advise the governor on reform. If there was no commission, we relied on the body charged by the governor to be the lead on this issue. The diversity in titles and organizations of those interviewed is likely to reflect the absence of state bureaucracies established to address health reform issues.
The sample consists of govenor's aides from all states except one (Idaho), 91 legislative analysts (at least one from all states but one (Hawaii), and 47 health reform entity staffers (excluding Arizona, Idaho, Kentucky, and New Jersey). All three kinds of respondents were asked a core set of questions on the importance of various kinds of data for health reform, their confidence in the available data, and how data were involved in state reform efforts, in approximately that order. The health reform staffer was also asked about barriers to enhancing data capabilities in addition to a supplemental set of questions designed to obtain factual information on data actually available in the state. These were elicited after the perceptual questions to avoid influencing the response of policymakers. The objective here was not to develop a detailed inventory of data available across states but to obtain details useful in interpreting or validating data on perceptions. In some cases, respondents misinterpreted questions or did not know about details requested in the supplemental questions. Unfortunately, we were able to conduct only limited follow-up.
Return to Top
Analysis Techniques and Assessment of Data Quality
The unit of analysis, for the most part, is the respondent. Because the number of respondents is generally proportional to the number of states, the information we obtained reflects perspectivesacross all states.
Our survey specifically aimed to focus on policy maker perceptions even though we recognize these are subjective. The idea was to discover how key advisors to policymakers view the availability and adequacy of data rather than to actually audit the data available. The survey was structured this way because our focus was policy utility, and the extent to which data are useful depends very much on policymakers' perceptions, knowledge, and confidence about what is available. In addition, data systems are sufficiently complex that we did not regard it feasible to realistically describe them using a survey approach. However, we also believed it was important to place policymakers' perspectives in some context in order to interpret and validate their perceptions. The survey therefore included requests for factual data collected from the health reform entity staffers. For these data there is one response per state so the unit of analysis is the state.
The analyses we have been able to conduct suggest that the patterns revealed by and general conclusions drawn from the survey are valid, though there may be some measurement error associated with particular items. This conclusion is supported by the face validity of the data patterns, which appear to be consistent with what is known (e.g., hospital data is more complete than physician data) or with what would be expected based on state responsibilities (e.g., states are more likely to know total health expenditures from state funds than from all sources in the state). Survey results are also congruent with other sources (e.g., in terms of the number of states with hospital discharge datasets). The reliability of the estimates should also be enhanced by the care with which respondents were selected, the diversity of respondents, and the high rate of response to the survey and to items. Where we were able to compare survey data on perspectives to factual reports on data available, the results suggest that any bias may be to overstate the availability of data. An exceptionmay be the factual reports on the availability of insurance data, since few health reform entity staffers were from the insurance department, and we were able to conduct only partial follow-up with such respondents.
For these and other reasons, this article emphasizes patterns across states rather than assessing performance of individual states. However, we do provide some information about which states report certain kinds of data activity so that others with an interest in this area may identify leading states to contact for guidance. This approach balances the need for respondent confidentiality, which encourages honest reporting, with the utility of state-specific information, which enhances cross-state cooperation.
Confidence in the Ability of State Health Data to Support
Reform
State policymakers were not very confident about the ability of their data to support state health reform (Figure 1). Almost half were not very confident (34 %) or not confident at all (11%). Only 10 % were very confident, and the rest (46%) were somewhat confident. The reasons given by the "somewhat confident" respondents were more likely to be negative than positive (42 % versus 21 %, with 37 % mixed), suggesting that almost all respondents have some concerns about the ability of their state data to support health reform. While statistics are not uniform across the three types of respondents, these differences are not significant (Chi Square test). The most common reasons for lack of confidence in the data (cited by half or more of the respondents) were problems related to availability (30%), such as lack of timeliness, incompleteness, inaccessibility or irrelevance; or to the system (20%), particularly fragmentation across components or datasets.
There was little variation in response to the same question about the ability of the state health data system to address President Clinton's health reform initiative just over half were not very (37%) or not at all confident (15 %), while most of the rest were somewhat confident (43 %). (Seven percent were very confident). The similarity between the two sets of responses may reflect both a lack of knowledge about the president's plan at the time of the survey (as several respondents noted) and the general level at which respondents make such assessments. It is interesting that health reform entity staffers, who are likely to be most knowledgeable about the president's plan , though also potentially most invested in state approaches, differed more than others in their responses to the two items. They were much more likely to be not very or at all confident about the ability of their data to support President Clinton's plan than their own state's efforts (62 % versus 40 %). In other words, they believe that their data can better support state reform efforts than federal efforts.
Return to Top
Importance of Various Kinds of Data
State government policy officials cannot easily tell the number and characteristics of those without coverage, how much money is spent in the state on health care each year and by whom, or the quality and how satisfied consumers are with the care in health plans. All of these are important sets of information upon which to base health reform because they relate to key objectives of access, cost, and quality of care. Data in each of the three areas relevant to health reform were covered in the survey: health insurance coverage, health expenditures, and health system or health plan performance. (Other parts of the survey, the results of which are reported elsewhere (Gold, 1994), were designed to analyze state data capacity more generally. Data on these areas, e.g., on health resources and utilization, public health programs, Medicaid, and health status data, can alsobe relevant to reform needs.)
Because the survey was designed before the recent round of legislation in 1994 was introduced, the design could not take advantage of specific legislative analysis. Instead, survey content was based on a general understanding of what health reform might require and some early assessments from the state experience. In the long run, this approach may be more useful because, while proposals may change, there are probably a core set of data generally related to policy issues that are associated with reform.
The survey results confirm that the areas covered in the survey are viewed by policymakers in a wide variety of states as important. Respondents perceived the data covered in the survey to be highly relevant to health reform. Data on health insurance coverage, health expenditures, and health system/health plan performance were perceived by most respondents as "very important" (87 %, 90%, and 73 % respectively).
Virtually all the rest perceived such data as "important", with only a negligible percentage (two percent or less) perceiving it as not very important.
Our approach to identifying the perceived quality of data was to ask if the data were available and if so, to ask whether the information was viewed as excellent, good, fair, or poor. However, it is possible that while perceptions may be similar, some respondents would indicate that data are unavailable, while others would assess it to be fair or poor. Given this possibility, we therefore converted the two items to a single five-point item (not available to excellent). Table I presents the distribution of data on this scale for all the questions about the three kinds of data. A more summarized version of these data are used to discuss results, namely, an exhibit made up of two panels, one showing the total percentage of respondents who assess data to be fair, poor, or notavailable, and the other showing the percentage assessing the data as excellent (good being the omitted category). This graphic arrangement highlights the areas most in need of improvement as well as those of excellence.
Return to Top
Quality of Data on Health Insurance Coverage
Most states relied on national surveys for data on insurance coverage of the population, but half to three-quarters perceived the available information to be no better than fair. As a result, state policymakers cannot readily assess the impact and distributional effects of proposed initiatives. Figure 2 shows how respondents perceived the quality of seven kinds of data on insurance coverage: five on the population and two on employers. Overall, few respondents (less than 10%) perceived any of the types of insurance data to be excellent, and over half perceived each type to be no better than fair. Aggregate data on the total percentage uninsured in the state was perceived as having the highest quality, with 47% perceiving it as excellent or good. No more than one-third perceived any of the other kinds of data to be excellent or good. However, most respondents perceived that some data were available on these topics, albeit of fair or poor quality (Table 1).
The factual reports we obtained from health reform entity staffers show that most states appear to rely on national data sources for information on insurance coverage even though only one source at best (the Current Population Survey) is designed for this purpose. It also appears that in many cases, the quality of state-collected data may be dubious. Twenty-nine of 46 responding states indicated that they collect population-based insurance data: 9 collect data from national surveys, 6 from state surveys, and 14 collect data from both.
The most commonly used national survey is the Current Populafion Survey (21 states). Thirteen said they use synthetic estimates from the National Medical Expenditure Survey (NMES) or state estimates from the National Health Interview Survey (NHIS). The NMES sample is not sufficient for deriving estimates for any state, while NHIS supports only estimates for the larger states, and even here, its sample design does not support state-wide estimation.
The intent of our survey was to identify the design and potential quality of state-sponsored surveys by requesting information on sample size, response rate, survey mode, and source of sample frame for each survey. Unfortunately, few respondents were knowledgeable about these parameters, which may apply to data collected by other entities. From the information provided, it appears that considerably fewer than the 20 states that indicated the use of state surveys actually do conduct such surveys, with the remainder relying on insurance filings or some other kind of information. It also appears that some state surveys are unlikely to be based on random samples, with respondents reporting 100% response rates or volunteer sample frames. A few states (e.g., the District of Columbia, Georgia) base insurance coverage information on items added to the CDC Behavioral Risk Survey or a similar state survey. Some others are able to rely on other data sources such as surveys developed by universities (e.g., North Carolina obtains information from a Duke University Study) or broader instruments (e.g. Nebraska's Social Indicators Survey). More states now have state-based survey information on insurance coverage because they were included in the Family Health Survey sponsored by the Robert Wood Johnson Foundation for 10 states in 1993-94. However, this is not designed to be an ongoing survey.
States are less likely to report collecting data specifically on employment-based coverage. While 18 indicated that employer data on insurance coverage were available, only 8 indicated in a follow-up item that they actually collected data on the number of employers who offer coverage. Four reported using a required report (the District of Columbia, Florida, Hawaii, Oregon), and six used a survey (the District of Columbia, Florida, Maine, North Dakota, Utah and West Virginia). Some states surveyed employers of all sizes, and others surveyed only small businesses. The most recent survey was typically fielded in 1992 or 1993. Some states that reported using surveys may be relying on the Robert Wood Johnson Foundation's employer survey in 10 states in 1994.
Respondents were more likely to report collecting payer or carrier data than population or employer-based coverage data. This is not surprising given states' historical responsibilities for regulating health insurance. Forty-two said they had such data, and 29 of 40 said they collected information on the contracts covered by each insurance carrier or Blue Cross/Blue Shield product, but only 12 of 38 indicated that information identified individuals and families covered by such insurers. State policymakers are therefore more likely to know the number of subscribers than covered lives. Typically, states reported obtaining insurance data from filings with the Insurance Comniission (28 of 29 states with information on contracts) but 10 (of 25 responding) said this was obtained from special surveys. These include Colorado, Florida, Hawaii, Illinois, Minnesota, North Carolina, Ohio, South Carolina, Texas, and Utah. Most surveys were conducted as recently as 1993 or later.
These results confirm that state data on insurance coverage is limited and suggest that existing data may be even more limited in quality than respondents perceive.
Return to Top
Quality of Data on Health Expenditures
State policymakers were relatively satisfied with data on state governnent spending on health and hospital expenditures, but the vast majority viewed data on total spending as fair at best, andvirtually none had good information on how much consumers spend out of pocket. It is hard to design or assess cost-contaiment efforts if data are weak in these key areas. Figure 3 shows how respondents perceived the quality of different types of health expenditure data. Perceptions differed sharply by kind of expenditure data, and the quality of data on health expenditures was perceived by state government to be much higher than the quality of data system-wide. Thirty-nine percent perceived data on health spending by state government to be excellent, and only 16% perceived it to be fair, poor, or not available (45% said good). This relatively good assessment is consistent with the fact that state budgets and accounting systems are available as a basis, albeit with shortcomings for determining state government health spending.
Almost two-thirds of respondents (64%) perceived the quality of data on total health care spending to be fair, poor, or not available. Data on total hospital expenditures were perceived to be much better: 31 % perceived such data as excellent, while only 28 % perceived it as fair, poor, or unavailable. In contrast, 75 % perceived data on total spending on physician services to be fair, poor, or unavailable. The reason for the disparity between hospital and physician data and for the inability to measure total health spending in the state is apparent from information on which providers are required to file financial reports. While 40 of 46 states require filings from acute-care hospitals, and 35 of 47 require filings from other hospitals, only 4 of 47 have physicians file, 14 of 45 have licensed clinics file, and 15 of 43 require ambulatory care facilities to file. Institutional providers like hospitals or nursing homes are much more likely than other providers to be required to file reports. For example, 36 of 45 states require nursing homes to file, compared with 7 of 40 that require pharmacies to file.
Data on total consumer out-of-pocket spending are least likely to be viewed as adequate: 92%of respondents perceived this information to be fair, poor, or unavailable (32% unavailable). Nationally, these data have also been among the most troublesome to obtain in constructing the national health accounts.
Overall, 21 states reported (through the health reform entity staffer) that they had developed an analysis of total health expenditures. None perceived these data to be excellent, while 9 perceived it as good, 9 as fair, and 3 as poor. In contrast, 32 said that an analysis of total state goverrunent health spending had been conducted, with 6 rating that data as excellent, 14 as good, and 5 each as fair or poor. In both cases, the analyses were reported to be relatively recent, typically conducted in 1993 or later, or in 1992.
Return to Top
Quality of Data on Health System or Health Plan
Performance
The data on health system or health plan performance was perceived by the three kinds of respondents to be the most problematic (Figure 4). While many health reforms call for giving consumers and purchasers information to drive decisions and competition, there currently is little information in states to provide. Virtually all respondents perceived the data on quality of care for plans or providers, consumer satisfaction with care, and consumer satisfaction with the insurer as fair, poor, or unavailable (37% or more reporting such data unavailable). Perceptions about the quality of data on the premium charges for insurance were only a little better: 67% perceived such data as fair, poor, or unavailable, and only 5% perceived the data as excellent. In contrast, data on the fiscal solvency of health plans was perceived to be better. Eighteen percent of states perceived such data to be excellent, and only about one-third (35%) found the information to be fair, poor, or unavailable (53% said good). As with other positive perceptions, this one is likely to reflect thestates' historical role in overseeing health plans. For example, 36 of 48 respondents indicated that health maintenance organizations are required to file financial reports with the state.
Only a handful of states reported efforts to collect information on consumer satisfaction with care and quality of care provided by health plans (9 and 6, respectively, of 48). The information we obtained suggests that even this overstates current data-related activity in these areas or the availability of information, with many efforts still under development or of limited scope. The reported efforts were more likely to involve a public than a private entity, though both were involved in collecting consumer satisfaction data. In virtually all cases, public disclosure of results was anticipated.
Consistent with other findings, states were much more likely to report collecting data on the performance of insurance plans. Half or more of the states collect data on rate increases, loss ratios, reserves, and benefits offered by plans. Requirements to file information in the area apply to Blue Cross/Blue Shield, commercial, and HMO plans as well as individual and group products, though the specific data required have the potential to vary across these categories.
Return to Top
Use of Data on Health Reforrn and Barriers to
Improvement
State respondents appeared to recognize that data are important to their reform efforts. Eighty-five percent said data availability influenced the debate on health reform in their state, and 65 % said it influenced the passage of health reform proposals.
When asked generally to describe how data influenced passage, respondents gave mixed replies: some answered generally about the value of data in identifying problems, increasing understanding, or enhancing legislative confidence or commented on the role of data in delaying decisions and werewary about its political use in an adversarial process. Others were more specific. Examples of positive ways data had been used to support policy development included: analyzing disease trends, infant mortality, and fatalities data to design preventive efforts; identifying the number of uninsured children to enact a targeted program for them; developing a program for uninsurables based on information on the number unable to obtain insurance due to pre-existing conditions; regulating physician self-referral in response to a study of joint ventures; and using data to avoid program cuts by illustrating their negative effects. Examples of difficulties created by the absence of data included the need to solve problems created by giving the legislature inaccurate data, delays in legislative consideration because of insufficient information on expenditures and quality issues, and having to amend an insurance reform bill originally targeted at those age 50 and younger because the only available data were for those 25 and younger. While the responses show data limitations, substantially more respondents cited positive use of data than negative effects caused by an absence of data. In addition, some noted that the absence of data itself became a focus for legislation, with appropriations or mandates issued to increase data collection. The tendency for positive examples may reflect the response set of those we interviewed and should not necessarily be interpreted to mean that data limitations have not impeded policy development. Possibly, they are so common that they cause little note.
Only 14% of respondents reported that data improvements were not included in any legislative proposals. While 30% said legislation had been defeated or was still pending, 55% said changes had been enacted. Information on legislated improvements in the health data system suggests that they are broad in scope and have the potential for a long time horizon. The two approaches to improvement, which are not mutually exclusive, are (1) to create a new structure, commission, orauthority with a mandate to collect data or (2) to legislate authority for specific but generally broad types of data collection, with most efforts focusing on quality of care, outcomes, cost, consumer satisfaction, and use of services.
Eighty-one percent of respondents indicated that they had taken steps not requiring legislation to improve or generate health data to support health reform. Efforts included analysis of access, cost, or quality issues; improvements in data systems; new or enhanced collaborative relationships between the public and private sectors for generating and sharing data; increasing resources to support improved systems or analysis; enhanced data collection; and creating task forces, commissions, or other structures to develop and analyze data. Such changes are consistent with the more general survey finding that 69% perceive that data systems have been improved over the past three years to better meet policy needs.
The survey was not sufficiently specific to identify the status of any efforts or the adequacy of support for these efforts. Most work in this area, particularly as it involves legislative efforts, appears to be at a reasonably early stage and relatively ambitious in its objectives. Hence, major short-term improvements may not be likely, though current work may set the basis for future change if it proves successful. Figure 5 shows the percentage of respondents who agreed that any given area was a barrier to improving the availability and quality of data for health reform. To identify the relative importance of different barriers, respondents were also asked to name the first and second most important. The responses indicate consensus on the three most important barriers: funding, comparability across datasets, and the willingness of providers and insurers to submit required data. These were named as the first or second most important barrier by 58 %, 42 % and 42 %, of respondents. In contrast, the next two most commonly cited barriers--software capabilities anddefinitions of core data elements--were identified by only 15% of respondents each.
There are major weaknesses in the ability of data to support health reform activity in states. They include critical weaknesses in the availability of data on insurance coverage, health expenditures, particularly outside the public sector, and health system and heath plan performance. Data are more available in areas for which states have traditionally been responsible, but this information appears to be a small part of the data needed for health reform. The results appear to show that state policymakers do value and use data when available, although this information is obviously not the sole input to policy. The findings also show that states have taken steps to improve their data, although most initiatives appear to be ambitious and will not be accomplished rapidly. In making these improvements, states perceive the three greatest barriers to be funding, comparability across datasets, and the willingness of providers and insurers to submit needed data. In a previous article, we discussed some strategies states may use in generating support for these kinds of changes.
From a policy perspective, the results raise a number of issues about how best to address the problems identified here. First is the issue of how much data really are required. It is clear that there are major gaps in existing state data. Whether it is realistic to assume data can be or need to be fully developed in all areas in all states and at substate levels is debatable. On the other hand, many current gaps seem fundamental to developing any plan for reform. Second is the issue of federal support to states in addressing funding constraints and uneven data capabilities. This support could prove critical to states in improving their systems, but unless public and private constituencies in thestate are active, the changes may neither be responsive to state needs nor feasible. Federal support can address funding barriers, but the combined public and private efforts within each state will be needed to create greater comparability across data sets and provider willingness to contribute to this. The development of federal programs in this area could prove to be a bone of contention between federal and state officials. The former want uniform data that could also support federal needs and may legitimately question the efficiency of a decentralized data collection strategy. The latter are likely to want flexibility to address needs unique to their environment and may be concerned that federal strategies may further fragment state systems and undermine state objectives. Third is the issue of progress in policymaking. The long time horizon for some data improvements means that reform will be delayed unless initiatives are designed to take into account current limitations and the improvement process. Phased-in approaches that use more sophisticated measures to refine systems over time as data improves may be better than assuming the availability of or waiting for an ideal data set. The fourth and final issue involves monitoring and evaluation. Our results imply that important data on baseline performance of the health system are lacking at the state level. Unless this gap is rapidly addressed, it will create a major constraint on the ability to monitor and evaluate health reform efforts with differential structures or effects across the states. At a time when states are assuming greater leadership in health policy as in social policy in general, this should be a critical concern since it limits the ability to monitor how the health system in each state is changing in ways that affect access, quality, and cost of care.
Return to Top
Return to INFOSHP Home Page