Citizen-Driven Government Performance

Bibliography of Performance Measurement

Brief Manual for Performance Measurement in Local Government

Performance Measurement Manuals

Citizen Surveys

Cases - Teaching Notes

The following can be downloaded as an Adobe® pdf. file - (Download here)

A BRIEF GUIDE FOR PERFORMANCE MEASUREMENT IN LOCAL GOVERNMENT

This brief manual is designed to assist local government managers, elected officials and citizens in developing performance measurement systems. The manual explains:

• The uses and values of performance measurement systems;

• How such a system operates; and

• A simple step-by-step process for developing a performance measurement system.

Performance Measurement: An Increasingly Popular Management Tool

Who has not heard complaints about difficulties measuring government's performance, about diverse and contradictory objectives, unreliable measurement tools, and lack of resources governments are willing to invest in new management techniques? While difficulties undoubtedly exist, the vast amount of literature suggests that Performance Measurement is an advanced management tool that is becoming more and more sophisticated in order to accommodate needs of different communities and levels of government over services ranging from public safety and public works to economic development. Among numerous programs nationwide that have focused on what programs actually deliver instead of what resources they spend, many have been recognized by national award programs such as the National Center for Public Performance's Exemplary State and Local Awards Program and the Kennedy School/Ford Foundation Innovation Program. Two such efforts have been featured in The Gore Report on Reinventing Government

Measuring Outcomes

Outcome-based management is not new in the public sector. Some U.S. cities have developed it over the past two decades; some states are beginning to; and foreign countries such as Great Britain, Australia, and New Zealand are on their way.

Sunnyvale, California, a city of 120,000 in the heart of the Silicon Valley, began the experiment 20 years ago. In each policy area, the city defines sets of "goals," "community condition indicators," "objectives," and "performance indicators." "In a normal political process, most decision makers never spend much time talking about the results they want from the money they spend," says City Manager Tom Lewcock. "With this system, for the first time they understand what the money is actually buying, and they can say yes or no."

Sunnyvale measures performance to reward successful managers. If a program exceeds its objectives for quality and productivity, its manager can receive a bonus of up to 10 percent. This generates pressure for ever-higher productivity. The result: average annual productivity increases of four percent. From 1985 to 1990, the city's average cost of service dropped 20 percent in inflation-adjusted dollars. According to a 1990 comparison, Sunnyvale used 35 to 45 percent fewer people to deliver more services than other cities of similar size and type.

At least a half-dozen states hope to follow in Sunnyvale's footsteps. Oregon has gone the farthest. In the late 1980s, Governor Neil Goldschmidt developed long term goals, with significant citizen input. He set up the Oregon Progress Board, comprising public and private leaders, to manage the process. The Board developed goals and benchmarks through 12 statewide meetings and written materials from over 200 groups and organizations. "Oregon," the Board stated, "will have the best chance of achieving an attractive future if Oregonians agree clearly on where we want to go and then join together to accomplish those goals."

The legislature approved the Board's recommended 160 benchmarks, measuring how Oregon is faring on three general goals: exceptional individuals; outstanding quality of life; and a diverse, robust economy. Seventeen measures are deemed short-term "lead" benchmarks, related to urgent problems on which the board seeks progress within 5 years. They include reducing the teen pregnancy rates, enrolling people in vocational programs, expanding access to basic health care, and cutting worker compensation costs.

Another 13 benchmarks are listed as "key"— fundamental, enduring measures of Oregon's vitality and health. These include improving basic student skills, reducing the crime rate, and raising Oregon's per capita income as a percentage of the U.S. average.

Barbara Roberts, today's Governor, has translated the broad goals and benchmarks into specific objectives for each agency. This year, for the first time, objectives were integrated into the budget giving Oregon the first performance-based budget among the states.

Government agencies, professional associations, think tanks such as the Urban Institute and researchers in universities have suggested many standards of performance measurement. The Government Accounting Standards Board, for example, has published a series of volumes suggesting standards for Service Efforts and Accomplishments (SEA).

In an era when "do more with less" has become a common adage directed at all levels of government, performance measurement is becoming an essential tool for addressing questions of productivity improvement in terms of efficiency, effectiveness, and accountability.

What is Performance Measurement and How Can it Be Used?

There is no universally accepted term for measuring an organization's performance. As a result, many terms such as productivity, work measurement, and effectiveness have been used synonymously with "performance measurement." As Paul D. Epstein suggests, the simplest way of thinking about it is the following: Performance Measurement is government's way of determining whether it is providing a quality product at a reasonable cost. Following the definition used by the Government Accounting Standards Board (GASB), Performance Measurement is often referred to as Service Efforts and Accomplishments (SEA) Reporting. In this manual, SEA reporting and Performance Measurement will be used interchangeably. Based on the Civil Service Reform Act of 1978, performance measurement was defined in 1980 by GAO as an assessment of an organization's performance, including measures of:

• Productivity, which quantifies the outputs and inputs of an organization and expresses the two as a ratio. Generally, the ratio is expressed as output to input (for example, inspections per staff-day).

• Effectiveness, which determines the relationship of an organization's outputs to what an organization is intended to accomplish.

• Quality, which examines an output or the process by which an output is produced. Quality is indicated by attributes such as accuracy (or error rate), thoroughness, and complexity.

• Timeliness, which evaluates the time involved producing an appropriate output.

Different performance measures should not be seen as divergent and contradictory, but supplementing each other, As Marc Holzer argues, multiple measures of performance are useful diagnostic tools. As a group, as pieces of a puzzle, multi-dimensional measures of specific services can help provide objective perspectives for defending or expanding a program, rather than allowing it to suffer from relatively arbitrary or habitual decisions. Ongoing monitoring systems, which emphasize indicators and analysis linked to improvement, can help track and improve results over time. Performance measurement can help identify promising areas, helping to select as targets those functions continually faced with large backlogs, slipping deadlines, high turnover, or many complaints. Particularly important in evaluating or tracking or assessing evidence of progress, for example, are timely data which reflect cost savings, additional services, independent evaluations of service levels, client satisfaction, and reductions in waiting or processing times. Measurement must apply to the entire organization-as-system. It is important to examine program effectiveness or outcomes, rather than just quantity or efficacy.

There are many uses for performance measures. To run their organizations successfully, public managers must have certain key items of information available about their agencies. This applies to all levels of management and to all kinds of organizations. It is essential that performance measurement be considered an inherent and indispensable part of the management process. Usually, performance management is described as contributing to the following:

• Better decision-making: it provides managers with information to perform their management control functions;

• Performance appraisal: it links both individual and organizational performance to aspects of personnel management and motivates public employees;

• Accountability: it fosters responsibility on the part of managers;

• Service delivery: Improvements in public service performance;

• Public participation: clear reporting of performance measures can stimulate the public to take a greater interest in and provide more encouragement for government employees to provide quality services; and

• Improvement of civic discourse: it helps to make public deliberations about service delivery more factual and specific.

Types of Performance Measurement Indicators

There are several types of performance indicators that are often used in performance measurement systems. The most important types of measures are reported below.

Input Indicators. Government Accounting Standards Board (GASB) defines them as indicators that are designed "to report the amount of resources, either financial or other (especially personnel), that have been used for a specific service or program. Input indicators are ordinarily presented in budget submissions and sometimes external management reports."

Output/Workload Indicators. These indicators report units produced or services provided by a program. Workload measures indicate the amount of work performed or the amount of services received.

Outcome/Effectiveness Indicators. These measures are designed to report the results (including quality) of the service. According to Paul D. Epstein, "effectiveness measurement is a method for examining how well a government is meeting the public purpose it is intended to fulfill. In other words, effectiveness refers to the degree to which services are responsive to the needs and desires of a community. It encompasses both quantity and quality aspects of a service." Examples of outcome indicators are the change in students' test scores, and the value of property lost due to crime.

Efficiency (and Cost-Effectiveness Indicators). As Epstein defines them, efficiency measurement is a method for examining how well a government is performing the things it is doing without regard to whether those are the right things for the government to do. Specifically, efficiency refers to the ratio of the quantity of the service provided (e.g., tons of refuse collected) to the cost, in dollars or labor, required to produce the service. According to GASB, these indicators are defined as indicators that measure the cost (whether in dollars or employee hours) per unit of output or outcome. Examples are cost per million gallons of drinking water delivered to consumers, or cost per thousand gallons of effluent treated to a certain level of quality.

Productivity indicators. David N. Ammons defines productivity indicators as combining the dimensions of efficiency and effectiveness in a single indicator. For instance, whereas "meters repaired per labor hour" reflects efficiency, and "percentage of meters repaired properly" (e.g., not returned for further repair within 6 months) reflects effectiveness, "unit costs (or labor-hours) per effective meter repair" reflects productivity. The costs (or labor-hours) of faulty meter repairs as well as the costs of effective repairs are included in the numerator of such a calculation, but only good repairs are counted in the denominator--thereby encouraging efficiency and effectiveness of and by meter repair personnel.

In many cases, along with the above-mentioned indicators, some additional information is needed to make a sound judgment about service provision. GASB, for example, specifies certain types of Explanatory Information for its suggested list of indicators for service efforts and accomplishments. GASB defines a variety of information about the environment and other factors that may affect an organization's performance on Service Efforts and Accomplishments indicators, for example weather conditions for road maintenance.

Table 1. Types of Performance Measurement Indicators: An Example

Criteria for a Good Set of Performance Measures

Indicators in a properly developed set of performance measures should satisfy the following criteria mentioned by David N. Ammons:

• Valid

• Reliable

• Understandable

• Timely

• Resistant to perverse behavior

• Comprehensive

• Nonredundant

• Sensitive to data collection cost

• Focused on controllable facets of performance

Performance Measurement Systems: Issues To Be Addressed

A well-designed Performance Measurement System should clearly articulate the service goals and objectives, and then demonstrate service outputs are, the expected quality levels for these outputs, and what productivity is expected from expended resources (usually measured as man-hours and dollars). It should also enable the manager to judge whether continuous improvement is being made in terms of efficiency and effectiveness, and whether or not reported improvement in either efficiency or effectiveness has been made at the sacrifice of one or the other.

According to GASB, when designing a Performance Measurement System, the following issues should be addressed:

1) What SEA indicators should be candidates for reporting? To what extent are these measurable, valid and comprehensive?

Ideally, outcomes, outputs, inputs, efficiency, effectiveness, and productivity indicators should be reported; of course, these are subject to costs, measurability, validity, etc.

2) How and to what extent should SEA indicators be disaggregated?

There is no standard way for disaggregation of performance measures. Most commonly performance measures are broken down by geographical areas (e.g., streets, neighborhoods, wards, etc.); organizational units (e.g., different schools within a school district, different facilities within a sanitation department, etc.); and by degree of difficulty of the incoming workload (e.g., mortality rates for different age and risk groups). Other levels of disaggregation (e.g., by the size of a jurisdiction, type of a service) are also possible.

3) What comparison information should be reported for the various indicators?

Comparisons can be made between current information and:

• Previous year’s performance;

• Similar jurisdictions;

• Technically developed standards or norms;

• Actual SEA with targets or goals set by the agency at the beginning of the year;

• Geographical areas of client groups within the same jurisdiction; and

• Public-sector/private-sector costs and results in similar organizations.

4) What explanatory data should be included along with SEA data, and how should these data be presented?

Explanatory data for Performance Measurement will vary from service to service, but most likely will include some physical and climatic characteristics (rain, snow, type of soil and water, terrain), as well as organizational/jurisdictional information (size of the municipality, number of people served, etc.). This information may be of great importance, especially for benchmarking purposes (e.g., when comparing to other jurisdictions). Except for those factors, Performance Measurement and other improvement approaches develop differently in different communities depending on:

• Conditions and problems in the community;

• Interest of its elected officials and managers;

• Abilities of the staff;

• Resources available for improvements;

• Strength of public employee unions;

• Cultural and social factors;

5) To what extent are the SEA indicators verifiable?

Performance measurement indicators can be verified by correlating them to other, independent measures (e.g., standardized tests for the schools), or the procedures of obtaining data can be carefully and critically examined (e.g., reviewing telephone survey procedures). Also, systematic reporting of the measures during an extended period of time will contribute to their reliability.

6) How should the information be communicated and displayed, and in what types of documents should the performance data be reported?

The data should be presented and communicated clearly and precisely. Both management and policy participants in the performance improvement process can utilize service and financial measures more powerfully if they are displayed clearly, i.e. graphically. Tabular compilations of data are useful, primarily for professionals who use that data daily. But if data are confined to tables then professionals have the advantage. Other participants in the public policy process--elected executives, legislators, board members, business advisory groups, policy-level administrators, union members, interest groups, the general public and the media--are not always as adept at drawing conclusions from detailed tables as are the professionals who construct them. Most people, however, can draw conclusions when the data is displayed visually, e.g. by the use of bar graphs and trend charts. Personal computers and widely available software make the conversion of tabular data to geographic information maps a relatively simple task. By relating information about service to information about demography and geography, graphics facilitate the identification and analyses of present and projected problems/services.

7) What are the costs and feasibility of obtaining and reporting performance information?

Collection of performance measurements, especially effectiveness and outcome indicators, can be costly (e.g. if a large surveys of citizen satisfaction are carried out), especially if they are unsystematic, ad-hoc procedures. Verification of data can be costly as well. It is important to remember both of these costs when designing a Performance Measurement System. If planned appropriately, the costs can be lowered significantly (e.g., by building in-house capability, or getting discounts for long-term large-scale polling service).

8) What are the uses for and who are the users of SEA data?

As mentioned above, performance measurement (or service efforts and accomplishments) reporting can be used for the following purposes:

• Decision-making;

• Performance appraisal;

• Accountability;

• Improvement in public service performance;

• Public participation;

• Improvement of civic discourse.

9) Do Performance Measurement Systems fail?

A Performance Measurement System sometimes may fail. If it happens, one or more of these causes can usually be identified:

• Excessive costs of data collection;

• Lack of funds and long-term support for Performance Measurement, especially from top management and/or public officials;

• Absence of a "quarterback" to keep the system running;

• Lack of training;

• Not using data generated by the system in actual operations;

• Overemphasis on output or workload data as opposed to performance indicators.

Designing and Implementing a Performance Measurement System

Designing of a Performance Measurement system is neither complex nor difficult. However, it may be challenging for those not accustomed to measuring and setting performance targets. Developing a system involves an understanding of what the program is trying to accomplish, who the main users/ customers are, and a basic knowledge of the level of service currently being provided by the program. The specific steps in the process are listed below.

Step 1. Identification of a program

Step 2. Statement of purpose

Step 3. Identification of program inputs, outputs, efficiency and productivity indicators

Step 4. Setting targets for accomplishment

Step 5. Monitoring

Step 6. Performance reporting

Step 7. Analysis and action

Step 1. Identification of the Programs to Measure

To start with, various activities that local governments carry out should be clearly identified and divided into distinct programs. Programs are groupings of routine activities aimed at providing support for a certain public service. Groupings of individual activities make up a program. For example, the following four activities--street resurfacing; street patching; seal coating; and curb repairing--are activities that constitute the program that is usually called street maintenance. Other popular local government programs are animal control; clerk's office; code enforcement; corrections; courts; fire prevention; fire suppression; housing code enforcement; landfill operations; library; parks maintenance; police patrol; recreation services; senior services; solid waste collection; street cleaning; utility billing; wastewater collection; water distribution; etc. Usually, programs are defined by governments and are generally listed on an organizational chart contained in the operating budget. Programs relate directly with the organizational structure and the managerial areas of responsibility. Programs also correspond with the expenditure classification structure, that is, with the formally recognized cost centers.

Choosing what programs to measure is a matter of judgment. On the one hand, programs should not be too few, so that only a tiny portion of services is covered or the information collected is insufficient. On the other hand, too much reporting can be excessively costly, overwhelming and impractical because it is not followed through. Generally, Performance Measurement Systems work best when they concentrate on collecting limited, but essential information about basic programs that need the most managerial oversight and where accountability reporting is most important.

Step 2. Statement of Purpose

Preparing a well-articulated statement of purpose for a program is a very important step. One can measure the performance of a program only if he or she knows what the program should accomplish. Ideally, a clear mission statement is where one should start. If that is not available, then a thorough program description is a good start. As a rule, for governmental entities, most programs are not self-serving but provide for some public service to the citizens of a certain jurisdiction. A statement of purpose for the program under measurement should reflect that, and if it lacks a clear description or purpose, the program should be critically examined. For example, GASB offers the following objective for public transportation services:

The basic purpose is to provide safe, dependable, convenient, and comfortable transportation services at minimum cost to the citizens, including special client groups such as the handicapped and elderly.

Step 3. Identification of Program Inputs, Outputs, Efficiency and Productivity Indicators

Usually, program inputs are expressed through expended money in an operating budget and number of man-hours or full-time equivalent employees. Outputs usually represent workload measures, or the quantity of the delivered service to the users. Most experts advise including the unit of output --the quantity of each service (output/activity) being produced by the program-- into the statement as well. Efficiency and productivity measures address the question of: How well is the service provided? Efficiency measures the cost (whether in dollars or employee hours) per unit of output or outcome, while productivity indicators combine the dimensions of efficiency and effectiveness in a single indicator (see Types of Measures). For example, in a Sanitation Department, input indicators will be the amount of labor-hours; the budget of the Department; number of vehicles operated, etc. Output measures will include tons of refuse collected, miles of roads cleaned, number of customers served, etc. Efficiency indicators will include labor-hours per ton of refuse collected, dollars spent for one mile of snow removal, etc.; and productivity indicators will include measures such as cost per mile of a clean street (i.e., total cost of all road cleaning divided by the total miles of clean streets), etc.

Step 4. Setting Targets for Accomplishment

In this step we should specify under what conditions we can claim that the goals of the program have been met. Basically, we should identify service effectiveness and quality, and explicitly state how we are going to check whether our objectives have been met. Usually, it is done by expressing the goals of a program through date (or period of time) and quantities (including percentages). For example, for a Street Cleaning program of the Sanitation Department, this may mean achieving by the end of the fiscal year 75% cleanliness rating in citizen surveys, and 80% cleanliness rating by trained observers. Targets for accomplishment are not necessarily confined to effectiveness indicators. Organizations can similarly set efficiency and productivity targets, or input/output targets. For example, in the case of resource scarcity, input targets may be very important.

Step 5. Monitoring

Each accomplishment target should be monitored on a continuous basis. Monitoring provides the user with the results needed to decide whether or not the target is accomplished or not. Systematic and periodic monitoring gives the manager an opportunity to keep tabs on the operation of the program, and take corrective action if requested during the program. For instance, if the accomplishment target calls for 95% of the park visitors to be satisfied with the state of cleanliness and general maintenance by the end of the year, it does not mean that results are going to be checked once every year. Lesser-scale customer surveys can be completed on a quarterly or monthly basis. They will help to see whether everything is going as planned, and check whether there is any seasonal (cyclical) pattern in customer satisfaction. Gathering of information should be reasonable and should not overtax an organization's fiscal and human capacities. Usually, monitoring will vary depending on the service and target of accomplishment. For most important services (programs), monthly data collection and reporting system should reflect all identified program outputs, inputs, effectiveness and efficiency (productivity) measures.

Step 6. Performance Reporting

Performance reporting summarizes all the indicators and compares actual results to the targets identified earlier. Besides comparisons with targets, reports may include comparisons with: 1) the previous period (and include a year-to-date roll-up); 2) similar jurisdictions; 3) technically developed standards or norms; 4) geographical areas of client groups within the same jurisdiction; and 5) public-sector/private-sector costs and results with similar organizations. Performance reporting formats will vary depending on the circumstances. Ideally, it should be an accessible and convenient format including the following: 1) program name, organizational jurisdiction, and statement of purpose; 2) workloads/outputs, inputs, productivity and/or efficiency ratios; 3) targets for accomplishment for crucial indicators, most notably, effectiveness and efficiency; 4) brief synopses of additional explanatory information. Information should be organized in such a way as to facilitate comparisons. Reporting and monitoring formats can coincide quite frequently.

Step 7. Analysis and Action

While many may consider this step not being part of a Performance Measurement System, analysis and action are its logical conclusion. A well-developed Performance Measurement System will enable its users to spot weaknesses and threats, as well as strengths and opportunities. Thus, better knowledge of strengths and weaknesses will give the manager (as well as other users) an opportunity to diagnose organizational growth capabilities and take relevant actions.

Conclusion

This guidebook aims to introduce municipal managers, elected officials and citizens to basic tenets of Performance Measurement. Once in place, Performance Measurement Systems can be used for the appraisal of managers and employees, will help to elaborate and execute strategic plans, and will facilitate citizens' active engagement in visioning the future of their community. Every great endeavor begins with a first step, and very often building a solid Performance Measurement System is the crucial first step that should be taken in order to build a prospering community. Development of a sound Performance Measurement System will take time and dedication, but the results are well worth the effort.