 |
  |
| The Office of Motor Carriers and Highway Safety: An Analysis-Driven Organization |
International Highway Transportation Safety Week:
June 1-6, 1998
About This Issue
Since 1996, analysts from the John A. Volpe National Transportation Systems Center in Cambridge, MA,
have teamed up with the Office of Motor Carriers and Highway Safety (OMCHS) Analysis Division staff
to assess the efficiency and effectiveness of key OMCHS safety programs and identify possible actions
to correct problems. This issue of MCSAFE reports findings from a recent analytical study:
An Effectiveness Analysis of OMCHS's SafeStat (Motor Carrier Safety Status Measurement
System).
(Go to Article)
First, though, this issue examines the results of the 1998 International Highway Transportation Safety
Week (June 1-6).
(Go to Article)
Highway Safety: A Shared Responsibility
International Highway Transportation Safety (IHTS) Week, sponsored jointly by the Federal Highway Administration (FHWA) and the Commercial Vehicle Safety Alliance (CVSA), was held June 1-6, 1998. This cooperative effort involved State agencies and other organizations interested in promoting truck and bus highway safety and commemorating the people, programs, and partnerships working to improve highway safety every day. All 50 States, the District of Columbia, Guam, American Samoa, Northern Marianas, and Ontario participated in 1998 IHTS Week, along with Transport Canada and commercial motor carriers and motor coach operators.
1998 IHTS Week activities included:
- truck and bus roadside safety inspections;
- safe driving educational events at rest areas and malls; and
- safety articles, interviews, and public service announcements to the media.
The data below summarize inspection activities during 1998 IHTS Week.
Summary of IHTS Week 1998 Inspection Activities
|
Table 1 Level 1 Truck Inspection Results: United States and Canada | |||||
| Nation | Truck Inspections | Truck OOS | % Truck OOS | Driver OOS | % Driver OOS |
|---|---|---|---|---|---|
| United States | 33,735 | 9,629 | 28.5% | 2,069 | 6.1% |
| Canada | 476 | 196 | 41.2% | 0 | 0% |
|
Table 2 All United States Inspections (Truck and Bus) | |||||
| Inspection Type | Total Inspections | Vehicle OOS | % Vehicle OOS | Driver OOS | % Driver OOS |
|---|---|---|---|---|---|
| Level 1 | 33,735 | 9,629 | 28.5% | 2,069 | 6.1% |
| Level 2 | 14,457 | 2,817 | 19.5% | 1,216 | 8.4% |
| Level 3 | 8,605 | 0 | 0 | 1,036 | 12.0% |
| Level 4 | 2,378 | 563 | 23.7% | 121 | 5.1% |
| Level 5 | 380 | 50 | 13.2% | 0 | 0 |
| Totals | 59,555 | 13,059 | 21.9% | 4,442 | 7.5% |
|
Table 3 Hazardous Materials Inspections | |||||
| Inspection Type | Total Inspections | Vehicle OOS | % Vehicle OOS | Driver OOS | % Driver OOS |
|---|---|---|---|---|---|
| Level 1 | 3,855 | 956 | 24.8% | 176 | 4.6% |
| Level 2 | 1,031 | 175 | 17.0% | 51 | 4.9% |
| Level 3, 4, 5 | 648 | 81 | 12.5% | 24 | 3.7% |
| Totals | 5,534 | 1,212 | 21.9% | 251 | 4.5% |
|
Table 4 Most Frequently Cited Inspection Violations |
||
| Violation Number | Description | Violations |
|---|---|---|
| 396.3A1BA | Brakes Out of Adjustment | 9,932 |
| 396.3A1 | Inspection, Repair and Maintenance of Parts and Accessories | 9,188 |
| 392.2 | Local Laws (General) | 8,589 |
| 393.9 | Inoperable Lamp | 6,299 |
| 396.3A1B | Brakes (General) | 5,517 |
| 395.8F1 | Driver's Record of Duty Status Not Current | 4,884 |
| 393.75C | Inadequate Tread Groove Pattern Depth | 4,294 |
| 393.11 | No or Defective Lighting Devices/Reflectors/Projected Loads | 4,210 |
| 393.45A4 | Damaged Brake Tubing and Hoses | 4,204 |
| 393.95A | No or Discharged Fire Extinguisher | 3,873 |
| 393.19 | No or Defective Turn or Hazard Lamp as Required | 3,386 |
| 396.17C | No Proof of Periodic Inspection on Vehicle | 3,385 |
| 395.8 | Driver's Record of Duty Status Violation | 3,219 |
| 393.25F | Stop Lamp Violation | 3,007 |
|
Table 5 States, Provinces, and Territories: Level 1 Truck Out-of-Service Results |
|||
| State/Province/Territory | Total Inspections | Truck OOS % | Driver OOS % |
|---|---|---|---|
| Alabama | 177 | 23.2% | 7.9% |
| Alaska | 198 | 30.8% | 1.0% |
| American Samoa | 94 | 46.8% | 3.2% |
| Arizona | 267 | 34.8% | 5.2% |
| Arkansas | 615 | 16.9% | 5.4% |
| California | 7,204 | 24.4% | 3.4% |
| Colorado | 639 | 23.5% | 5.3% |
| Connecticut | 367 | 44.4% | 18.8% |
| Delaware | 1 | 100.0% | 0.0% |
| DC | 2 | 50.0% | 0.0% |
| Florida | 323 | 26.9% | 9.0% |
| Georgia | 127 | 54.3% | 15.7% |
| Guam | 149 | 24.8% | 0.0% |
| Hawaii | 223 | 18.8% | 1.3% |
| Idaho | 194 | 37.1% | 11.9% |
| Illinois | 747 | 28.5% | 54.0% |
| Indiana | 321 | 30.8% | 5.3% |
| Iowa | 845 | 33.8% | 7.2% |
| Kansas | 99 | 22.2% | 3.0% |
| Kentucky | 701 | 19.5% | 5.3% |
| Louisiana | 296 | 33.8% | 11.8% |
| Maine | 176 | 36.4% | 4.0% |
| Maryland | 981 | 29.7% | 4.0% |
| Massachusetts | 920 | 26.8% | 6.3% |
| Michigan | 148 | 22.3% | 5.4% |
| Minnesota | 528 | 33.3% | 7.2% |
| Mississippi | 572 | 25.2% | 7.7% |
| Missouri | 2,445 | 29.9% | 7.1% |
| Montana | 486 | 21.6% | 4.5% |
| Northern Marianas | 28 | 17.9% | 0.0% |
| Nebraska | 166 | 37.3% | 4.2% |
| Nevada | 302 | 35.4% | 7.9% |
| New Hampshire | 55 | 32.7% | 9.1% |
| New Jersey | 307 | 25.1% | 3.6% |
| New Mexico | 553 | 31.5% | 4.2% |
| New York | 1,147 | 37.5% | 9.2% |
| North Carolina | 507 | 30.2% | 4.7% |
| North Dakota | 145 | 29.0% | 10.3% |
| Ohio | 1,341 | 28.4% | 7.4% |
| Oklahoma | 631 | 26.3% | 4.1% |
| Oregon | 671 | 28.0% | 4.0% |
| Pennsylvania | 1,714 | 34.4% | 7.8% |
| Rhode Island | 276 | 15.9% | 6.5% |
| South Carolina | 206 | 37.9% | 9.2% |
| South Dakota | 50 | 16.0% | 4.0% |
| Tennessee | 768 | 35.5% | 13.8% |
| Texas | 1,451 | 38.0% | 9.3% |
| Utah | 98 | 22.4% | 8.2% |
| Vermont | 441 | 27.2% | 7.7% |
| Virginia | 733 | 25.6% | 7.5% |
| Washington | 749 | 28.8% | 2.8% |
| West Virginia | 484 | 25.4% | 11.0% |
| Wisconsin | 713 | 21.9% | 2.7% |
| Wyoming | 354 | 23.7% | 8.2% |
| Ontario | 476 | 41.2% | 0.0% |
An Effectiveness Analysis of SafeStat—The Motor Carrier Safety Status Measurement System
SafeStat (Safety Status Measurement System) is an automated analysis system developed for the Federal Highway Administration's (FHWA's) Office of Motor Carriers and Highway Safety (OMCHS). The system combines current and historical safety performance data to measure the relative safety fitness of interstate commercial motor carriers. SafeStat enables OMCHS to quantify and monitor the safety status of motor carriers and guides the deployment of resources to focus on carriers posing the greatest safety risk.
SafeStat resulted from research performed for OMCHS by the U.S. DOT's John A. Volpe National Transportation Systems Center, to improve motor carrier safety fitness assessment and prescribe actions to correct safety deficiencies. SafeStat was initially developed and implemented as part of a Federal/State pilot program. It has since been implemented nationally by OMCHS to identify and prioritize individual motor carriers for subsequent on-site safety compliance reviews.
An effectiveness analysis was devised to confirm that SafeStat-identified carriers were indeed high-safety-risk carriers. The study examined post-identification carrier crash experience and tested SafeStat's effectiveness by comparing the crash rates of SafeStat-identified and non-identified carriers.
Safestat Methodology Summary
SafeStat evaluates the relative safety status of individual motor carriers with respect to the rest of the motor carrier population in four analytic Safety Evaluation Areas (SEAs): Accident, Driver, Vehicle, and Safety Management. The system uses up to 30 months of motor carrier safety and normalizing data to develop measures and indicators in the four SEAs. The four SEA values are then combined into an overall safety status assessment, known as a SafeStat score.
SafeStat determines a value for each SEA in which a carrier has sufficient safety data. Each SEA value approximates the carrier's percentile rank relative to all the other carriers that have sufficient data to be assessed within the SEA. SEA values range from 0 to 100. The higher a carrier's SEA value, the worse its safety status. An unacceptable SEA value is defined as any SEA value in the worst 25th percentile (i.e., a value of 75 to 100). Figure 1 summarizes the SafeStat methodology.
SafeStat identifies only those commercial motor carriers with sufficient safety event data that have the poorest safety status. Specifically, it produces a SafeStat score for each carrier found to have two or more unacceptable SEA values. SafeStat further characterizes the worst of these carriers as "at-risk." An "at-risk" carrier is unacceptable in three or more SEAs, with an unacceptable Accident SEA counting twice. This approach is designed to identify the carriers that have the worst safety performance at any given time and, hence, are the most logical candidates for safety improvement programs or enforcement action.
The Volpe Center staff, in cooperation with OMCHS, devised an effectiveness analysis to test whether the carriers identified by SafeStat were indeed high-safety-risk carriers. Safety risk at any given time is defined as the likelihood of having crashes in the near future. By examining the post-identification crash experience of SafeStat-identified carriers, this study sought to test SafeStat's crash rate prediction capability and refine its emphasis on the components of the system that are the most closely related to high future crash rates, and to evaluate the contribution of potential new measures and indicators.
The effectiveness analysis involved the following:
- Performing a simulated SafeStat carrier identification using historical data;
- Observing the crash involvement over the immediate 18 months after SafeStat was run, both for the carriers identified by SafeStat as having a poor safety status and for other carriers that were not so identified by SafeStat but had sufficient data to be identified; and
- Comparing the post-identification crash rates of both groups of carriers.
If SafeStat is effective in identifying unsafe carriers (i.e., carriers having a high risk of being involved in future crashes), then the carriers identified as having a poor safety status would be expected to have higher post-selection crash rates than the carriers that were not identified by SafeStat. The greater the post-selection crash rate for the identified carriers relative to those carriers not identified, the more effective SafeStat would be at identifying unsafe motor carriers.
The analysis simulated carrier identification by SafeStat, using data available at an earlier date (April 1, 1996) and then observing the carriers' crash involvement that occurred over the next 18 months (from April 1996 to October 1997). This procedure simulated carrier identification by SafeStat as if it had been run as of April 1, 1996, using safety events that occurred prior to that date, and allowed for sufficient subsequent crash reporting to provide an accurate measure of the post-identification crash rates.
From this simulation run of SafeStat, carriers that had sufficient data to be scored were placed into the following groups, based on their overall SafeStat results, in order to compare their post-selection crash performance:
- Carriers identified as "at-risk" (worst SafeStat Scores—carriers with three or more unacceptable SEAs, with the unacceptable Accident SEA counting twice);
- Other carriers identified as having a poor safety status according to SafeStat (carriers with two unacceptable SEAs without an unacceptable Accident SEA);
- Carriers with sufficient data but not identified by SafeStat as having a poor safety status.
The post-identification crash rate for each group was represented by the number of reported crashes per 1,000 power units (PUs). The number of PUs is defined by the total number of trucks, tractors, hazardous material tank trucks, motor coaches, school buses, minibuses/vans, and limousines owned or term-leased by a motor carrier. The carrier PU information was based on census data that reside in the centralized OMCHS national database, the Motor Carrier Management Information System (MCMIS).
The crash data were based on crashes reported by the States (according to the National Governors' Association [NGA] standard) that occurred during the post-selection period (April 1996 to October 1997). These data also reside in the MCMIS. Each reported crash was weighted on the basis of severity and timing of the crash. The severity-weighting scheme placed emphasis on crashes with greater consequences, while the time weighting place emphasis on crashes that occurred soon after the SafeStat identification run. Severity weights were assigned as follows: a weight of 0.5 for property damage only, a weight of 1.0 for crashes involving injuries/fatalities or hazardous material release, and a weight of 1.5 for crashes involving injuries/fatalities and hazardous material release. Time weights were assigned to crashes as follows: a weight of 1.5 for crashes that occurred during the first six months after the SafeStat run, a weight of 1.0 for crashes that occurred 7 to 12 months after the SafeStat run, and a weight of 0.5 for crashes that occurred 13 to 18 months following the SafeStat run. For each crash, the severity weight was multiplied by the time weight to obtain on overall weight. In each carrier group, the weighted crashes were summed and divided by the number of PUs to provide a weighted crash rate for the group.
Data IssuesIn this analysis, "power units" (PUs) were chosen as the means of measuring exposure to normalize the State-reported NGA crash data. Assuming relatively consistent vehicle utilization rates, the number of PUs provides an estimate of the time spent traveling (when crashes can potentially occur). Discussions with other developers of safety measurement systems (Ontario and Quebec) supported the use of PUs to calculate crash rates. Vehicle miles traveled (VMT), another popular measurement of exposure, were not used because the data were either not available or not current for most carriers. VMT data as a measurement of exposure also have potential bias problems of overstating the exposure and hence favoring long-haul carriers—primarily operating at high speeds on interstate highways—relative to carriers with short-haul operations—primarily operating at low speeds, often on local roads. There was concern that inaccurate PU data (especially in cases where the PUs are understated) could bias the effectiveness analysis. To mitigate this potential bias, the Poisson distribution was used to identify carriers that had unreasonably high crash rates for the post-identification period.1 While a vast majority of the carriers (74,073) had reasonable crash rates, 52 carriers were identified as having unreasonably high crash rates, which were assumed to be based on inaccurate PU normalizing data. Thus, data on these carriers were not included in the effectiveness analysis. State-reported NGA crash data, which represent the largest, most complete set of carrier crash information that can be linked to the specific carriers involved, were used in the analysis. Although the NGA data do not provide a complete record of all carrier crashes, a recent OMCHS analysis estimated that a majority of all large truck carrier crashes are being recorded in the NGA data. There may be concern that the missing crash information could possibly bias the results of the effectiveness study, but the likelihood of a crash being recorded or not recorded in the NGA data is independent of whether the carrier has been identified by SafeStat. This independence allows the NGA data to serve as a large, unbiased sampling of crashes to be used in the post-selection period. Another possible problem of using NGA crash data is the potential for long delays between when the crashes occur and when the crashes are recorded in the NGA data. A vast majority of the crashes, however, are entered into the data system within 6 months. The effectiveness analysis used crash data available as of April 1998—6 months after the end of the post-selection monitoring period. The use of this cutoff date ensured that most of crashes that occurred during the monitoring period and were eventually recorded in the data system were used in the study. 1 The Poisson Distribution is often used as a model for the number of events (such as telephone calls at a business or crashes at an intersection) occurring in a specific time period. |
Results: Overall Effectiveness of SafeStat
The post-selection crash rates for the SafeStat-identified and non-identified carrier groups were examined in terms of (1) their overall SafeStat scores and (2) the four SEAs—Accident, Driver, Vehicle, and Safety Management—that determine the overall SafeStat score. The results are shown in Table 1.
|
Table 1 Post-Selection Crash Rates | |||
| Carrier Group | Number of Carriers | Weighted Crash Rate* | % Higher Than Non-Identified Carriers |
|---|---|---|---|
| All Identified | 4,276 | 56.4 | 85% |
| At-Risk (With Worst SafeStat Scores) | 1,450 | 82.3 | 169% |
| Other Identified (With Poor SafeStat Scores) | 2,826 | 43.2 | 41% |
| Non-Identified | 69,797 | 30.5 | — |
| *Weighted crashes per 1,000 power units. | |||
These results confirm that SafeStat did identify carriers with a higher crash risk. The group of all carriers that SafeStat identified as poor performers had a crash rate 85% higher than carriers that were not identified. The carriers designated as "at-risk" by SafeStat had a much higher crash rate (169% greater) than the carriers that were not identified.
A majority of the "at-risk" carriers were identified in part because they had previous problems with respect to their crash rates (i.e., they had unacceptable Accident SEA values); however, even the SafeStat-identified carriers in the "other identified" group, which did not have high Accident SEA values but were in the worst 25th percentile in two of the other SEAs, posed a crash risk 41% greater than the carriers that were not identified, as shown in Figure 2. This result shows that SafeStat has the ability to identify carriers that are likely to be involved in crashes, even though they have not previously had exceptionally high crash rates.
Effectiveness of Individual SEAs
Further testing was done to determine the effectiveness of the principal components of SafeStat. This was accomplished by placing carriers into groups based on their performance results for each SEA (Accident, Driver, Vehicle, and Safety Management). The results for carriers with high individual SEA values compared to those with lower SEA values are shown in Table 2. (Note that carriers with high SEA values were in the worst 25th percentile and were designated as the worst performers in that particular evaluation area. Conversely, carriers with no SEA values were not in the worst 25th percentile and, therefore, were not among the poorest performers in that SEA.)
|
Table 2 Individual SEA Values vs. No SEA Values | |||
| Safety Evaluation Area | Number of Carriers | Weighted Crash Rate* | % Higher Than Carriers With No SEA |
|---|---|---|---|
| Accident SEA | 2,596 | 81.4 | 172% |
| No Accident SEA | 71,477 | 29.9 | — |
| Driver SEA | 7,036 | 56.2 | 90% |
| No Driver SEA | 67,037 | 29.5 | — |
| Vehicle SEA | 12,456 | 38.3 | 22% |
| No Vehicle SEA | 61,617 | 31.4 | — |
| Safet Management SEA | 4,442 | 42.0 | 35% |
| No Safety Management SEA | 69,631 | 31.0 | — |
| *Weighted crashes per 1,000 power units. | |||
Discussion
Accident SEA: The results confirm what intuitively may seem obvious: carriers with high crash rates in the past are likely to continue to have high crash rates in the future. In other words, past crash rate performance is a good indicator of future crash rate performance. The effectiveness analysis shows a 172% greater post-selection crash rate for carriers with poor Accident SEAs than for carriers that were not identified as having poor Accident SEAs. Comparing SEAs, the Accident SEA is by far the most effective SEA for identifying high-risk carriers, justifying the double-weighting of the Accident SEA in calculating a SafeStat Score.
Driver SEA: The Driver SEA (with a 90% higher crash rate for carriers with poor Driver SEAs) is the next most effective SEA. These analytical results are especially impressive because the criteria for the Driver SEA are based on violations and are independent of crash history.
Vehicle SEA: Carriers with poor Vehicle SEAs did have a higher crash rate (22%) than carriers without poor Vehicle SEAs, but the difference was not as great as those for the Accident and Driver SEAs. As with the Driver SEA, the criteria for the Vehicle SEA are based on violations and are independent of crash history. Also of importance to the analysis, due to the large amount of vehicle roadside inspection data, Vehicle SEA values were computed for many more carriers than were Accident or Driver SEA values (12,456 Vehicle SEAs, compared with 2,596 Accident SEAs and 7,036 Driver SEAs). Thus, in absolute terms, the Vehicle SEA has the potential to identify more carriers.
Safety Management SEA: The Safety Management SEA is also effective in identifying carriers with high crash rates. Indicators in this SEA are based on safety regulation compliance, supporting the association of safety regulations with crash risk. The post-identification crash rate for carriers with high Safety Management SEAs was 35% higher than that for carriers without high Safety Management SEAs.
Conclusion
SafeStat does work. The effectiveness analysis shows that all the individual parts of SafeStat, and SafeStat as a whole, do indeed identify carriers that are likely to have significantly higher crash rates than carriers not identified. The effectiveness analysis has also proven to be a useful tool in quantifying the performance of SafeStat. SafeStat was designed to be continuously improved. The results of the analysis will enable the SafeStat developers and OMCHS to assess the relative strengths of SafeStat's component parts and to continue making enhancements to improve its efficiency. Finally, SafeStat continues to be strengthened and improved through the addition of better data and new indicators (most recently, a Moving Violation Indicator in the Driver SEA, which a separate analysis has shown will further increase SafeStat's effectiveness).
For more information about SafeStat, please refer to the complete description of SafeStat found in SafeStat Motor Carrier Safety Status Measurement System Methodology: Version 6.1 (October 1998), The Volpe Center, DTS-42, 55 Broadway, Cambridge, MA 02142. The Editor would like to thank Don Wright and Dave Madsen of the Volpe Center for contributing this article to MCSAFE.
For More Information . . .
 |
Information on large truck and motor coach crashes and the nature and effectiveness of the Office of Motor Carrier and Highway Safety's safety programs is available from: |
The
Office of Motor Carriers and Highway Safety
Analysis Division (HIA-20)
400 Seventh Street, SW
Washington, D.C. 20590
HIA-20 has designated four “Data Analysis Coordinators” to assist field staff with data analysis inquiries; in their absence, inquiries may be directed to any other member of the HIA-20 staff at (202) 366-1861. Faxes may be sent to (202) 366-8842.
|
Employees operating in the States served by the Eastern Resource Center should contact Richard Gruberg, (202) 366-2959. |
 |
 |
Employees operating in the States served by the Southern Resource Center should contact Ralph Craft, (202) 366-0324. |
|
Employees operating in the States served by the Midwest Resource Center should contact Chuck Rombro, (202) 366-5615. |
 |
 |
Employees operating in the States served by the Western Resource Center should contact Dale Sienicki, (202) 366-9039. |
To obtain assistance in the application and interpretation of statistics, please call Richard Gruberg, (202) 366-2959. |
|
|
DON’T FORGET TO LOOK FOR THE NEXT ISSUE OF MCSAFE! |
FMCSA Home | Feedback | Home United States Department of Transportation - Federal Highway Administration
Home |
Feedback |
Site Map |
What's New |
Site Guide
Privacy Policy | Accessibility |
DOT User
