What the National Academies of Sciences CSA Recommendations Will Mean
Written by: Sean Garney, Director, Safety Policy, ATA
Compliance, Safety, Accountability (CSA) has been studied no less than nine times, by private organizations, research institutes, universities as well as several agencies inside the Department of Transportation, and now, the National Academies of Science. It has been bashed and vindicated, depending on your vantage point. Serious flaws in the system have been uncovered and FMCSA has, in some ways, worked to improve them. The ability of the CSA scoring system to accurately portray the crash risk of each individual motor carrier is still in question, which has become of paramount importance to the trucking industry as more and more people — bankers, insurers, shippers, plaintiff’s attorneys, to name a few — are looking to CSA as the final word on the safety of the fleet.
Sure, Congress removed the scores from public view in 2015 when they passed the Fixing America’s Surface Transportation Act but, the proverbial cat is already out of the bag. Post-crash liability concerns have scared everyone, even those who question the validity of these scores, into relying on them and even requiring them as a condition of contract. To motor carriers who may have been unfairly labeled as unsafe, this is a travesty. To the hundreds of thousands of motor carriers who have not been scored by the system, well, they get a free pass, regardless of the quality of their equipment, drivers or safety programs.
This is the frame through which we must view the National Academies of Sciences (NAS) report on CSA, “Improving Motor Carrier Safety Measurement.” Congress gave NAS a long list of things to study and consider and this diligently prepared report makes many interesting recommendations.
First and foremost, NAS concluded that while some of the assumptions made in crafting CSA were “reasonable” and “defensible,” FMCSA missed the mark by not using the correct statistical model. They should have used the far more complicated and dynamic Item Response Theory (IRT) model. This model, the researchers claim, can take some of the guesswork out of the system by calibrating peer groups and violation time and severity weights, account for data insufficiency, help determine which violations belong in which Behavioral Analysis and Safety Improvement Categories (BASICs), and even how many BASICs there should be. This model may hold promise if it can account for some complaints the industry has been leveling at FMCSA for nearly a decade, like the impact of geographic enforcement disparity, the subjectivity of violation severity weights, and peer group variability, to name a few. Don’t ask me how it works though. It’ll look something like this:
NAS also concluded that CSA has some glaring deficiencies in motor carrier exposure data. Essentially, the system may be less accurate because it doesn’t do a great job understanding a carrier’s exposure to crash risk (e.g. how many miles it runs, where those miles occur, and over how many power units those miles should be spread) or how many crashes are actually occurring. If we understand the degree to which motor carriers are exposed to crash risk (especially considering nearly 80% of all car-truck crashes can be attributable to the actions of the car driver) or the actual number of crashes a carrier is involved with, then we can’t possibly measure risk properly.
With these three recommendations however, we approach what will likely be the biggest challenge of any reform effort. How do we gather enough reliable, high quality data to make a complicated system like IRT, or even a relatively simple system like today’s CSA methodology, work accurately and efficiently? Better mileage data, collected monthly and sorted by state may be available in International Fuel Tariff Agreement (IFTA) data, but obtaining it may require an act of Congress. Certainly, the collection of crash data could improve through efforts already under way in States, but carriers will be hard pressed to begin divulging driver compensation data, especially if it will be publicly available.
Another frequent complaint of CSA has been the use of percentile BASIC scores, which rank motor carriers against others in their peer group, to place motor carriers into alert status. Whether or not a carrier is placed in Alert status is important because their customers and business partners may withhold business opportunities for having a single BASIC in alert status. Here, the argument goes that a motor carrier, who has been working hard to improve safety, may not see a measurable improvement in their safety scores if their peers have also improved. These carriers deserve recognition that may never materialize. Some argue that relative percentile scores discourage collaboration among peers however, as motor carriers seek to gain a competitive advantage. Defenders argue that relative percentile scores will push motor carriers toward continuous improvement because the safety threshold will continue to elevate as carriers improve.
To remedy this, NAS recommended that FMCSA consider a hybrid approach to determining whether or not a carrier should be placed in alert status. This method should utilize both percentile score and the underlying measure to make this important distinction.
Unfortunately, NAS avoided taking a position on the most contentious debate since the inception of CSA: whether or not scores should be publicly available information. While acknowledging the various ways the data could be misunderstood and misinterpreted, NAS ultimately recommended that FMCSA study the issue to determine the consequences of public CSA scores and the rate at which it misidentifies motor carriers for alert when in fact, they are operating safely.
What does the future hold? According to the FAST Act, FMCSA has until late October 2017 to develop a corrective action plan to address the recommendations of NAS. FMCSA has indicated they are committed to pursuing the IRT model and asked for stakeholder feedback and suggestions as they develop their corrective action plan. ATA will remain engaged in this process in pursuit of our shared goal of finding a fair and equitable process to identify the least safe motor carriers and targeting them for enforcement intervention. Doing this will improve the safety of all highway users.
2017 Transportation Spotlight Report
2017 Transportation Spotlight Report
2017 marks the 10-year anniversary of HireRight’s Employment Background Screening Benchmark Survey.
The American Trucking Associations prepared these materials for HireRight for informational purposes only. These materials are not intended to be comprehensive, and are not a substitute for, and should not be construed as, legal advice. The American Trucking Associations and HireRight do not warrant any statements in these materials. Employers should direct to their own experienced legal counsel questions involving their organization’s compliance with or interpretation or application of laws or regulations and any additional legal requirements that may apply.