Salt Lake Valley Health Department SLVHD provides services to 1,029,655 persons living within Salt Lake County, Utah. Salt Lake County consists of 16 incorporated cities and an unincorporated area comprised of various townships and communities. Salt Lake County is a culturally diverse community with notable representation from many different ethnic groups. The median age is 28.5 years. Approximately 3,924 permitted food establishments in the County are inspected by personnel within the Bureau of Food Protection. Over 5,000 routine restaurant inspections are conducted annually in Salt Lake County and posted on the Department’s website which provides information regarding restaurant inspection results, scores, rankings, and restaurant closures.
In April 2009, SLVHD launched the food inspection website targeting residents who eat at the County permitted restaurants and provides general public access to restaurant inspection data. The website was designed by a team in the Bureau of Food Protection who then collaborated with the Utah Restaurant Association, Utah Food Industry Association, and the Salt Lake Environmental Quality Advisory Commission for feedback.
The objectives of the website are: (1) Improve public health. Foodborne diseases result in 9.4 million illnesses, 55,961 hospitalizations and 3,151 deaths annually in the United States. One in six Americans suffer from foodborne illness each year. A primary objective of the website is to reduce critical violations and protecting public health. Effectively reducing the number of critical violations cited during routine restaurant inspections through internet posting of inspection results should decrease the occurrence of foodborne illnesses, thus improving public health.(2) Benefits the public. The inspection website offers meaningful information to the general public. It contains a multi-field search engine in which a user is able to check a restaurant’s inspection score and rating, view each violation found by inspectors, the public health rationale for each violation, the actual food code verbiage for each violation, and whether or not each violation was corrected on site. Other useful features include an inspector jargon glossary, a top ten list of most common violations in Salt Lake County, a list of current food establishment closures, links to FDA and CDC foodborne illness web pages, and a comparison function that allows a user to compare up to three restaurants side by side to see how they rate.(3) Benefits the food service operators. Many of the web site features available to the public also benefit food service operators. Establishment owners are able to keep track of how their establishment(s) performed on the most recent inspection, as well as on inspections dating back to January 2008. Since rankings are relative to other food establishments within a risk level, owners and managers can see how they match up to the competition. Managers have all of the food code and public health rationale information literally at their fingertips to aid them in understanding why a violation was cited, and in training their employees to avoid practices that lead to violations. Critical violations and critical factors appear in red to highlight the most important items on which food mangers should focus. (4) Benefits SLVHD. Food service managers’ interest in their inspection results noticeably increased after the launch of the inspection web site.
Not only were their inspection scores on display, but every detail of all of their violations was readily available to everyone in the general public. Shortly after the web site went live, corporate executives of two major restaurant chains requested meetings with SLVHD so that they could become better informed about how to perform well on routine inspections.
Responsiveness
The public health issue that this practice addresses. Preventing foodborne illness is an important public health issue. Foodborne diseases result in 9.4 million illnesses, 55,961 hospitalizations and 3,151 deaths annually in the United States. The annual economic cost of foodborne illness is estimated to be $152 billion. In 2008 the projected incidence rate for all reported foodborne illnesses in the United States was 40.22 per 100,000 population. The estimated incidence in Salt Lake County for the same time period was 36.67 per 100,000. Most foodborne illnesses can be linked to improper food handling practices. State and local health departments are responsible for regulating food safety in restaurants and implementing those new and innovative strategies. Health departments have relied primarily on training and inspections to ensure proper food handling behavior.
These two procedures have not proven effective in creating long term behavioral compliance. Studies have shown, however, that improving food safety practices, resulting in better inspection scores, reduces risk and improves public health. In response, the United States Food and Drug Administration (FDA) has called for more innovative and effective strategies to improve food safety practices in retail food establishments. Foodborne disease is a winnable battle identified by the CDC. Posting inspections scores on the internet is an innovative and effective strategy for reducing critical violations identified during routine inspections.
The process used to determine the relevancy of the public health issue to the communityThe relevance of foodborne illness and effective strategies for reducing critical violations in retail establishments was verified through a literature review of studies and information gathered from the U.S. Food and Drug Administration (FDA) and the Centers for Disease Control and Prevention (CDC) which suggest an association between inspection scores and foodborne illness. Studies conducted in Seattle-King County, Washington, and Los Angeles County, California concluded that inspection scores can identify restaurants with increased risk of foodborne illness outbreaks. A study in California compared restaurants associated with foodborne illness outbreaks with previous inspections and found inspection scores to be predictive of foodborne illnesses.
How the practice address the issueThe heightened interest in inspection scores has ultimately led to cleaner kitchens and reduced numbers of violations. It has prompted food service managers to take a more active role in training their staff to control critical factors. A comparison of average inspection scores from 6 months before the web site launch and 6 months after the web site launch revealed an average decrease of 4.3 points per inspection. The implementation of a web-based system for public access to restaurant inspection results appears to have had an impact on the frequency of critical violations noted by inspectors. Based on these results, implementing such a web-based system may lead to a 20 to 30 % reduction in the number of inspections with critical violations. Posting inspection scores on the internet appears to be an effective and efficient tool for protecting public health.
Innovation
Evidence based strategies used in developing this practiceThe practice of posting restaurant inspections on the internet was a result of a review of studies conducted by the U.S. Food and Drug Administration (FDA) and the Centers for Disease Control and Prevention (CDC) which suggest an association between inspection scores and foodborne illness. Additional studies conducted in California concluded that posting inspection scores in restaurant windows appear to have an effect on the frequency of foodborne illnesses. Internet postings were the logical next step.
This practice is new to the field of public healthProcess used to determine that the practice is new to the field of public healthAlthough posting inspection scores in establishment windows has been common in local jurisdictions, internet postings are relatively new. Analyzing the website’s effectiveness in reducing critical violations identified during routine inspections is unique to SLVHD. Municipalities nationwide have begun posting inspection scores on the internet to improve the accessibility of scores to the public; however the efficacy of internet postings is unknown. A search of PubMed and EBSCOhost databases yielded no papers examining the impact of internet access to inspection scores on subsequent scores or foodborne illness outbreaks.
How this practice differs from other approaches used to address the public health issueInternet posting of inspection scores differs from window posting in several key areas. Because the inspection results are not posted immediately, the inspector is not pressured to give the restaurant a favorable score, thus the score is a better representation of the restaurant. It is impossible for the restaurant owner to alter the inspection results once posted on the internet.
Primary stakeholders Salt Lake Valley Health Department Bureau of Food Protection, Utah Restaurant Association, University of Utah Division of Public Health, Salt Lake Environmental Quality Advisory Commission
LHD's role The website was researched and designed by a team in the Bureau of Food Protection. Bureau personnel monitor the website for accuracy and completeness.
Stakeholders/partners The Utah Restaurant Association, Utah Food Industry Association, and the Salt Lake Environmental Quality Advisory Commission provide input and feedback to the Bureau. The effectiveness analysis was a collaboration between SLVHD and the University of Utah. SLVHD personnel provided the data, conducted literature reviews, developed the analysis models, conducted the analysis, and co-authored the report. University of Utah faculty provided direction and advice regarding measures, models and processes, and co-authored the report.
SLVHD has an ongoing relationship with several of the departments at the University of Utah, Westminster College, and Salt Lake Community College in addition to the Utah Restaurant Association and the Salt Lake Environmental Quality Advisory Commission. These relationships have been reciprocal over the years wherein both the shareholders and SLVHD have initiated and benefited from the various arrangements. SLVHD has benefitted from their expertise and the community stakeholders have gained access to data and other SLVHD resources. Communication and collaboration with the community are part of the SLVHD values. Many of the programs administered by SLVHD are collaborative efforts with community stakeholders.
Program success indicators include 1) programs which are based on community needs and 2) the community is knowledgeable and satisfied with our services. The quality, credibility and results of the evaluations conducted in this study were greatly enhanced due to the collaboration with the University of Utah faculty. Their expertise in designing and conducting effectiveness evaluations saved SLVHD countless hours of time and resources. The study was reviewed and critiqued throughout the process to ensure an accurate and valuable assessment.
Lessons learned Stakeholders need to be involved in every step of the program. Resistance from some stakeholders regarding the most appropriate method of ranking the restaurants slowed the process at critical times. It is advantageous to have those who will be conducting the analysis participate in the process to ensure all of the required data will be collected. Stakeholders need to remember the purpose of the website and not be distracted by their personal agendas or preferences.
Tasks taken that achieve each goal and objective A committee consisting of food inspectors and food bureau supervisors was formed and given the assignment to develop a website which would reduce critical violations by providing restaurant inspection information to the public and restaurant owners or operators. The committee reviewed all existing restaurant inspection websites and identified the features they felt would benefit the website and those which were detrimental to the website purpose. Search and ranking features were identified and evaluated. Once the basic website template was developed, feedback was received from the Utah Restaurant Association. The revised template was then given to the Department’s data contractor for website development and beta testing. Eighteen months after the committee was formed, the website was launched.
Time frame The committee and stakeholders accomplished their tasks within six months. The greatest amount of time was spent in website development and beta testing. The time required to develop and implement an inspection website depends largely on the development contractor. The website could have been implemented sooner if the Department had internal resources to devote to the project rather than contracting with an outside firm for website development. Obviously, the more complex the website, the longer it will take to develop and implement. Because SLVHD included searchable fields and a ranking system, the project took longer to complete.
Succinct outlineThe steps taken to implement the website are as follows: 1. Develop committee assigned to determine objectives and purposes of the website. 2. Review other websites, talk with other jurisdictions which have websites, retain the useful functions and discard the less useful elements of those websites. 3. Review draft website design with stakeholders and revise as needed. 4. Deliver draft website design, purposes and elements to the website design contractor for development, beta testing and implementation. 5. Launch website. 6. Evaluate website based on objectives. 7. Contractor maintains website and updates as needed.
Lessons learnedImplementing an inspection website takes longer than expected. Software development and beta testing were very time consuming. It is also important to plan for a large number of hits when the website is launched. Hits will reduce after the initial launch, but without proper planning, the server may crash.
Cost of implementationThe cost to develop, launch and monitor the website is included a data contract cost and is not broken out as a separate item, therefore the exact cost of development and implementation is unknown. However, the cost to design, implement and monitor a website is minimal compared to other programs designed to reduce critical violations.
Objective 1:Website Objective. The primary objective of the website is to reduce critical violations by providing restaurant inspection information to the public and restaurant owners or operators. The design of the website included a search engine in which a user is able to check a restaurant’s inspection score and rating, view each violation found by inspectors, the public health rationale for each violation, the actual food code verbiage for each violation, and whether or not each violation was corrected on site. Other useful features include an inspector jargon glossary, a top ten list of most common violations in Salt Lake County, a list of current food establishment closures, links to FDA and CDC foodborne illness web pages, and a comparison function that allows a user to compare up to three restaurants side by side to see how they rate.
Performance Measures. The occurrence of five specific, FDA defined, critical violations were included as outcome measures: poor hygiene practices (lack of handwashing and eating, drinking or using tobacco in the food preparation area), improper holding temperatures (hot and cold holding temperature citations), inadequate equipment cleanliness (dirty equipment, utensils or food contact surfaces), lack of protection from cross-contamination (food separation, packaging and segregation citations), and improper sanitizer concentration (improper manual and mechanical ware washing chemical sanitizer levels). These are referred to as ‘target critical violations’. These violations were selected because they were included in the domains identified by the FDA as “most commonly reported to the Centers for Disease Control and Prevention (CDC) as contributing factors in foodborne illness outbreaks”.
Data. This study uses a longitudinal analysis of inspection results. Study data were limited to routine inspections performed by licensed environmental health specialists at full service and fast food restaurants in Salt Lake County, Utah. Inspections conducted 12 months before and after the website launch were included. Only those restaurants which were inspected at least twice before and twice after launch were included in the study. A total of 2,995 inspections conducted at 795 restaurants were evaluated. Secondary data were obtained from the SLVHD database and redacted to ensure anonymity. Univariate and bivariate methods were used to assess the changes.
Multivariable methods were used to assess the changes in inspection results. Logistic regressions using a Generalized Estimating Equation (GEE) were used to account for the repeated measures from each restaurant. The outcome variables were each dichotomous indicating whether the restaurant was cited for that type of critical violation. A first order autoregressive (AR1) correlation structure was assumed to account for any correlation between the previous and current inspection. Based on the quasi-likelihood criteria, the AR1 structure performed as well as other correlation structures. Odds ratios were used to describe the effect of the intervention. Data were analyzed using STATA software version 9 (College Station, Texas).
Models were adjusted for restaurant type (full service compared to fast food), inspector experience and season. Experienced inspectors were defined as those who conducted 200 or more inspections during the study period. Season was grouped into four 3 month periods: Summer (June, July, August), Fall (September, October, November), Winter (December, January, February), Spring (March, April, May).
Evaluation Results. The proportion of inspections with any of the targeted violations was lower during the twelve months after the launch compared to before the launch. The difference in proportions ranged from 2.07 (cross-contamination) to 8.99 (equipment cleanliness).
The probability of having a violation decreased in routine inspections conducted after the website launch when adjusted for inspector experience, risk level and seasonality. The adjusted odds ratios, ranging from 0.64 to 0.80, were statistically significant for all critical violations, with the exception of cross-contamination which was borderline significant (p=0.053). The largest effect was found in equipment cleanliness violations (aOR=0.64).
Full service restaurants were more likely to be cited for all of the targeted critical violations than fast food facilities, with adjusted odds ratios ranging from 1.43 (p=0.015) in sanitizer concentration to 2.12 (p=0.000) in hygiene violations.
Feedback. Results of the assessment have been presented to University of Utah faculty, the SLVHD Board of Health and the Utah Environmental Health Association Fall Conference. The implementation of a web-based system for public access to restaurant inspection results appears to have had some impact on the frequency of critical violations noted by inspectors. It was determined the website is an effective and efficient tool for reducing critical violations cited during restaurant inspections.
Stakeholder commitmentThe Utah Restaurant Association continues to monitor the website and make suggestions for its improvement. Most restaurant owners support the website and have learned to use the information to promote their establishment and revise their marketing. The website provides motivation to owners of poor scoring restaurants to improve their inspection scores and possibly increase the number of customers coming to their establishment.
Public input regarding use of the website indicates overall support. The public have provided ideas and comments for improving the website in the future. The Department is exploring the use of QR technology to link restaurant customers to the website after a suggestion from the public. Continued improvement following public comments ensures the website will be a service the public seeks out and utilizes.
The University of Utah is committed to continuing evaluation of the website through additional studies. The Division of Public Health is continually suggesting changes to the website which results in an opportunity to study the effectiveness of those changes in reducing risk. The studies which have been conducted to date have been presented at national conferences and have been accepted for publication in leading journals. This brings notoriety to the University and opportunity for further studies.
SustainabilityThe cost of sustaining the website is included in our data maintenance contract. Administration has committed to retaining and improving the website due to its effectiveness in reducing critical violations and public acceptance.