Identifying foodborne outbreaks using social media

Posted on January 10, 2018 by Douglas Powell

As a new survey shows 95% of chefs cite customers getting sick as their top concern, a computer system developed by Columbia University with Health Department epidemiologists can detect foodborne illness and outbreaks in New York City restaurants based on keywords in Yelp reviews.

Using Yelp, 311, and reports from health care providers, the Health Department has identified and investigated approximately 28,000 complaints of suspected foodborne illness overall since 2012 and helped Health Department staff identify approximately 1,500 complaints of foodborne illness in NYC each year, for a total of 8,523 since July 2012.

Improvements to the computer system are the subject of a joint study published this week by the Journal of the American Medical Informatics Association. The Health Department and Columbia continue to expand the system to include other social media sources, such as Twitter, which was added to the system in November 2016. The computer system allows the Health Department to investigate incidents and outbreaks that might otherwise go undetected. New Yorkers are encouraged to call 311 to report any suspected foodborne illness.

“Working with our partners at Columbia University, the Health Department continues to expand its foodborne illness surveillance capabilities,” said Health Commissioner Dr. Mary T. Bassett. “Today we not only look at complaints from 311, but we also monitor online sites and social media. I look forward to working with Columbia University on future efforts to build on this system. The Health Department follows up on all reports of foodborne illness – whether it is reported to 311 or Yelp.”

Each year,

“Effective information extraction regarding foodborne illness from social media is of high importance–online restaurant review sites are popular and many people are more likely to discuss food poisoning incidents in such sites than on official government channels,” said Luis Gravano and Daniel Hsu, who are coauthors of the study and professors of Computer Science at Columbia Engineering. “Using machine learning has already had a significant impact on the detection of outbreaks of foodborne illnesses.”

“The collaboration with Columbia University to identify reports of food poisoning in social media is crucial to improve foodborne illness outbreak detection efforts in New York City,” said Health Department epidemiologists Vasudha Reddy and Katelynn Devinney, who are coauthors of the publication. “The incorporation of new data sources allows us to detect outbreaks that may not have been reported and for the earlier identification of outbreaks to prevent more New Yorkers from becoming sick.”

“I applaud DOHMH Commissioner Bassett for embracing the role that crowdsourcing technology can play in identifying outbreaks of foodborne illness. Public health must be forward-thinking in its approach to triaging both everyday and acute medical concerns,” said Brooklyn Borough President Eric Adams.

Most restaurant-associated outbreaks are identified through the Health Department’s complaint system, which includes 311, Yelp, and reports from health care providers. Since 2012, the Department has identified and investigated approximately 28,000 suspected complaints of foodborne illness overall. The Health Department reviews and investigates all complaints of suspected foodborne illness in New York City.

Yelp pages to display California county health inspection ratings

Posted on June 11, 2016 by Douglas Powell

Soon, diners checking out Sacramento County restaurants on Yelp will be getting a bit more information: each restaurant’s health inspection report and its green, yellow or red rating reports Cathie Anderson of The Sacremento Bee

Luther Lowe, Yelp’s vice president of public policy, told The Sacremento Bee the goal is to put vital health information in a place where consumers can see it rather than having it at a .gov website that very few people actually access. The company’s fact sheet says millions of unique visitors access Yelp’s database each month: 21 million on the mobile app, 69 million on the mobile web page and 77 million on the desktop website.

“When people use Yelp to find a restaurant, they’re in the middle of deciding where they’re going to go eat,” Lowe said, “and so if we can show them the restaurant hygiene score when they’re looking, that’s incredibly powerful information for the consumer.”

Yelp already publishes restaurant inspections for Los Angeles County, San Bernardino County, San Diego County, Riverside County and other government agencies around the nation. Over the past three years, it has added jurisdictions as they have put their data in a format compatible with Yelp’s system.

“We have our own format,” Lowe said, “and we announced to the world, ‘Listen, if you put the restaurant grade in Column B and restaurant address in Column C, then we’ve basically created a template format that your data will import into.’ That’s all they have to do to participate in the program. Now, Sacramento adheres to the format we provided.”

Can Yelp help in tracking outbreaks of food poisoning?

Posted on October 22, 2015 by Douglas Powell

Doug Powell, a former professor of food safety at Kansas State University who now lives in Australia and writes for barfblog.com, regards Yelp and social media as potentially useful tools for public-health investigators.

“But it doesn’t replace boots on the street, the epidemiological work that people have to do,” he said. “All these things have to be taken with a grain of salt, because Yelp is a business.”

That’s what I told Barbara Feder Ostrov, writing for The Atlantic and PBS, who says that when an outbreak of Shigella sickened 98 diners at a San Jose restaurant last weekend, Yelp reviewers were on the case, right alongside public health officials.

“PLEASE DO NOT EAT HERE!!!!” Pauline A. wrote in her Oct. 18 review of the Mariscos San Juan #3 restaurant. “My sister in and brother-in-law along with his parents ate here Friday night and all four of them ended up in the hospital with food poisoning!!!”

That same day, the Santa Clara County Public Health Department shut down the restaurant. Two days later, officials announced that more than 80 people who had eaten there had become acutely ill, with many requiring hospitalization. Twelve diners went to intensive care units.

Since then, the outbreak has grown to more than 90 cases in Santa Clara and Santa Cruz counties.

Some health researchers and public health professionals believe consumer review sites like Yelp might just help them identify and investigate food poisoning outbreaks similar to this one. It’s not unlike using Google searches to track potential flu and Dengue outbreaks.

Public health workers in New York, aided by Columbia University researchers, scanned thousands of Yelp reviews in 2012 and 2013 to find previously undetected food-borne illness, unearthing nearly 900 cases that were worthy of further investigation by epidemiologists. Ultimately, the researchers found three previously unreported restaurant-related outbreaks linked to 16 illnesses that would have merited a public health investigation if officials had known of them at the time. Follow-up inspections of the restaurants found food-handling violations.

In another study, researchers from Boston Children’s Hospital analyzed more than 5,800 Yelp reviews of food services businesses near 29 colleges in 15 states, concluding that reviews describing food poisoning tracked closely with food-borne illness data maintained by the U.S. Centers for Disease Control and Prevention. The timeliness and often-graphic details of the reviews could prove useful for public health agencies investigating food poisoning outbreaks, the researchers concluded.

Researchers also have examined Twitter and Facebook as possible food-borne illness surveillance tools, and Chicago’s public health agency automatically sends information about its Foodborne Chicago reporting site to local Twitter users who complain of food poisoning.

But Yelp’s usefulness for epidemiologists is going to depend a lot on how it handles food poisoning complaints down the road.

The company has been accused of approaching restaurants to remove negative reviews in exchange for advertising dollars, although a class action lawsuit on those grounds was dismissed.

On Tuesday, the company placed an “Active Cleanup Alert” notice on Mariscos San Juan #3’s review page noting that because the business “recently made waves in the news,” Yelp would “remove both positive and negative posts that appear to be motivated more by the news coverage itself than the reviewer’s personal consumer experience with the business.”

While some reviews were easily visible, others were segregated into Yelp’s “not currently recommended” category, which requires readers to click to see them and do not figure in the establishment’s overall rating.

The key is supplement: Statisticians using social media to track foodborne illness

Posted on August 13, 2015 by Douglas Powell

The American Statistical Association reports the growing popularity and use of social media around the world is presenting new opportunities for statisticians to glean insightful information from the infinite stream of posts, tweets and other online communications that will help improve public safety.

Two such examples–one that enhances systems to track foodborne illness outbreaks and another designed to improve disaster-response activities–were presented this week at the 2015 Joint Statistical Meetings (JSM 2015) in Seattle.

In a presentation titled “Digital Surveillance of Foodborne Illnesses and Outbreaks”, biostatistician Elaine Nsoesie unveiled a method for tracking foodborne illness and disease outbreaks using social media sites such as Twitter and business review sites such as Yelp to supplement traditional surveillance systems. Nsoesie is a research fellow in pediatrics at Boston Children’s Hospital.

The study’s purpose was to assess whether crowdsourcing via online reviews of restaurants and other foodservice institutions can be used as a surveillance tool to augment the efforts of local public health departments. These traditional surveillance systems capture only a fraction of the estimated 48 million foodborne illness cases in the country each year, primarily because few affected individuals seek medical care or report their condition to the appropriate authorities.

Nsoesie and collaborators tested their nontraditional approach to track these outbreaks. The results showed foods–for example, poultry, leafy lettuce and mollusks–implicated in foodborne illness reports on Yelp were similar to those reported in outbreak reports issued by the U.S. Centers for Disease Control and Prevention.

“Online reviews of foodservice businesses offer a unique resource for disease surveillance. Similar to notification or complaint systems, reports of foodborne illness on review sites could serve as early indicators of foodborne disease outbreaks and spur investigation by local health authorities. Information gleaned from such novel data streams could aid traditional surveillance systems in near real-time monitoring of foodborne related illnesses,” said Nsoesie.

The lack of near real-time reports of foodborne outbreaks reinforces the need for alternative data sources to supplement traditional approaches to foodborne disease surveillance, explained Nsoesie. She added Yelp.com data can be combined with additional data from other social media sites and crowdsourced websites to further improve coverage of foodborne disease reports.

Cities using social media to police restaurants

Posted on March 25, 2015 by Douglas Powell

While cities like Guelph, Ontario, are being dragged into the age of public disclosure, countries like Singapore have been training and using restaurant patrons as gumshoes for a decade to help public health types identify possible infractions through the use of cell phones (with nifty cameras).

The U.S. is slowly catching on, using Yelp to check health inspection scores for eateries in San Francisco, Louisville, Kentucky, and several other communities.

Local governments increasingly are turning to social media to alert the public to health violations and to nudge establishments into cleaning up their acts. A few cities are even mining users’ comments to track foodborne illnesses or predict which establishments are likely to have sanitation problems.

“For consumers, posting inspection information on Yelp is a good thing because they’re able to make better, informed decisions about where to eat,” said Michael Luca, an assistant professor at Harvard Business School who specializes in the economics of online businesses. “It also holds restaurants more accountable about cleanliness.”

In recent years, dozens of city and county health departments have been posting restaurant inspection results on government websites to share with the public. Turning to Yelp or other social media, or using crowd-sourced information to increase public awareness, is the next logical step, some officials say.

“Yelp is a window into the restaurant. The restaurateurs don’t want a bad (health) score on Yelp. They’ll be more attentive about getting the restaurants cleaned up and safer,” said Rajiv Bhatia, former environmental health director for the San Francisco Department of Public Health.

“It’s also valuable because it allows the public to see the workings of a government agency, and puts some pressure on the agency to do its job,” said Bhatia, a physician who is now a public health consultant.

The National Restaurant Association, the industry’s trade group, said that while it supports transparency and consumers’ access to information, it worries that because inspection standards differ from city to city, Yelp users might not be familiar with rating terminology and therefore could draw incorrect conclusions.

David Matthews, the association’s general counsel, also said the timing of postings is crucial because restaurants often correct findings and generate different ratings after a re-inspection.

Luther Lowe, Yelp’s director of public policy said putting health scores and inspection results in an accessible place where consumers already are searching for restaurant information makes a lot more sense than “relying on those clunky (health department) dot-gov websites.”

People love this stuff: Yelp partners with Socrata to access and distribute restaurant inspection data

Posted on January 15, 2015 by Douglas Powell

A new partnership between Yelp and Socrata, the Seattle-based government data technology company, promises to bring restaurant inspection information to more Yelp users around the word, by giving cities, counties and governmental agencies new ways to connect their databases to the widely used business review service.

Under the partnership, Yelp will become a member of Socrata’s Open Data Network, and government users of Socrata’s Open Data Portal will receive tools and guidance to help them connect their data to Yelp’s systems, according to a news release from the companies this morning.

“With this behind-the-scenes data integration, millions of people will be able to benefit from better health information, which will ultimately improve their lives,” said Socrata CEO Kevin Merritt in the release.

Accessing and understanding public health data about restaurants can be a major challenge for restaurant customers, as illustrated by Seattle resident Sarah Schacht’s quest to get King County to make its inspection data more accessible to the public, using placards at restaurants.

Yelp makes restaurant inspection data available in some cities, but it’s not yet a widespread practice.

“Extracting restaurant inspections data from government databases is not always easy,” explains Socrata. “Additionally, the majority of cities throughout the United States are not yet publishing their restaurant inspection information in a format that can be consumed by ‘business-to-consumer’ solutions such as Yelp. The strategic partnership between Socrata and Yelp, as well as their new solution, is a first in that it helps governments unlock this data while normalizing and presenting it to everyday consumers in a way that is easy to understand and in a context with which they are familiar.”

The partnership will use the Local Inspector Value Entry Specification (LIVES) open data standard, which helps software developers analyze restaurant inspection data. Socrata says this approach is already being applied by one of its customers, the DataSF, the data portal for the City and County of San Francisco.

Tweeting for foodborne illness

Posted on October 19, 2014 by Douglas Powell

In less than a year, a Chicago Department of Public Health website launched to track Twitter traffic for foodborne illness complaints turned up 21 restaurants that failed unannounced health inspections (Harris JK et al. MMWR Morb Mortal Wkly Rep. 2014;63[32]:681-685).

Dubbed Foodborne Chicago, the website uses an algorithm that parses Chicago-area tweets that include the words “food poisoning.” Project staff members then review the tweets for references to stomach cramps, diarrhea, vomiting, or other terms than may indicate food-borne illness. Staff members respond and ask the Twitter users to report on Foodborne Chicago their illness and where they ate. The web forms go directly to the Chicago 311 system that handles nonemergency city services. From March 2013 to January 2014, Foodborne Chicago identified 2241 “food poisoning” tweets, of which 270 described specific food-borne illness complaints. Eight of those 270 tweets mentioned a visit to a physician or a hospital emergency department. Overall, 193 food poisoning complaints were submitted through Foodborne Chicago. About 10% sought medical care.

The complaints triggered unannounced health inspections at 133 restaurants; 21 failed their inspections and were closed. Another 33 restaurants passed with conditions, indicating that serious or critical violations were identified and corrected.

In related news, Carol Beach of The Packer says Foodborne Chicago researcher, Jenine Harris of Washington University in St. Louis, reported health officials in Boston and New York City are considering similar Twitter taps.

In September, researchers at Virginia Polytechnic Institute published results in the journal “Prevention Medicine” that showed a strong correlation between negative customer reviews on the website Yelp and foodborne illness outbreaks tracked by the CDC.

The study included more than 5,800 reviews of restaurants posted from 2005 through 2012.

Results showed that social media reviews could complement traditional outbreak surveillance methods by providing rapid information on suspected foodborne illnesses, the implicated foods and the restaurants involved, according to the research report.

The Virginia researchers looked at five categories of food and the rates at which Yelp reviewers reported an illness compared to the rates of CDC’s reported illness information and found very similar results:

Vegetables implicated in 22% of illnesses reported on Yelp, 25% from CDC;

Fruits and nuts implicated in 7% on Yelp, 7% from CDC;

Meat and poultry implicated in 32% on Yelp, 33% from CDC;

Dairy and eggs implicated in 23% on Yelp, 23% from CDC; and

Seafood implicated in 16% percent on Yelp, 12% from CDC.

Elaine Nsoesie, co-author of the study and postdoctoral research fellow at Boston Children’s Hospital and Harvard University, wrote that consumer reviews or tweets about illnesses could be an an additional tool to help public health authorities detect outbreaks earlier.

‘More info better for consumers’ NC County restaurant inspection ratings to appear on Yelp pages

Posted on October 9, 2014 by Douglas Powell

Restaurant inspections have long revealed the dirtier side of food locales, including the restaurants frequented by residents of Chapel Hill and Carrboro.

In an effort to make inspection ratings more accessible, Orange County announced Sept. 16 that it would begin posting its health inspection ratings to Yelp. A restaurant’s most recent letter grade now appears in the right-hand column of its Yelp page.

Victoria Hudson, an environmental health specialist for the Orange County Health Department, conducts inspections in Carrboro and parts of Chapel Hill and UNC. She said more access to information is beneficial to consumers.

“People should be able to use these scores to assign risk,” Hudson said. “The letter ‘A’ does not necessarily give you the full picture as much as the list of comments does.”

Clicking on the inspection score on Yelp reveals more information on previous inspections, including the dates and the number of health code violations found.

In May, for example, an inspection found pink and black mold in the ice machine at R&R Grill, though mold in ice machines was not an uncommon violation at restaurants in 2013 findings. The restaurant lost 1.5 points — a half deduction — and received a 98.5 total score.

Ross Moll, the owner of R&R Grill, said employees cleaned the machine after it was discovered. The machine is cleaned weekly and inspected to prevent the problem.

“I think they do a good job coming down on people who are not up to snuff on things, and they definitely work with people to get things fixed,” Moll said about the Health Department.

Tony Sustaita, owner of Bandido’s, said the inspections help reinforce safe practices.

“Obviously the policy is to be clean all the time, but people mess up once in a while,” he said. “Any issue that is brought up in an inspection is addressed immediately.”

Both Moll and Sustaitia said displaying scores in restaurants and on Yelp helps consumers make decisions.

“I think the only ones who would be concerned would be the ones with negative scores,” Sustaita said. “We’ve had pretty good

It’s called barfblog.com for a reason: sick, vomit, diarrhea key words on the social media radar of food safety authorities

Posted on June 3, 2014 by Douglas Powell

Australian health authorities are tracking a New York City initiative that uses social media and restaurant review websites to investigate cases of food poisoning.

Duh.

A collaboration between the New York City Department of Health and Mental Hygiene, Columbia University and popular review site Yelp resulted in the discovery of previously undocumented cases of food-borne illness originating in restaurants.

A spokesperson for the New South Wales Food Authority, said it will monitor the New York program and “any other jurisdictions that may have something similar.”

The NSW Department of Health, which manages complaints about restaurant hygiene from the public, said it did not use social media to research incidents of foodborne illness but the approach had merit.

“The City of New York example demonstrates an interesting use of social media which allows consumers to share information regarding foodborne illnesses,” a NSW Department of Health spokesperson told IT Pro.

“NSW Health does not currently use this type of approach, but is looking at ways to increase the use of innovative technology to identify food-borne outbreaks.

The Victoria Department of Health did not respond to a request for comment.

Sharing barf stories from Yelp helps NYC identify unreported cases of foodborne illness

Posted on May 23, 2014 by Douglas Powell

Social media and sharing of information has tremendous potential to identify foodborne and other illnesses. It also has tremendous potential to drain scarce public health resources.

The U.S. Centers for Disease Control reports that while investigating an outbreak of gastrointestinal disease associated with a restaurant, the New York City Department of Health and Mental Hygiene (DOHMH) noted that patrons had reported illnesses on the business review website Yelp (http://www.yelp.com) that had not been reported to DOHMH. To explore the potential of using Yelp to identify unreported outbreaks, DOHMH worked with Columbia University and Yelp on a pilot project to prospectively identify restaurant reviews on Yelp that referred to foodborne illness. During July 1, 2012–March 31, 2013, approximately 294,000 Yelp restaurant reviews were analyzed by a software program developed for the project. The program identified 893 reviews that required further evaluation by a foodborne disease epidemiologist. Of the 893 reviews, 499 (56%) described an event consistent with foodborne illness (e.g., patrons reported diarrhea or vomiting after their meal), and 468 of those described an illness within 4 weeks of the review or did not provide a period. Only 3% of the illnesses referred to in the 468 reviews had also been reported directly to DOHMH via telephone and online systems during the same period. Closer examination determined that 129 of the 468 reviews required further investigation, resulting in telephone interviews with 27 reviewers. From those 27 interviews, three previously unreported restaurant-related outbreaks linked to 16 illnesses met DOHMH outbreak investigation criteria; environmental investigation of the three restaurants identified multiple food-handling violations. The results suggest that online restaurant reviews might help to identify unreported outbreaks of foodborne illness and restaurants with deficiencies in food handling. However, investigating reports of illness in this manner might require considerable time and resources.

Project Protocol

Beginning in April 2012, Yelp provided DOHMH with a private data feed of New York City restaurant reviews. The feed provided data publicly available on the website but in an XML format, and text classification programs were trained to automatically analyze reviews. For this pilot project, a narrow set of criteria were chosen to identify those reviews with a high likelihood of describing foodborne illness. Reviews were assessed retrospectively, using the following criteria: 1) presence of the keywords “sick,” “vomit,” “diarrhea,” or “food poisoning” in contexts denoting foodborne illness; 2) two or more persons reported ill; and 3) an incubation period ≥10 hours. Ten hours was chosen because most foodborne illnesses are not caused by toxins but rather by organisms with an incubation period of ≥10 hours (1). Data mining software was used to train the text classification programs (2). A foodborne disease epidemiologist manually examined output results to determine whether reviews selected by text classification met the criteria for inclusion, and programs with the highest accuracy rate were incorporated into the final software used for the pilot project to analyze reviews prospectively.

The software program downloaded weekly data and provided the date of the restaurant review, a link to the review, the full review text, establishment name, establishment address, and scores for each of three outbreak criteria (i.e., keywords, number of persons ill, and incubation period), plus an average of the three criteria. Scores for individual criteria ranged from 0 to 1, with a score closer to 1 indicating the review likely met the score criteria.

Reviews submitted to Yelp during July 1, 2012–March 31, 2013 were analyzed. All reviews with an average review score of ≥0.5 were evaluated by a foodborne disease epidemiologist (Figure). Because the average review score was calculated by averaging the individual criteria scores, reviews could receive an average score of ≥0.5 without meeting all individual criteria. Reviews with an average review score of ≥0.5 were evaluated for the following three criteria: 1) consistent with foodborne illness occurring after a meal, rather than an alternative explanation for the illness keyword; 2) meal date within 4 weeks of review (or no meal date provided); 3) two or more persons ill or a single person with symptoms of scombroid poisoning or severe neurologic illness. Reviews that met all three of these criteria were then investigated further by DOHMH. In addition, reviews were investigated further if manual checking identified multiple reviews within 1 week that described recent foodborne illness at the same restaurant.

To identify previously reported complaints of foodborne illness, reviews were compared with complaints reported to DOHMH by telephone or online at 311, New York City’s nonemergency information service that can be used by the public to report suspected foodborne illness (3). Yelp reviews categorized as indicating recent or potentially recent illness were compared with complaints from the previous 4 weeks in the 311 database. To follow up with reviewers, DOHMH created a Yelp account to send private messages to reviewers’ Yelp accounts. Reviewers needed to log in at Yelp to view their messages.

For reviews not requiring further investigation and not found in the 311 database, DOHMH sent messages advising reviewers of the availability of 311 reporting. For reviews requiring further investigation, DOHMH sent messages requesting telephone interviews. Reviewers consenting to interviews were asked to provide details about the restaurant visit, meal date, foods consumed during the meal, party size, illness symptoms, and a history of foods consumed in the 3 days before symptom onset.

Review-Based Findings

During July 1, 2012–March 31, 2013, the software system screened approximately 294,000 reviews and identified 893 with an average score of ≥0.5, indicating possible foodborne illness (Figure). Of these reviews, 499 (56%) described an event consistent with foodborne illness, as determined by the manual checking of a foodborne epidemiologist. This equated to an average of 23 reviews evaluated by a foodborne epidemiologist each week, with an average of 13 reviews categorized as consistent with foodborne illness. The remaining 394 (44%) reviews contained keywords but did not suggest foodborne illness (e.g., “I didn’t get sick at all after my meal”).

Of the 499 reviews describing an event consistent with foodborne illness, 468 (94%) indicated recent or potentially recent illness. Of these 468 reviews, only 15 (3%) were also reported to 311 during the same period. A total of 339 reviews that indicated only one person became ill and had no scombroid poisoning or severe neurologic symptoms were excluded, leaving 129 reviews that required further investigation (Figure). Of the 129, a total of 27 (21%) reviewers completed a telephone interview inquiring about meals and illnesses. The median time from review date to DOHMH contact to schedule a telephone interview was 8 days. The interviews provided information on 27 restaurants, and 24 restaurants were identified as potential locations of recent exposure because the meal dates were within 4 weeks of the interview.

From the 27 interviews, DOHMH determined whether the complaints warranted an outbreak investigation by considering the following criteria: 1) more than one person became ill, 2) no other common meals were suspected, 3) ill persons lived in different households, and 4) the cases had similar onset periods (indicating a likely foodborne cause rather than person-to-person transmission). For scombroid poisoning or neurologic symptoms, DOHMH considered whether symptoms and onset were consistent with scombrotoxin, ciguatera toxin, or botulism poisoning.

Three outbreaks meeting DOHMH outbreak investigation criteria were identified, accounting for 16 illnesses not previously reported to DOHMH. Interviews with reviewers identified likely food items associated with illness at each of the three restaurants: house salad, shrimp and lobster cannelloni, and macaroni and cheese spring rolls (Table). The reviews of the three restaurants had been posted on Yelp 2–5 days after the meals. Environmental investigations were conducted at two of the three restaurants during the week after the interviews; a routine DOHMH inspection had already been conducted at the other restaurant 2 days after the meal. The two investigations and the routine inspection identified multiple violations at each of the outbreak restaurants (Table). Investigators were unable to obtain laboratory data that might have identified the infectious agents.

Discussion

In a New York City DOHMH pilot project, of 468 recent or potentially recent online foodborne illness complaints posted on Yelp and reviewed by foodborne epidemiologists, three previously unreported restaurant outbreaks were identified. Because foodborne cases have a common exposure, a restaurant patron review-based system can identify small, point-source outbreaks that are not easily found by systems reviewing large sources of data, such as syndromic surveillance of emergency department visits (4), Google Flu Trends (5), and analysis of Twitter data for influenza and other public health trends (6–8). Most importantly, foodborne epidemiologists can confirm reports because Yelp offers a way to follow-up with reviewers for interview.

In this project, only 15 (3%) of the 468 recent or potentially recent illnesses identified on Yelp were also reported directly to New York City’s nonemergency 311 service, suggesting that knowledge about 311 reporting is limited. Of further note, after messages regarding the availability of 311 were sent to 290 reviewers who did not meet the project criteria, 32 responded, of whom 25 (78%) said they were unaware of the 311 system or would keep 311 in mind for the future. The 311 service receives approximately 3,000 food poisoning complaints each year, and from that number, about 1% are identified as outbreak-related (DOHMH, unpublished data, 2014).

As social media usage continues to grow among U.S. adults (9), health departments might consider additional surveillance methods to capture illness reports from those more likely to post a restaurant review online than to contact a health department. By incorporating website review data into public health surveillance programs, health departments might find additional illnesses and improve detection of foodborne disease outbreaks in the community. Similar programs could be developed to identify other public health hazards that reviewers might describe, such as vermin in food establishments.

The findings in this report are subject to at least four limitations. First, to increase the likelihood of identifying true foodborne illness, a narrow focus was chosen for the individual criteria used to score reviews. Therefore, it is possible that some foodborne illnesses were not picked up by the screening software because of low average review scores (e.g., because of illnesses resulting from toxins with short incubation periods). Second, personal contact information for reviewers was unavailable, requiring reviewers to check their Yelp accounts and provide a telephone number to participate, which extended the time from review to interview and might have affected the response rate. Third, investigators were not able to identify any of the infectious agents in the outbreaks. Finally, the system required substantial resources; in addition to programming expertise, staff members were needed to read reviews, send e-mails, interview reviewers, and perform follow-up inspections.

Additional work using social media might improve health department abilities to use the Internet for disease detection. Working with the Chicago Department of Public Health, the Smart Chicago Collaborative recently developed a system to contact those who post foodborne illness complaints either on its website or on Twitter.* For health departments looking for an alternative to analyzing review data weekly, creating an illness-reporting vehicle such as the Utah Department of Health’s “I Got Sick” website (10) could be a more practical solution, although it might be less widely used than a review website such as Yelp. Review websites could assist by offering a link to the reviewer’s local health department’s reporting system at the time of review posting.

DOHMH plans to continue to refine this project. To shorten the time from review to investigation, Yelp will provide daily instead of weekly review feeds, and, to increase sensitivity, the project will be expanded to include additional review websites. To improve response rates, DOHMH will offer a link to an electronic survey. Finally, DOHMH is exploring the possibility of linking multiple complaints pertaining to the same restaurant, using data from different review websites and DOHMH databases.

What is already known on this topic?

Health departments rely on the public to report restaurant-related foodborne illness directly to them, yet many outbreaks go unreported. A large amount of publicly reported information about foodborne illness is available on restaurant review websites.

What is added by this report?

During a 9-month period, approximately 294,000 reviews of New York City restaurants posted on Yelp.com were screened by software programs for possible cases of foodborne illness. The software flagged 893 reviews for evaluation by an epidemiologist, resulting in the identification of 468 reviews that were consistent with recent or potentially recent foodborne illness. Only 15 (3%) of these reviews described events that had been reported to the health department. After further evaluation of reviews and interviews with 27 reviewers, three previously unreported restaurant-related outbreaks were identified.

What are the implications for public health practice?

Review websites might be a valuable source of data in the public health setting. Restaurant patron reviews can help identify small, point-source outbreaks of foodborne illness because cases have a known common exposure. Such reviews might be particularly useful if the website offers a way to reach reviewers for follow-up interviews.

barfblog

safe food from farm to fork

Tag Archives: yelp