I’ve come across a fascinating little site named “Review Skeptic”. There they explain …
Review Skeptic is based on research at Cornell University that uses machine learning to identify fake hotel reviews with nearly 90% accuracy.
So the idea here is that you can cut and paste any review in and then click “Test It”, and the machine analysis will crunch and advise if it is fake or real.
This has so much potential value, especially if it has a 90% accuracy rate, and so I would truly love to see something like this in so many places, for example TripAdviser or even Amazon.com because we all know that much of what we get presented with as a supposedly “independent honest” review is simply marketing hype that has been entered via a sock-puppet account.
What exactly do they mean by Fake?
I tried ReviewSkeptic and simply made up a few, but it then marked them as real, and yet they claimed …
In a test on 800 reviews of Chicago hotels, a computer was able to pick out deceptive reviews with almost 90 percent accuracy.
… so I do wonder what exactly it is fishing for.
As I wrote my fake reviews, I confess that I did have in mind real stays at real hotels, so marking them as “real” is perhaps correct because I was writing it as a real review.
What it goes after is specifically the stuff that is marketing hype designed to look real, and so when the owner writes, the way they would compose a review is distinctly different than a real reviewer … “The Hotel had amenities like a fully stocked mini-bar, such luxury” … writes nobody ever, but the owner would. As they conducted their tests, they did also discover that humans were not very good at spotting the fake reviews, and that the machine algorithm was far better at doing that.
Truthful hotel reviews, for example, are more likely to use concrete words relating to the hotel, like “bathroom,” “check-in” or “price.” Deceivers write more about things that set the scene, like “vacation,” “business trip” or “my husband.” Truth-tellers and deceivers also differ in the use of keywords referring to human behavior and personal life, and sometimes in features like the amount of punctuation or frequency of “large words.” In parallel with previous analysis of imaginative vs. informative writing, deceivers use more verbs and truth-tellers use more nouns.
Using these approaches, the researchers trained a computer on a subset of true and false reviews, then tested it against the rest of the database. The best results, they found, came from combining keyword analysis with the ways certain words are combined in pairs. Adding these two scores identified deceptive reviews with 89.8 percent accuracy.
Lets Test this
Below are two reviews – one of these is real and the other is deceptive and yet both appear to the human reader to be very similar … can you, as a human, spot which is which?
I have stayed at many hotels traveling for both business and pleasure and I can honestly stay that The James is tops. The service at the hotel is first class. The rooms are modern and very comfortable. The location is perfect within walking distance to all of the great sights and restaurants. Highly recommend to both business travellers and couples.
My husband and I stayed at the James Chicago Hotel for our anniversary. This place is fantastic! We knew as soon as we arrived we made the right choice! The rooms are BEAUTIFUL and the staff very attentive and wonderful!! The area of the hotel is great, since I love to shop I couldn’t ask for more!! We will defiantly be back to Chicago and we will for sure be back to the James Chicago
If you then test them in review skeptic, you will see which is which.
The Full paper
A rather detailed technical paper can be found here.
Just Hotel Reviews?
Sadly yes.
Personally I would love to see the scope expanded to include all reviews … Books, Products on Amazon, Movies, Restaurants, etc…
However, I also suspect that if something like this was in place more formally, then the deceptive reviewers would adapt and craft their fake reviews to pass the test.
My Most Favourite Review
Now for a quick bit of fun.
When looking for somewhere close to Manchester Airport last year, I came across Pymgate lodge. Now to be strictly fair, they are under new management today and so this no longer applies, but here from the old regime is a true gem of a review, and yes, this is very real, I checked it with Skeptic Review, and no I did not actually stay there. During that period there were rather a lot of bad reviews, but this just was truly one of the best. The review below turned out to be rather helpful in motivating me to cross Pymgate off my list. I did ask a few locals I was working with and the look of pure horror on their faces that I had even considered it told me that I had indeed had a very close shave.
The Pymgate Inn… aaah, where to begin?!
There’s really nothing quite like feeling
like a colossal burden immediately
upon your arrival.
Bumping into a Manc Andy Pipkin,
complete with his welcoming grunt,
immediately endeared us to the
frankly unparalleled hospitality of
this gem of a hostelry.
Having to round Manc Andy, as he
vacuumed the foyer with a Hoover fit
for the Museum of History, proved
difficult in itself as he seemingly didn’t
give a rat’s by not moving.
We naively confused M’Andy with the
cleaner… how wrong we were as he
actually proved to be half of the crack
married proprietor unit… and about as welcoming as the dinner remains
adorning his shirt.
Our reservation was categorically
cocked up, resulting in us having to
fork out the sum total of 120 quid for
the four of us to stay the night.
The high tech HB pencil and dog-eared
notepad method was clearly right and
our internet booking similarly wrong.
The welcoming hum of damp added
some mystique as we climbed the
stairs and were shown our rooms by
the delightful Waynetta of Heald Green.
The room was coated in a fully matured
layer of dust and the carpet boasted
that well worn, filthy studenty, ‘Young
Ones’-esque lair feel… eerily and
remarkably authentic.
Dust-riddled and chipped china was
generously provided as was the touching and very thoughtful blood on the shower tiles.
There are few luxuries that can compare
to a wooden bog seat ensemble dating
back to Roman occupancy… an unmitigated joy for the weary and hygienic traveler.
After swerving using the bathroom, fully
spacious enough in which to swing around an adult gerbil, we attempted to get some kip.
To their credit, we scoured for bedbugs
and found none. Even these most unchoosy of filth dwellers passed on the free colony on offer.
Major league kudos to either Manc Andy or Waynetta for the novel idea of not changing the bedding and my daughter revelling in the ‘beach away from home’ experience by way of the sand in her bed. We’ve no idea how they knew how much she does love her flip flops.
Why bother to attend a spa or a
massage therapist after a long journey
when an array of random and archaic
bed springs gently caress you into painless sweet slumber?
Said slumber is prematurely arrested
when the telly sees fit to turn itself on
at precisely 3:00am… a unique touch
that we were told occurs in all of the
Pymgate’s suites.
Upon removing all belongings from
anything that kept them off the floor,
we decided against any possible
infection by avoiding the bathroom
murder scene altogether and proceeded to head down for our included Continental Brekkie.
We were overwhelmed by the choice
of mini Kellog’s cereal boxes and
lovingly mismatched cutlery. Toast was
duly ordered by three whilst I stupidly
braved the Full English.
Stone cold upon arrival, I was past
caring and cautiously picked away
at the pre-prepared culinary delight.
To be fair, they went above and
beyond by generously providing a
suitably lengthy hair in my bacon.
We promptly legged it and had a wonderful stay down the way at The
Village in Cheadle.
All ’round then, an absolutey blissful experience that we shall never likely
forget.
We have now coined a new word, used heavily of late, that pertains to anything that is of the quality of our stay, ie. crap… as pym.
Eg. Those pair of keks are unequivocally pym