PDF version of this report
You must have Adobe Acrobat reader to view, save, or print PDF files. The reader
is available for free
download.
ARCHIVED REPORT:
Amazon
Web Services
Mechanical Turk
by Geoff Keston
Copyright 2015, Faulkner
Information Services. All Rights Reserved.
Docid: 00021356
Publication Date: 1505
Publication Type: PRODUCT
Preview
Amazon Mechanical Turk is a
cloud-based service that lets employers hire
workers for very small contract assignments, often called “microtasks.”
The service provides a largely automated system that helps workers find
available assignments and helps employers determine which workers are
suitable for a given task. Mechanical Turk is the leader in
the fast growing microtask market, but there are many competitors, and
Amazon’s position in the field will be continually challenged.
Report Contents:
- Description
- Related
Faulkner Reports - Vendor
- Applications
- Environment
- Support
- Pricing
- Competitors
- References
- Web Links
Description
[return to top
of this report]
Over the past few years,
a new approach to
contracting work, called “microtasking,” has emerged.
| Related Faulkner Reports |
|
Amazon CloudFront Service Product Profile |
|
AWS Infrastructure Services Product Profile |
In this model, employers
use online services to
solicit help in completing very small tasks, sometimes as simple as
counting the number of people in a photograph.
The leader in this
already crowded market is
Amazon’s Mechanical Turk, named after an 18th century device that
purported to be a chess-playing machine but actually had a person
hidden inside. Amazon’s version has not one but hundreds of thousands
of people, hiding not under a chess table, but in the cloud, where they
remain out of an employer’s sight.
Users sign up for
Mechanical Turk with their
Amazon accounts. When they join, they can search for assignments that
Mechanical Turk calls Human Intelligence Tasks (HITs). HITs are
searched by keywords and sorted by criteria such as the payment they
offer, the time allotted to complete them, and when they are set to
expire.
| Vendor |
|
Name: |
Requesters
promise a
pre-set financial reward
for completing a HIT. The rewards are typically very small, sometimes as small as $0.01. And the highest paying one available
on May 12, 2015, for instance, was
for $41.28 with 11 hours allotted to complete it.
Amazon
makes money by
charging requesters to
list HITs. The company also brokers payments between requesters and
workers.
Qualifications.
Requesters can restrict HITs to only those
workers who meet certain qualifications. Mechanical Turk has a list of
“qualification types” that are based on automatically gathered data,
such as a worker’s approval rate.
Mechanical
Turk also
lets requesters create
their own qualification types, which can then be used by other
requesters. Users must ask to be qualified for HITs of a certain type.
(Only the creator of a type can certify a worker as qualified.)
Requesters can also design tests that workers must pass. Most tests are
simple, entailing multiple choice questions, but some require essays.1
Creating and configuring qualification types demands a small bit of
programming savvy.2
Amazon
offers a category
called “Masters” that
is given to workers who, in Amazon’s words, “have demonstrated superior
performance while completing thousands of HITs across the Marketplace.”
This status is based on automatically determined Amazon criteria, not
on a judgment made by the company. Requesters can publish HITs
specifically to a Masters board.
Data
Gathering and Analysis. As workers use Mechanical Turk,
the
system records information about
their activities and maintains statistics. One of the uses to which
this automatically recorded data is put is to create “system
qualifications,” which are the qualification types Amazon automatically
assesses.
The
system manages the
transition of a HIT
through each step of its lifecycle, from when it is posted for workers
to find to when it has been completed and approved by the requester.
(For a detailed
flowchart depicting the lifecycle of a HIT, see “Overview: Lifecycle of
a HIT.”3)
Strengths
Amazon
has built a
strong reputation in the
corporate world through its cloud services, and it has garnered a large
following in the consumer market by selling books and other products.
In the microtasking market, a strong reputation is a major
strength, attracting good requesters and lots of workers.
Amazon
also has many
other revenue streams, so
Mechanical Turk will not face the same make-or-break pressure as other
microtask services.
Limitations
A
paper by faculty
members at New York
University’s Stern School of Business observed the following about
Mechanical Turk and quality:
[S]ince
manually
verifying the quality of the
submitted results is hard, malicious workers often take advantage of
the verification difficulty and submit answers of low quality.
Currently, most requesters rely on redundancy to identify the correct
answers. However, redundancy is not a panacea. Massive redundancy is
expensive, increasing significantly the cost of crowdsourced solutions.
Therefore, we need techniques that will accurately estimate the quality
of the workers, allowing for the rejection and blocking of the
low-performing workers and spammers.4
The researchers
recommend that these
limitations be overcome by “redundancy (hiring several workers to do
the same job and comparing their results) or through use of ‘gold data’
— questions to which employers already know the answer,
randomly
inserted as a test of worker competence.”5
Furthermore, Mechanical Turk’s
apparent promise of user
anonymity may not be fulfilled. An academic study published in 2013
found that personally identifying information could be uncovered
even if users intended to be anonymous.6
One
potentially growing problem for Mechanical Turk is that some of
its workers are organizing to fight for better conditions. These
workers, collectively called “Dynamo,” have written letters to Amazon and sought to raise awareness in
the media.7 They argue that Mechanical
Turk
should not emphasize the availability of low-cost labor but should
instead present the service as providing access to a “skilled,
flexible” workforce.8 They also want
Amazon to do more to prevent workers from being cheated by job
requesters, which is the type of dispute that the company does not
involve itself in now. Dynamo
is currently small
— there were 533 members of the community as of May 2015 — but such
efforts might one day pressure Amazon to alter Mechanical Turk’s
business model. And the movement might be a sign that there is not an
unlimited pool of workers who are willing to work within the current
system.
Applications
[return
to top
of this
report]
Common
uses of
Mechanical Turk include the
following:
- Categorization
– Putting items into
a category, such as putting a pictured object into one of a handful of
classifications in a sales catalog. - Digitization
– Creating a digital file from an analog source, such as handwritten notes. - Moderation
– Reviewing user-generated content submitted to
a Web site to determine whether it meets defined standards. - Sentiment
Analysis – Reviewing blog posts,
social
networking comments, and other sources to learn what users are saying
about a particular product or topic.
The
uses to which
enterprises and other large
organizations put Mechanical Turk will play a major role in its
future. There is already some stigma attached to the service, however.
“Spamming and fake reviewing can be easily commissioned,” wrote
economist Nancy Folbre in the New York Times.9
And raising
a darker possibility, Jonathan Zittrain, a specialist in Internet
issues at Harvard Law School, has speculated about possibilities such
as the Iranian government using the service to easily and cheaply
identify protesters from photographs.10
But
some large (and more
highly regarded)
organizations are using the service in different ways.
For instance,
the US Defense Advanced Research Projects Agency (DARPA) used the
service to translate Arabic social media posts into English, with the
aim of completing a project faster and for a lower cost than by using
professional translators.11 Another
example of a model business use comes from the Food and Drug
Administration’s hiring of the firm Captricity, which uses Mechanical Turk to help
convert hardcopy
forms to electronic files.12 And
to cite another
example, Twitter uses
human-based Mechanical Turk
in combination with automated tools: When a term is found through an
automated process to be “trending,” humans are used to determine the
term’s meeting, such as distinguishing between a reference to “Eagles”
as a football team or as a type of bird.13
For some
applications, it remains an open question whether
Mechanical Turk is a useful tool. A prime example is scientific
research. Mechanical Turk is being used to find human subjects, who can
easily and inexpensively be recruited online to do things like complete
a survey. In the research community, opinions vary about whether
Mechanical Turk provides good results.
Thomas Leeper, a postdoctorate in political science, and Kevin J.
Mullinix, a Ph.D. candidate in political science, analyzed various ways
in which human subjects are recruited, and they concluded that the
results of
research studies were “comparable” whether Mechanical Turk was used or
whether participants were found through other means.14 But other researchers argue that
Mechanical
Turk has created a pool of subjects who have participated in so many
studies that they have learned to see through questions and now respond
differently than do ordinary subjects.15 It may take time for researchers
to settle
questions about whether crowdsourced research subjects are good to use,
and they may develop ways to overcome any shortcomings that this new
approach may have.
Environment
[return
to top
of this
report]
Workers
and requesters
alike use Mechanical Turk
through a browser interface, and the storing of data and management of
settings are cloud-based. The service is available only to registered
users, and users and requesters need password-protected accounts.
The
future of this model
and the prospects for
its success may depend greatly on how gathered data is
analyzed. One of the challenges of making this model applicable to more
sophisticated applications will be breaking down complex tasks into
units of the size that Mechanical Turk naturally handles.
Support
[return
to top
of this
report]
Amazon publishes
detailed online guides for
requesters and for workers. It also offers advice about how to take
full advantage of the service.16
Users can submit
questions to a support message board or send an email directly to
Amazon.
In
addition to the
aforementioned free services,
premium support that provides direct access to Amazon technical
personnel is available to requesters. Pricing for premium support is
done at a flat rate or as a percentage of the requester’s Amazon
Web Services bill.17
Pricing
[return
to top
of this
report]
Joining
the site as a
worker or requester is
free. Requesters then set their own price for each HIT, and Amazon
takes 10-percent of each HIT’s payout.
Competitors
[return
to top
of this report]
Mechanical
Turk is
part of a large trend of Web-based services that offer work that can be
done from home with flexible hours. The service is less notable for its
technology — which at its heart consists more or less of job board
postings, online payments, and basic data analysis — than for its
vision of what employment can be in the Internet age. Amazon’s size and
marketing muscle put the company in a strong position to lead this
segment, but the model is replicable. Just as Facebook supplanted
MySpace, an alternative could take Amazon’s leadership position. There
are already many competitors, and even if they do not become as popular
as Mechanical Turk, they could consume a large share of the overall
market.
Many
of these
services focus on what, in the words of Mechanical Turk competitor
CrowdFlower, are “enormous, simple, repetitive, data-centric projects.”
Other members of this niche include Clickworker, CloudFactory, and
Microtask, all of which focus on a narrow set of services that conform
to CrowdFlower’s definition. These services are difficult to
distinguish. Their success is likely to be determined not by bold
technical innovation but by fine tuning the look, feel, and features of
the service to meet customer needs. The differences between services
may be small, but small differences can significantly affect
convenience.
Some
competitors are
taking different approaches, however. For instance, Crowdmug
specializes in one type of service, letting users request that a worker
take a picture of a certain place at a defined time, and Samasource is a
non-profit that aims to help
poor women and young people find
jobs. Others sites are
working to build better working conditions: MobileWorks aims to reform
the microtasking market to pay better and to establish more personal
bonds between employers and workers.18
The development of
these and other alternative models could cause some workers to turn
away from Mechanical Turk, whose tasks are often criticized as being
too simplistic and unsatisfying. Speaking of her experience using
Mechanical Turk, Web content editor Jennifer Davis says, “I looked at a
few different pictures of the same image and identified the
differences. I don’t remember what else I did because I only did a
couple of tasks, which totaled together amounted to less than a dollar.”19
Davis then switched to using Textbroker. As with using Mechanical Turk,
she typically does not know who she is working for, but the pay is
better and the jobs are more interesting. “Textbroker allows me to
write a variety of things, mainly advertorial-type things designed to
sell products but I also learn new things in the process. Some writers
consider it mindless and an affront to real writing, and I can see that
argument, but it pays the bills for now so I’m satisfied.”
The fate of these
services will also be influenced by broader trends. There are
thoughtful, serious-minded analysts who believe that the nature of work
has fundamentally changed in the Internet age. For instance, popular
business writer Seth Godin says that traditional full-time jobs are
now, and probably always will be, less common than they once were
because “[t]he internet has squeezed inefficiencies out of many
systems, and the ability to move work around, coordinate activity and
digitize data all combine to eliminate a wide swath of the jobs the
industrial age created.”20
And in an op-ed, computer
scientist Jaron Lanier writes that “[t]here used to be a path that led
to a reasonably reliable, reasonably secure middle class,” but online
services that solicit free or low cost contributions have been designed
in a way that blocks that path.21
If Godin and Lanier are
right, Mechanical Turk could, for better or worse, both follow and push
along this trend.
References
[return to top
of this report]
1
Sharon Chiarella. “Amazon Mechanical Turk Crowdsourcing
Marketplace for Work.” The Cloud Show. Aug 21, 2012.
2 To
see the types of programming that are required, refer to
“QualificationRequirement.” Amazon. August 2014.
3
Amazon. “Overview: Lifecycle of a HIT.” The Mechanical Turk Blog. April
11, 2011.
4 Panagiotis
G. Ipeirotis, Foster Provost, and Jing Wang.
“Quality Management on Amazon Mechanical Turk.” Proceedings of the
Second
Human Computation Workshop. July 2010. 64-67.
5 Nancy Folbre. “The Unregulated Work of
Mechanical Turk.” The New York
Times. March 18, 2013.
6 Matthew Lease, Jessica Hullman, Jeffrey P.
Bigham, Michael
S. Bernstein, Juho Kim, Walter Lasecki, Saeideh Bakhshi, Tanushree
Mitra, Robert C. Miller. “Mechanical Turk Is Not Anonymous.” March 6, 2013.
7 James Vincent. “Amazon’s Mechanical Turkers
Want to be Recognized as ‘Actual Human Beings.'” The Verge. December 4,
2014.
8 Dynamo. “Email Campaign to Jeff Bezos.”
wearedynamo.org. Cited May 13, 2015.
9 Nancy
Folbre. “The Unregulated Work of Mechanical Turk.” The New York
Times. March 18, 2013.
10 Jonathan Zittrain. “Mechanical Turk and the
Danger of Digital Sweatshops.” FORA.tv. January 5, 2010.
11 Amazon.
“How DARPA Uses Mechanical Turk to Translate Social Media.” The
Mechanical Turk Blog. May 5, 2012.
12 Neal Ungerleider. “FDA Adopts Amazon
Mechanical Turk for Drug Safety Backlog.” Fast Company. November
14, 2013.
13
Tcarmody. “Twitter Just Told us How Cool its Real-Time
Search Is…and How it Makes
its Money.” The Verge. January 8, 2013.
14 Thomas Leeper and Kevin
Mullinix. “What If You Had Done Things
Differently? Testing the Generalizability of Framing Effects with
Parallel Experiments.” Published online at:
https://dl.dropboxusercontent.com/u/414906/ParallelExperimentsOnFraming.pdf.
February 24, 2015.
15
Douglas Perry. “How Amazon’s Crowdsourcing Forum, Mechanical Turk,
Could be Destroying Academic Research.” The Oregonian. February 17,
2015.
16 See,
for instance
Amazon Mechanical Turk. “Requester Best Practices Guide”
Amazon Mechanical Turk. “Choosing the Right Tool.”
17
Amazon. “AWS Support Pricing.”
18 Rip Empson. “Backed by
Andreessen, Virtual Workforce
MobileWorks Completes 1M Tasks for Startups in 1st Year.” TechCrunch.
August 14, 2012.
19 Jennifer
Davis. Private Communication. 2013.
20 Seth Godin. “The Forever Recession (and the Coming
Revolution).”
Seth Godin Blog. September 29, 2011.
21 Jaron Lanier. “Own the Future!” NY Post. May
2013.
Web
Links
[return to top
of this report]
Amazon Mechanical
Turk:
https://www.mturk.com/mturk/
Clickworker: http://clickworker.com/
CloudFactory: http://cloudfactory.com/
CrowdFlower: http://crowdflower.com/
Crowdmug: http://www.crowdmug.com/
Dynamo: http://wearedynamo.org/
Microtask: http://microtask.com/
MobileWorks: https://www.mobileworks.com/
RapidWorkers:
http://rapidworkers.com/
Samasource: http://samasource.org/
Textbroker: http://www.textbroker.com/
US Defense Advanced Research Projects Agency (DARPA):
http://www.darpa.mil/
About the Author
[return to top
of this report]
Geoff Keston is the
author of more than 250 articles that help organizations find
opportunities in business trends and technology. He also works directly
with clients to develop communications strategies that improve
processes and customer relationships. Mr. Keston has worked as a
project manager for a major technology consulting and services company
and is a Microsoft Certified Systems Engineer and a Certified Novell
Administrator.
[return to top
of this report]