# Designs for a future crowd work market

M. Six Silberman

## Introduction

A. What is this about?

B. This is about two things. First, it is about crowd work. Specifically, it is about microtasking. More specifically still, it is about what a future microtask market might look like. Second, it is about computing, social value, and social science.

A. Let's start with crowd work. What is crowd work? What is microtasking? What is a microtask market? And why does this matter to anyone?

B. 'Crowd work' is a term that points to at least three things. These things are related but not the same. First, crowd work is an industry. That is, it is a specific set of people, relationships, organizations, artifacts, and practices. Second, crowd work is a way to get things done. That is, it is a set of ideas or techniques. Third, crowd work is an active research area in computing.

The crowd work industry is made up of companies like Amazon, oDesk, Microtask, CrowdFlower, and 99designs, the people who work for them, the technologies they build and run, and the discourse they share through blogs, formal publications, and events like CrowdConf.

Crowd work as a way to get things done is made up of a set of ideas and techniques. We can sketch this way of doing things with its language -- the nouns and verbs that populate the discourse of its practitioners. The central verb in the world of crowd work is 'crowdsource'. Crowd employers 'crowdsource' work 'to' workers 'over' an online 'platform'. There are different ways to crowdsource work. The most well-known are 'contest-based crowdsourcing' and 'microtasking'. Contest-based crowdsourcing involves holding a contest around a task. Many workers submit entries to the contest, but only one or a few winners are paid. This model is popular in graphic design. The most well-known contest-based crowdsourcing platform is 99designs. Microtasking involves cutting work into small pieces and giving each piece to a worker. Two pieces of related or adjacent work might be given to the same worker, or to two workers on opposite sides of the world. These workers might never talk to each other. In fact, they might not know of each other's existence, except in the sense that each knows they are part of a large 'crowd' working on a job. Take for example the task of entering handwritten data from a scanned form into a database. A crowd employer might turn each field into a separate image, then turn the transcription of each field into a separate task. A worker in India might transcribe the 'last name' field; a worker in Kenya the 'first name'; and a worker in the United States the date of birth. Three microtasking platforms are Amazon Mechanical Turk, or AMT; Microtask; and MobileWorks. Some large technology companies run their own 'internal' microtask platforms. Microsoft's, for example, is called Universal Human Relevance System, or UHRS. There are different kinds of microtasking platforms. For example, AMT is a market, while MobileWorks assigns tasks algorithmically to workers. In this text I focus on microtasking, and specifically on microtask markets.

The central generic nouns in the world of microtask markets are 'requester', 'worker', 'platform', 'task', 'price', and 'time'. The central verbs are 'post', 'do', 'accept', and 'reject'. Requesters post tasks to a platform. Workers choose tasks. Foremost among the factors weighed by workers in choosing tasks is a task's price and how long it will take. After a worker does a task, the requester can 'accept' or 'reject' it. If the task is rejected, the worker is not paid.

Crowd work is an area of active research in at least three fields of computing research: human computation, human-computer interaction, and computer supported cooperative work. Human computation is an approach to solving computational problems by incorporating human input into an algorithmic process. The field draws on a long tradition of organizing humans to do computation: before computers were silicon, they were human (e.g., Light 1999, Grier 2005). But in its current form, human computation grew out of artificial intelligence research in the middle of the first decade of the 2000s. On one hand, the most rigorous interpretations of "human computation" (e.g., Law and von Ahn 2011, pp. 3-5) do not include all examples of crowdsourcing. On the other, some interpretations are much broader, including all of crowdsourcing and a wide range of other phenomena (e.g., Michelucci, ed., 2013). But human computation as a subfield of computer science research and body of knowledge contributes significantly to crowd work and crowdsourcing practice. The full name, for example, of the main human computation conference -- "HCOMP" -- is "Conference on Human Computation & Crowdsourcing". If human computation as a subfield of computer science is "about" one thing in particular, it is crowdsourcing.

Human-computer interaction, or HCI, is an interdisciplinary field that emerged in the 1970s and 1980s. Its early focus was on practical problems such as designing aircraft cockpits to reduce (sometimes deadly) pilot error. Early HCI theory drew on computer science and psychology. The problems it addressed usually concerned one person interacting with one computer. In the 1980s and 1990s, the focus of the field shifted on expanding access to computing, mainly by increasing its usability. One result of this shift was the development of the graphical user interface. Later developments include mobile computing (i.e., smartphones), interactive displays, and tabletop computing. By the mid 2000s, the effort to expand access to computing by increasing its usability had largely succeeded. By this time, a more broadly scoped strain of HCI research had developed. Researchers in this "third wave" base their work in frameworks from the social and cognitive sciences such as activity theory, phenomenology, and distributed cognition (e.g., Harrison, Tatar, and Sengers, 2007; Kaptelinin and Nardi 2012, pp. 1-2). And by the late 2000s, HCI researchers had begun to examine some of the unexpected, and perhaps unfortunate, consequences of the field's success. These researchers began to discuss new directions for the field in the context of the global information society the field had itself helped bring about. These discussions have led to a proliferation of new concepts, theories, and methods in the field.

Computer supported cooperative work, or CSCW, grew out of HCI in the early 1990s. It was started by HCI researchers and computing practitioners making software for groups and organizations. CSCW has thus had a longer and more rigorous alliance with theory and method from the social sciences than HCI. But as HCI has broadened its scope, some researchers and research funders have seen fit to include both fields in a new category, "human-centered computing", or HCC.

People, papers, ideas, money, and specific working technologies travel easily between HCI and CSCW, and sometimes between HCI, CSCW, and human computation. Researchers in the three fields can often be found working in the same lab, department, or school. Yet they have their own emphases, bodies of knowledge, and, to some extent, values and cultures. They overlap and influence one another, but are distinct. As the recent re-categorization suggests, the overlap between HCI and CSCW is greater than the overlap between "human-centered computing" (i.e., HCI and CSCW together) and human computation.

Just as human computation, HCI, and CSCW are related and overlapping but not identical, the crowd work industry, crowd work as a set of ideas, and crowd work research are related and overlapping but not identical. People, specific working technologies, ideas, money, and practices do circulate between these realms. But industry and research have, at least ostensibly, their own rules. And "crowd work" as a model has been applied beyond both -- for example, in the nonprofit sector.

Whether you think crowd work matters and is worth thinking and talking seriously about may depend on how popular you think it will get. One reasonable view was expressed by a well-known crowd worker in Amazon's Mechanical Turk system (AMT) in 2010. An interviewer asked her: "What do you think about the role of crowdsourcing in the future of employment?" She said:

> I see it as fairly insignificant. I don't think most work can be done this way, and I think the work that can be completed is, for the most part, too cheap for most to bother wanting to do it. (spamgirl 2010)

Considering crowd work from "inside" AMT in 2010, this view is easy to understand. It was, and still is, easy for requesters to make task design mistakes. It was, and still is, hard to "learn the ropes" as a worker. It was, and still is, hard for workers and requesters to communicate about work. It was, and still is, hard for workers to find well-paying work. And it was, and still is, hard for requesters to find skilled workers and figure out how much to pay them.

But some of these problems are specific to AMT. None need be insurmountable. And, crucially, to the extent that they are general problems, many technologists in research and industry are hard at work solving them. The potential financial rewards are large. Organizations of all kinds see crowdsourcing as a way to cut costs on existing processes and cheaply expand operational capabilities. Researchers, especially in the social sciences, see it as a cheap way to get data. And the flexibility of crowd work appeals to many workers (e.g., Martin, Hanrahan, O'Neill, and Gupta 2014). Technologists and business people call crowdsourcing the "future of labor". New crowdsourcing startups appear with high frequency. And the volume and scope of crowd work research is growing. Major HCI and CSCW conferences devote multiple sessions to the topic, and it has a new conference of its own. From within this context, Kittur, Nickerson, Bernstein, Gerber, Shaw, Zimmerman, Lease, and Horton (2013) write:

> While not all jobs are amenable to being sent down a wire, there are portions of almost any job that can be performed by the crowd. We foresee a world in which crowd work continues to expand...

More recently still, Cefkin, Anya, Dill, Moore, Stucky, and Omokaro (2014) write:

> Businesses increasingly accomplish work through innovative sourcing models that leverage the crowd... It is our position that crowdwork is likely to be increasingly integrated into existing ways of doing organizational work.

All that is needed for crowd work to keep growing is for it to keep making, or saving, employers money. It is doing so now. And given the practical and intellectual effort going into crowd work, it seems likely to do so for some time. But this growth says little about workers' experience. Indeed, the lived experiences reported by crowd workers do not match up to the breathless excitement of technologists, managers, and pundits hailing crowd work as the future of labor. Organizational incentives are such that technologists are often allocated to make sure workers can work effectively. But often, "effectively" means "just effectively enough".

So why does crowd work matter? In short, because crowd work is likely to grow. As it grows, it will play a bigger role in the lives of a growing number of people. It will do this largely by playing a bigger role in work arrangements in many fields. This growth will create both new opportunities and new expectations. It will 'disrupt' existing business strategies and livelihood strategies for both organizations -- potential crowd employers -- and employees -- potential crowd workers. The number of people who make a living or second job through crowd work will grow.

There are at least two big open questions about this change. The first is the distribution of benefits. If current trends hold, the distribution of benefits will be starkly uneven. At present, crowd employers benefit tremendously from greatly reduced costs and expanded operational capabilities compared to traditional employment arrangements. But these benefits are secured, to a large extent, at workers' expense. This is not to say that workers do not value the unique opportunity crowd work affords. They do. Specifically, they value the flexibility it affords -- the opportunity to work from home, on their own hours. But this flexibility has a cost (e.g., Silberman, Ross, Irani, and Tomlinson 2010). Crowd work often pays less than similar work within a traditional employment relation. Legally, crowd workers are "independent contractors". As such, they are not entitled to minimum wage, group health insurance, or overtime pay. And crowd work pay is uncertain. In a microtask market such as AMT, a requester can reject work at any time, for any or no reason. Workers are not paid for rejected work. And they have few clues about how likely this is to happen for a given task. The ability of requesters to refuse to pay for submitted work -- work they may keep and use -- with impunity has adverse consequences for both workers and requesters. For example, it makes workers' livelihoods vulnerable to predictable technical or administrative errors without encouraging employers to take reasonable steps to prevent them. This risk incentivizes workers to spend less time on any given task. Less time on task often means inferior work. Large amounts of inferior work means employers must develop sophisticated quality control strategies. This raises the effective cost of crowdsourcing. But this cost does not benefit anyone: it is just lost time. And it contributes to a common view among employers and researchers of crowd workers as low-skill, interchangeable, and untrustworthy. Aggressive quality control strategies lead workers to see many employers as miserly, nitpicking, or even cheating. Even from an entirely financial perspective, this adversarial environment can be shown to be inferior to more cooperative arrangements that are easily imagined. Indeed this and many other adverse consequences of the current approach to microtask market design are practical, quantifiable, explicable, and even to some extent predictable.

The second big open question about crowd work is harder to address quantitatively. The philosopher Donna Haraway (Nakamura 2003) asks it pithily: "What kind of world is this?" Put as a question to computing researchers, crowd employers, and crowd work platform operators, the question is: What kind of world are we building? To ask this question is not to argue that workers do not value crowd work. They do. But if we pay close attention to their discourse (e.g., spamgirl 2010; Martin, Hanrahan, O'Neill, and Gupta 2014), we see that they participate in it much in the same way that I would agree to give someone $50 for use of a ladder after they had pushed me into a deep hole. That is, workers seem to participate in crowd work not with the excitement of participating in a gleaming new technological future, but wearily, with resigned acceptance. There is no straightforward or universally unobjectionable way to talk about this. But it seems irresponsible to ignore it.

At stake, ultimately, is the future of crowd work -- and perhaps the future of work. Like Kittur et al. (2013), I expect crowd work to continue to grow. Employers and technologists will respond to the great institutional, economic, and cultural incentives for technological and organizational innovations that drive down labor costs and expand operational capabilities. Workers will respond to the opportunity to work in a time and place of their choice. In this context, crowd work arrangements that support viable livelihoods could indeed unlock "an incredible number of opportunities for careers in skilled work" (Kittur et al. 2013, p. 1301). Such growth could contribute to socioeconomic recovery in the recession-hit North, sustainable socioeconomic development in the South, and socioeconomically beneficial globalization. Human computation and HCC research has already begun to explore the potential benefits -- for employers and workers -- of bringing crowd work to "developing regions", "the bottom of the pyramid", and "low-income workers" (e.g., Khanna et al. 2010; Narula et al. 2011; Gawade et al. 2012; Gupta et al. 2012; Samdaria et al. 2012). Yet Kittur et al.'s pessimistic scenario, in which crowd work falls "into an intellectual framing focused on low-cost results and exploitative labor", with workers assumed "interchangeable and untrustworthy, [with] low or static skill sets and strong motivations to shirk" (2013, p. 1301), is easier to imagine -- in part because it is already realized in AMT, the dominant microtask market.

Put shortly, we are on a bad road. This text is about how to get off it.

A. Fine. What about computing, social value, and social science?

[Talk about this.]

[In computing, better technology == social value.]

[In industry, more money == social value. This is because business is shaped by neoclassical economics and so is computing discourse on value, ethics, and people. If somebody is using it, it must be because it makes them better off. If two people engage freely in a transaction, it must be because it makes them both better off. If the parties to the transaction are not being visibly coerced, they are transacting freely.]

[Sociology and contemporary economics both understand that there are many complex and invisible forms of coercion. This "unfreedom" should trouble our simplistic models of how computing is valuable.]

[We need new models.]

A. Fine. You said this text is about how to get off the bad road we are on in crowd work. Can you explain the goal of the text more precisely?

[Explain goals. First, to show how many workers' problems in AMT are practical, quantifiable, explicable, and predictable. Second, to actionably articulate another model. Third, to use crowd work as a case to show how contemporary social science can improve design and help computing researchers and the computing industry produce more lasting and substantive value for a broader range of stakeholders than it does now.]

A. How do you do this in the text?

[With agent-based modeling.]

A. What is the contribution to research?

[We have not talked much about the _sources_ of workers' problems in the current microtask market paradigm. We have also not talked much about their implications for future crowd work markets. And we have talked very little, in practical terms, about how to address them. This text brings the large-scale, long-term socioeconomic stakes of these problems into full view. It examines the sources of these problems at three levels: technical, institutional, and intellectual or ideological. It proposes new models of crowd work (i.e., it addresses the intellectual gap), new institutions, and concrete plans for new software systems that would address these problems. In doing so, it offers a rigorous yet clearly design-actionable theoretical framework that connects computing to contemporary social science and shows the potential practical benefit of social science theory (not just method) for computing research and practice.]

A. What is the structure of the text?

[The text proceeds in five sections. First, I discuss crowd work, its socioeconomic context, and the stakes in the future of microtask market design. Second, I build a series of agent-based models to explore the dynamics of an abstract microtask market based loosely on AMT. In doing so, I introduce three concepts from contemporary social science: power, fairness, and governance. Third, I offer a series of designs for a future microtask market. I express these designs as agent-based models and examine the advantages and problems of each. Finally, I discuss the relationship between computing research, computing practice, social value, and social science. We have not really taken social science seriously in computing research and practice, and it is starting to give us real problems. Taking social science seriously poses difficulties for computing research and practice, but it is possible. In this section I discuss possible strategies and benefits. I conclude with practical implications for crowd work research in human computation and human-centered computing.]

## References

Cefkin, M., O. Anya, S. Dill, R. Moore, S. Stucky, O. Omokaro. 2014. Back to the future of organizational work: crowdsourcing and digital work marketplaces. _CSCW '14 Companion_: 313-316.

Grier, D. A. 2007. _When Computers Were Human_. Princeton University Press.

Harrison, S., D. Tatar, P. Sengers. 2007. The three paradigms of HCI. _Proc. alt.chi '07_.

Kaptelinin, V., B. Nardi. 2012. _Activity Theory in HCI: Fundamentals and Reflections_. Morgan & Claypool.

Kittur, A., J. V. Nickerson, M. S. Bernstein, E. M. Gerber, A. Shaw, J. Zimmerman, M. Lease, J. J. Horton. 2013. The future of crowd work. _Proc. CSCW '13_: 1301-1318.

Law, E., L. von Ahn. 2011. _Human Computation_. Morgan & Claypool.

Light, J. S. 1999. When computers were women. _Technology and Culture_ **40**(3): 455-483.

Martin, D., B. V. Hanrahan, J. O'Neill, N. Gupta. Being a Turker. _Proc. CSCW '14_: 224-235.

Michelucci, P., ed. 2013. _Handbook of Human Computation_. Springer.

Nakamura, L. 2003. [Prospects for a materialist informatics: an interview with Donna Haraway](http://www.electronicbookreview.com/thread/technocapitalism/interview). _Electronic Book Review_, 30 Aug 2003.

Silberman, M. S., J. Ross, L. Irani, B. Tomlinson. Sellers' problems in human computation markets. _Proc. HCOMP '10_: 18-21.

<code>spamgirl</code>. 2010. [Spamgirl speaks! An interview (part 3 of 3)](http://brokenturk.blogspot.com/2010/11/spamgirl-speaks-interview-part-3-of-3.html). _Broken Turk_, Nov 2010.