The compensation structure a research organization or AI company uses for paying professionals who annotate datasets can highly influence their productivity, output quality, and overall satisfaction. There are two primary models for compensating workers: pay per task (PPT) and hourly payment, each presenting a handful of unique advantages and drawbacks for all stakeholders.
We obtained valuable insights on this topic by extensively researching and gathering feedback from two main groups:We engaged with individuals responsible for managing, annotating, and preparing labeled datasets for language models (also called “data workers” or “data labelers”) to understand their preferences and opinions.Additionally, we gathered feedback from researchers and AI companies, (also known as requesters), who rely on these data labeling tasks for training AI/ML models.
Our aim? To engineer a payment system that not only optimizes the performance of data workers but also champions fairness across the board.
Besides trying to figure out the optimal payment structure for high-quality data work, we also delved deep into understanding worker psychology to learn what would incentivize them to work at their best.
What is the difference between Pay Per Task (PPT) and Hourly Pay?
Pay Per Task (PPT) and hourly pay are two distinct compensation models prevalent in the realm of data labeling and AI training. Most crowd work platforms incorporate a combination of payment models, although they lean towards paying per hour in order to manage their fixed costs.
Pay Per Task (PPT): This model involves compensating workers based on the completion of individual tasks. Workers receive payment for each task accomplished, irrespective of the time taken to finish it. PPT focuses on output and deliverables, aligning compensation directly with productivity and task completion.
Hourly Pay: In contrast, hourly pay involves compensating workers based on the number of hours worked. Workers receive payment for the time spent on tasks, regardless of the number of tasks completed within that time frame. Hourly pay emphasizes time spent on work rather than output or task completion.
Both models have their own set of advantages and considerations, catering to different preferences and requirements of workers, requesters, and platforms involved in data-related tasks.
How Pay Per Task aligns worker, requester, and platform interests
Unlike hourly pay, PPT aligns incentives by focusing on productivity and output rather than the hours spent on a task. This correlation between payment and task completion ensures that workers are rewarded based on their efficiency and the quality of their deliverables, fostering a mutually beneficial relationship between workers striving for excellence and requesters seeking high-quality outcomes. This alignment of interests under PPT creates a collaborative environment where both parties are motivated to achieve optimal results.
Let’s dig deeper into the advantages a pay-per-task compensation structure offers for both requesters and labelers:
Advantages for Requesters
Attracting Premier Talent: PPT entices top-tier talent by rewarding productivity over time spent. This alignment of interests fosters excellence and efficiency, benefiting both the requester and the worker.
Cost-Efficiency: Paying per task often leads to lower overall project costs as downtime or inefficiencies aren't factored into payment. A clear budget upfront ensures potential savings compared to hourly rates.
Minute-Rounding Elimination: Minute rounding on tasks is avoided, ensuring payment only for the actual work done. This particularly benefits short, repetitive tasks, preventing unnecessary cost escalation.
Prevention of “Sandbagging”: Task-based payment mitigates sandbagging tendencies, or the act of purposely working slower to “game the clock.” Workers are incentivized to complete tasks promptly rather than prolonging them for higher hourly earnings.
Simplified Project Budgeting: Setting a flat rate for each task enables precise project budgeting, simplifying cost forecasting and management without hourly rate fluctuations.
Accommodation of Urgent Projects: Adjusting payments for urgent tasks is easier with task-based compensation, ensuring swift turnaround times without renegotiating hourly rates.
Advantages for Workers
Leveraging Proficiency for Higher Earnings: Skilled workers can maximize their talents, earning more for tasks they excel at, fostering satisfaction and engagement.
Simplified Workflow: PPT projects eliminate the need for time tracking or additional addons, streamlining the work process for data workers.
Fairness for Efficient Workers: Quicker workers are fairly compensated for their efficiency, promoting fairness and encouraging optimized performance.
Understanding worker preferences
Our conclusion regarding the preference for payment methods among experienced data workers is rooted in comprehensive research and insights gathered from seasoned labelers. These professionals encompass a spectrum of expertise, including PhD holders, language specialists, researchers, and individuals possessing specialized knowledge in diverse fields.
This is a subset of people among the data worker pool we refer to as “expert workers.”
What constitutes an “expert worker”? They do not necessarily need to have experience with labeling data; an expert would be an individual who possess deep subject matter expertise for the type of data they are labeling, strong aptitude, a high skill level, attention to detail, and fast learning capabilities.
Contrary to prevailing notions supported by published research suggesting a preference for hourly compensation among labelers in general, our survey of expert workers reveals a strong inclination towards being remunerated per task. The crucial distinction lies in the proficiency of these expert workers, who demonstrate the ability to complete tasks swiftly and effectively without compromising on quality.
The rationale behind their preference for task-based payment is evident: expert workers often feel disadvantaged by an hourly rate structure. Despite their significantly higher productivity—sometimes exceeding that of an average worker by a considerable margin—their accuracy and quality remain consistently high. Their expertise and efficiency enables them to deliver a substantially higher output, yet an hourly rate fails to adequately compensate them for their superior contributions.
Is hourly pay always a no-go?
Not necessarily. While pay per task (PPT) presents numerous advantages, there are specific scenarios where hourly pay stands out as a more practical choice.
Hourly pay becomes advantageous in batches of tasks that are complex, unpredictable, or continuously evolving, making it challenging to quantify work into distinct tasks for PPT. In instances where project scopes fluctuate or tasks require frequent adjustments, tracking time spent becomes more efficient than delineating specific tasks.
Additionally, during initial learning phases or collaborative projects where there is a lot of back-and-forth between labelers and requesters, hourly pay offers flexibility and fairness. Some clients or contracts may also specify hourly payment preferences, ensuring alignment with contractual obligations or regulatory requirements.
That being said, having a forced hourly timeframe for tasks is tasks are almost always suboptimal, both in terms of data quality and worker productivity.
For example, if a requester wants 10 images annotated in an hour and an expert finishes that batch of tasks in 30 minutes, they are unable to access the next batch until 60 minutes have elapsed. As a result, expert labelers would have to slow down or spend time idle, which would lead to frustration and disincentivize them over the long run. Additionally, this would also lead to worse results for requesters, as their projects would take longer to complete.
Tracking hours spent on tasks via self-reporting could be a viable alternative, although requesters might be reluctant due to concerns about potential over-reporting. The idea here is to establish an anticipated completion time for tasks, potentially allowing experts to finish assignments more quickly. For instance, if a batch of 1000 images is expected to be annotated in 40 hours and experts complete it in 20 hours, they effectively earn double the average worker's pay.
In summary, while pay per task excels in many aspects, hourly pay prevails in situations involving intricacies, evolving scopes, collaborative efforts, client preferences, or contractual obligations, providing a pragmatic alternative in these scenarios.
Closing thoughts
In conclusion, the advantages of pay per task for data workers and requesters are multifold. It promotes efficiency, fairness, quality output, and competitive advantage, making it an indispensable choice in the landscape of data work compensation models. Why does this topic matter to us? It's quite simple: we're dedicated to building an industry leading talent-first data annotation platform. We are well aware of the current landscape in this industry and are committed to constructing a solution that incentivizes high-quality work and enables worker agency. Our entire approach revolves around a research-supported methodology—from project setup to worker compensation and custom systems for bespoke projects.
Need help with data annotation or AI training? Reach out to discover how we can help you obtain premium-quality data for your projects. Alternatively, If you are an expert in your field and want to join our network of elite labelers, let's collaborate! Drop us a line for potential partnership opportunities.
