Hi friends,
Three weeks ago we mapped how AI companies get human voices and video, and one line in that issue said expertise is moving “to judging, not just doing”; this issue is about that part of the market: the doctors, lawyers, analysts, engineers and scientists who are paid to write and grade material for AI models.
Here is what we think the public pages add up to. They advertise a price list of ceilings. Where one platform, Outlier, gives totals, they work out to about $556 for each person counted. And the decisions we can see (which expertise is wanted, what counts as good work, how many hours there are) sit with whoever writes the task (the lab’s buyer, the vendor that runs the project, a benchmark’s authors and funders), not with the expert. Most of the money does appear to reach experts as a group: the two press reports we found put contractors’ share of vendor revenue at 60 to 70%.
The issue is built from pages read on November 18, 2026: the public pages of eight platforms and vendors that work with experts, job postings from three AI labs, the papers or project pages of four benchmarks (the public tests used to score AI models), and for the longer sections the help pages of four platforms, the papers or pages of five more benchmarks, and press reports, with figures marked official, reported or our calculation.
Rates are posted rates, not what anyone was actually paid. None of the companies named here sponsored this issue or saw it before publication.
The Glasp Newsletter goes to 550K+ subscribers. Sponsor an issue.
If you want to reread or highlight this newsletter, save it to Glasp.
What experts are asked to do
Start with the work, because it shows where the standard sits: in most of the kinds below, someone other than the expert has already defined the task, and the expert supplies the answer key, the rubric or the verdict. The listings describe six kinds of work.
1. Writing hard questions with a reference answer. Humanity’s Last Exam is the public example: its project page gives 2,500 questions written by “nearly 1,000 subject expert contributors affiliated with over 500 institutions across 50 countries”. A Mercor listing for employment lawyers asked for the same thing in a profession: “Drafting a motion or a brief from a fact pattern, plus the reference answer a model’s attempt gets graded against.” The page now says that listing “is no longer accepting applications” and shows Mercor’s general legal network in its place, which is open and carries the same text (seen in a browser on November 18, 2026).
2. Grading against a rubric. The HealthBench benchmark, according to its paper’s abstract, has 5,000 conversations, scored with rubrics “created by 262 physicians” that hold “48,562 unique rubric criteria”. The expert is not writing the answer but the list of what a good answer must contain.
3. Checking reasoning step by step. xAI’s own posting for an applied math tutor: “Evaluate model outputs, independently verify mathematical solutions, identify subtle errors in reasoning”.
4. Red teaming. Trying to make a model fail or misbehave: Alignerr lists an “AI Red Team Tester” role on its own page.
5. Designing tasks for agents. Agents are models that carry out multi-step work with tools. Agents’ Last Exam, from Berkeley, says in Berkeley RDI’s announcement that each task comes from a real project a human expert completed, across 55 occupations. Terminal-Bench-Science, in its call for contributions, asked for “complex, real-world computational workflows from practicing scientists”.
6. Case review in medicine and law. Mindrift lists, on its own page, a project for licensed US physicians who review how a model reasons through clinical cases. The Mercor listing described grading model-drafted contract and discovery work: “whether the provision does what it claims, whether the privilege call is right, whether the citation supports the proposition it is offered for.”
The chain: lab, vendor, expert
The chain shows who decides what gets made and who handles the money before it reaches an expert. Three parties sit between a research plan and a paid hour.
The model developer decides what data it needs. The labs now hire people whose job is to buy it. Anthropic has its own posting for a “Data Operations Manager, Human Data” who will “Drive strategic vendor partnerships”, with a posted salary of $270,000 to $365,000. OpenAI’s “Research Program Manager, Human Data Campaigns” is described as “a key interface between our research roadmap, external vendors, AI trainers, and the Human Data engineering team” in a copy on an investor job board, dated July 22, 2026; we could not open the original on openai.com.
The vendor or expert network finds the people, tests them, runs the project and pays them. The sizes these companies state are large. They are totals in the companies’ own words, and none of the pages says how many people are working this week. Outlier says “900K+” and “$500M+” paid to experts. Handshake says “750k+ students, professionals, and experts”. CrowdGen, Appen’s platform, says “500,000+” contributors paid. Alignerr says “100,000+”. All five figures are from the companies’ own pages.
The expert is a contractor, paid by the hour or by the task.
None of these pages says what the vendor keeps. We found two reported figures, neither of them official. Dealroom reported on April 14, 2026, citing The Information, that at Handshake “Contractor costs run at 60%–70% of gross revenue”. Dealroom reported on July 6, 2026, citing a person with knowledge of the company’s financials, that Mercor “pays out 60% to 70% of its topline revenue to contractors”. Our calculation, as an illustration only: if that share held for a single hour, an expert paid $100 would be billed to the buyer at about $143 to $167. We found no lab or vendor page with a billing rate for expert work, so we cannot check it.
Who commissions it, and what changed in twelve months
The past year’s news shows the same thing as the postings: the decisions and the growth we can see sit with the buyer and the vendor.
The buying job is written down, the budget is not. The Anthropic posting quoted above also says, in the company’s own words, what the data is for: “RLHF, safety, tool use, and agentic workflows”. RLHF is reinforcement learning from human feedback, where people’s judgments steer a model. In its text as we read it, we did not find a named expert network or a budget figure. A second OpenAI role, “Data Program Manager, Human Data”, in a copy on the same job board, which is marked “Posted 6+ months ago”, says its team “turns human feedback into reliable signals for training and evaluation”. What the labs spend on human data is not on any page we read.
One lab’s shift was reported in its own words. TechCrunch reported on September 13, 2025, citing Business Insider, that xAI “laid off 500 team members” from its data annotation team. This is a press report, not an official statement. The internal emails, as quoted there, spoke of “scaling back our focus on general AI tutor roles”. According to the same report, the company said in a public post that it “will immediately surge our Specialist AI tutor team by 10x”. We read these words in TechCrunch, not in the post itself. It is one company, and a plan, not a count of people hired. What it shows for our argument is who chooses which expertise is wanted: the buyer.
Another network neared $1 billion, gross. Handshake’s AI training business was reported, in the April Dealroom article linked above, which cites The Information, at “nearly $1B” in gross annualized revenue in April 2026, up from $550 million in January, with close to $300 million left after paying contractors. The same report says three other companies had already reached that mark.
One vendor, three sets of numbers. These are the clearest case of how far a network total can sit from a count of people working.
Official: on November 10, 2026, Mercor announced on its blog “our $750 million Series D valuing Mercor at $20 billion”. The round “included an employee tender”, which means some of the money bought shares from staff.
Reported, not official: by the Dealroom article of July 6, 2026, linked above, Mercor “hit over $2 billion in gross annualized revenue in June 2026”, which was “double its pace at the start of the year”. Annualized means a short period scaled up to a year. Gross means before contractors are paid.
Head count, three ways: the Series D post, Mercor’s own, says “more than five million experts in our network”. Mercor’s own blog index carries an undated news blurb about “a network of more than 100,000 experts”. The same Dealroom article reports that the company “now pays over $2 million a day to more than 30,000 weekly active contractors”. The three figures count different things: people “in our network”, people described in a news item, and people paid in a week. The last is the only weekly figure we found for any vendor, and it is reported, not official.
The Dealroom article also reports that Mercor raised $350 million at a $10 billion valuation in October 2025. Set beside the official $20 billion, the valuation doubled in about thirteen months (our comparison of one reported and one official figure).
Posted pay, by field
This list is the price list the argument starts from, and its largest figures are ceilings. All rates below are in US dollars per hour, from the company’s own page, read on November 18, 2026. “Up to” figures are ceilings. Several cells are a single listing seen that day, not a platform-wide rate.
Medicine: Mercor: one Otolaryngologist (ENT) listing at $150. Mindrift: one project “Up to $100/hour”. DataAnnotation: up to $125+. CrowdGen: Medical Clinicians “Up to $450/hr”
Law: Mercor: “$55–$150+/hour”; US contract attorney roles $100 to $125. DataAnnotation: up to $125+. CrowdGen: Legal Experts “Up to $400/hr”
Finance: Handshake: FP&A Analyst “Up to $80/hr”. Mercor: one FP&A Expert listing at $100. CrowdGen: FP&A Analysts “Up to $180/hr”, Accounting Experts “Up to $160/hr”
Software: Mercor listings: $50 to $80 (backend engineer), $90 to $120 (MLOps), $150 (open source maintainer). CrowdGen and Handshake: up to $150. DataAnnotation: up to $150+
Science and engineering: xAI: applied math tutor $45 to $75 (US rate). Mercor listings: engineering and simulation experts $70 to $90, economics research $150 to $160
Languages: DataAnnotation: language specialists up to $40. Alignerr: one Portuguese reviewer role $15 to $35. CrowdGen: Linguists “Up to $95/hr”
No specialty: Handshake: AI Evaluation Specialist “Up to $17/hr”. DataAnnotation: generalist up to $50
Mindrift posts tiers without naming a field: “Entry-level: $15-30 per hour” and “Domain experts: $60-100+/hr”.
One thing can be read straight off the list. What needs arithmetic is in the next section.
The headline numbers are ceilings. CrowdGen attaches the note “Pay rate is based on location and skills.” to its top pay figure. A ceiling is the most a listing says it can pay. It is not a count of hours paid at that rate.
What the pages’ own numbers imply
This section does what the pages do not: it sets their own figures beside each other. Every input is stated in the sentence that uses it. The arithmetic is ours.
1. A large total is a small average. Outlier’s own page says “900K+” and “$500M+” paid to experts. $500 million divided by 900,000 people is about $556 a person, over whatever period the two totals cover (our calculation). Both totals carry a plus sign, so the true ratio could sit on either side. The page describes the 900K+ as “working and earning as AI trainers”, and we did not find on it a separate count of how many were ever paid, or the period either total covers. As arithmetic, $556 is about 14 hours at $40 an hour, or under 6 hours at $100 an hour (our calculation: 556 / 40 = 13.9; 556 / 100 = 5.6). There are two honest readings. Either most people in the total earned little, or most never worked and a smaller group earned much more. We could not tell which from the page. Neither is the picture a ceiling gives.
2. The one weekly figure is modest too. Mercor, by the Dealroom report quoted above, “now pays over $2 million a day to more than 30,000 weekly active contractors”; that is reported, not official. $2 million a day for seven days is $14 million a week, and $14 million divided by 30,000 is about $467 per active contractor per week (our calculation; if only five days a week are paid, about $333). At $100 an hour that is under 5 hours a week, and at $50 an hour about 9 (our calculation: 467 / 100 = 4.7; 467 / 50 = 9.3). Both inputs are “over” and “more than”, and this is an average, so some people work far more and many work less. The same report’s 60 to 70% payout share would give a higher figure, about $770 to $900 a week (our calculation: 60 to 70% of $2 billion is $1,200 million to $1,400 million a year, divided by 52 weeks and by 30,000), so the two reported figures do not agree, as point 6 shows. Even the higher figure is about 9 hours a week at $100 an hour. Our October 26 issue said “A sign-up is not hours” as an expectation; these two results are a first rough number for it, and the second says the same of people counted as active.
3. The one posted average equals the midpoint of the posted range. Alignerr posts $40 to $120 for five of the six roles its page shows, and an “Average pay” of $80. Alignerr does not say how its average is worked out. $80 is exactly the midpoint of $40 and $120 (our calculation: (40 + 120) / 2 = 80). We cannot tell whether it is a measured average of what people were paid or the middle of the range, and we did not find a method on the page.
4. The platform moves the figure too, not only the field. The highest medicine ceiling in our list, CrowdGen’s $450, is 9 times DataAnnotation’s generalist ceiling of $50 (our calculation: 450 / 50). That looks like the price of a medical license. But the two figures come from different platforms. Hold the platform fixed, and on DataAnnotation medical work is posted at up to $125+ against up to $50 for a generalist, 2.5 times or somewhat more, since the plus sign leaves the top unstated (our calculation: 125 / 50). Other platforms show a wider gap between fields: on Handshake, up to $150 for a software engineer against up to $17 for an AI Evaluation Specialist, 8.8 times (our calculation: 150 / 17). Hold the field fixed, and the ceilings for medical work run from $100 at Mindrift to $450 at CrowdGen, 4.5 times (our calculation: 450 / 100). Finance shows it with nearly the same title. An FP&A role is posted at up to $80 on one platform, $100 on another and up to $180 on a third. That is a 2.25 times spread for nearly the same title (our calculation, from the three posted figures). The pages do not say whether the tasks are the same.
So which platform posts the role moves the figure by a factor of two or more. A ceiling describes the platform as well as the field.
5. The buyer’s desk is the steadier contract. Anthropic’s own posting gives $270,000 to $365,000. Taken as a yearly salary over a 2,000-hour year (50 weeks of 40 hours, our assumption), that is $135 to about $183 an hour (our calculation: 270,000 / 2,000 = 135; 365,000 / 2,000 = 182.5), salaried and year-round. Several posted figures in our list sit above the bottom of that band ($150, $160 and $180), and two sit above its top, CrowdGen’s ceilings of $450 and $400. On the posted figures alone, an expert with a full year of hours at those rates would match the buyer. The larger difference is the unit. A salary covers every working hour of the year. An expert’s ceiling is per hour of task work, and what we found stated about hours were limits and waits (a weekly cap, a match that could come within a week or take several months), not a minimum. An expert paid $100 an hour would need 2,700 paid hours to match the bottom of the band, 700 more than that full-time year; at $150 an hour, 1,800 hours (our calculation: 270,000 / 100 and 270,000 / 150). This says nothing about what Anthropic pays experts, which no page we read states.
6. The vendor’s share is reported at 30 to 40%, and one report does not add up. We have two reported figures, both published by Dealroom: contractor costs of “60%–70% of gross revenue” at Handshake (April 14, 2026) and 60% to 70% of topline revenue paid out to contractors at Mercor (July 6, 2026). If contractors take 60 to 70% of gross, 30 to 40% stays with the vendor before its other costs (our calculation: 100 - 70 and 100 - 60). One figure fits, from the same Handshake report: Handshake’s “nearly $1B” gross with close to $300 million left after contractors is about 30% (our calculation: 300 / 1,000, both inputs approximate). One pair of figures, both from the Mercor report, does not fit on its face: “over $2 million a day” for 365 days is about $730 million a year, about 37% of “over $2 billion” in gross annualized revenue, not 60 to 70% (our calculation: 730 / 2,000). A 60 to 70% share of $2 billion would be about $3.3 million to $3.8 million a day (our calculation: 1,200 / 365 and 1,400 / 365, in millions). The word “over” may be hiding a lot, or the figures cover different things. We cannot tell, so treat the share as rough.
Where the evidence does not fit. Three things in this issue do not fit the argument above. First, Mercor’s reported daily pay, taken at face value, gives about 37% of gross (point 6), not the 60 to 70% the same report states; “over” may hide a lot, or the figures measure different things. Second, some posted ceilings exceed the lab buyer’s hourly equivalent of $135 to about $183 (point 5): $450 and $400 sit above the top of that band, and three more posted figures ($150, $160 and $180) sit above its bottom. Third, on benchmarks, experts themselves write the standard, as the benchmark section below shows; what we did not find there is a fee amount.
One correction to our own earlier wording follows from this. Our October 26 issue called judging work “already the best-paid work in this market”, and the rates it cited for credentialed work were advertised top rates. On this issue’s evidence we would say it has the highest posted ceilings. What it pays is not on any page we read.
Getting in, and how quality is checked
The steps between applying and being paid show who carries the risk: the expert pays the entry cost in time and personal data, and the platform sets the test, the cap, the acceptance rule and the exit. The sources in this section are the companies’ own pages, or a project’s own paper, unless we say otherwise.
Identity. Mercor’s contributor docs describe ID verification as “A secure step in the application process, completed with a third-party verification provider”, using a government ID and a selfie. A background check by an outside firm follows, and “Completion is required before starting work on any project.” DataAnnotation’s FAQ says it stores biometric data “for purposes of identity verification only.”
Tests, and the test may be one attempt. Mercor’s listing: “a short AI interview, about 20 minutes total.” Outlier: “skill screenings” and “identity verification”, with onboarding that “typically takes 30 to 90 minutes”. DataAnnotation’s FAQ: “Most Starter Assessments take about an hour to complete”. Then: “You can only take the Starter Assessment once.” Mercor’s page on assessments says “Most assessments permit only a single submission.” It adds that scoring and weighting may vary. On these two pages (DataAnnotation’s FAQ and Mercor’s page on assessments) we did not find a statement that the first test is paid.
Note what passing gets you. On the Mercor listing, it put you in a pool. Projects came later, and the page said a match could come within a week or take several months.
We found one clearly paid entry step. Mercor calls it a work trial: “Work trials are single, remunerated assessments designed to evaluate talent before an offer is issued.” A typical one “requires approximately four to six hours for completion.” The pay is a fixed amount, not an hourly rate, and we did not find the amount on that page.
Per hour and per task put the risk in different places. Outlier’s FAQ says it pays “for completed work, either hourly or per task”. Mindrift’s FAQ describes the second: “Each task has a clearly defined, fixed reward that is visible before work begins”, and “Payment is provided only for tasks that are successfully completed and accepted”. Under that model, a rejected task costs the expert the time. Mercor tracks hours with a time-tracking tool, and its pay policy says the tool “may take periodic screenshots to confirm you are working.” Its glossary defines a weekly cap as “The maximum number of hours you’re allowed to work (and be paid for) per week on a project.” So the posted rate is one of two numbers. The other is the cap.
One benchmark paper shows the layers end to end. The authors of GDPval, a set of work tasks from 44 occupations, required “a minimum of 4 years of professional experience”. Experts had “to pass a video interview, a background check, a training and a quiz”, and “Less than 10% of applicants were selected”. Then came review: “Each task received an average of five human reviews (with a minimum of three reviews).” Grading was by “blinded expert pairwise comparisons”, and “On average, grading each comparison for the gold subset took over an hour.” The expert’s judgment is itself judged, by a process the expert does not run. On pay, the one sentence we found in the paper is: “Experts were well compensated for their time and experience.” The process is described step by step. The pay gets one sentence, with no amount.
Low quality can end access. DataAnnotation’s FAQ: “subpar or problematic contributions may result in loss of access to future projects.” Mercor’s offboarding page names “Substandard work or the necessity for multiple revisions” as a warning sign, and says that if work does not improve, “the project team reserves the right to terminate your contract.” We did not find an appeal process described on that page.
Confidentiality, and your own file. The work belongs to the client. Mercor requires a signed agreement that, in its words, “protects the client’s IP and data.” Its conduct rules prohibit “Mentioning clients by name or any client-specific work” and “Reusing work produced for Mercor or its clients elsewhere”. On the expert’s own file, Mercor says “your data is never used to train third-party large language models”. It also says: “Deleting your account does not remove data from companies that may have already accessed it.”
Vetting became a risk for the expert. Staffing Industry Analysts reported that Mercor acknowledged a data breach and said it was “one of thousands” of companies hit by a supply chain attack, and that by April 13, 2026, workers had filed six proposed class actions. One complaint, as reported there, said the platform collected “Social Security numbers, birth dates, passports, face and voice biometric data” and recorded video interviews. These are allegations in a complaint, not findings. We could not open the court dockets, so this is the state as of April 13, 2026, not today. Either way, the tests that prove you are an expert create a file about you.
The benchmark side: a second market for expertise
Benchmarks are where the standard for a field gets written down, and they complicate our argument in a useful way. Here experts do write the standard. But on the pages we read we found authorship and acknowledgements and no fee amounts, and in one case the lab that funded the test holds part of it.
A public test needs the same people a vendor does. FrontierMath, a mathematics benchmark published by Epoch AI, was written, its paper says, “with over 60 mathematicians from universities across more than a dozen countries”. Each problem went through “blind peer review by at least one expert mathematician”. The paper says “Solving a typical problem requires multiple hours of effort from a researcher”. We did not find in the paper what the authors received. It does thank OpenAI “for their support in creating the benchmark”, and Epoch’s page says: “FrontierMath was developed with funding from OpenAI, who has exclusive access to a subset of the benchmark.” A public test can also be a funded one.
Law, software and medicine have their own, and the pay is not on the page. LegalBench, according to its paper’s abstract, has “162 tasks covering six different types of legal reasoning”, “designed and hand-crafted by legal professionals”. Its arXiv page lists 40 authors, and we did not find a mention of payment on it. SWE-bench Verified is described on its project page as “A human-validated subset of 500 SWE-bench instances”, “created in collaboration with OpenAI”. The page says “Human annotators reviewed each instance to ensure the problem descriptions are clear.” We did not find on the project page who the annotators were or what they received. We did not find whether the physicians behind HealthBench were paid.
Whole jobs, not questions. Agents’ Last Exam (June 2026) reports more than 1,500 tasks “built with 300+ experts from 100+ institutions” in Berkeley’s announcement; the paper, submitted June 3, says “1K+ tasks” and “250+ industry experts”. Both are the project’s own sources.
How the scores have moved. The FrontierMath paper, first submitted in November 2024, said: “Current state-of-the-art AI models solve under 2% of problems”. Epoch’s page for that first version showed one model at 38% when we read it on November 18, 2026. We did not find a date for that score, and we did not see the full leaderboard. The GDPval paper, from October 2025, says “the current best frontier models are approaching industry experts in deliverable quality”. The Remote Labor Index paper, also from October 2025, put its best agent at “an automation rate of 2.5%” in its abstract. Our reading: a test that models start to pass has to be replaced by a harder one, and the people who decide when, and what the next one asks, are the test’s owners.
Expert-written does not mean error-free. On June 12, 2026 Epoch released a second version of FrontierMath, which, by Epoch’s own page, “addressed errors in 42% of problems”. Checking is part of the work, and it was still going on more than a year after launch (our calculation, from the two dates).
How to read these numbers
Official, reported, our calculation. Official means the company’s or project’s own page or paper. Reported means a press article. Our calculation means arithmetic we did on those figures.
Ceiling versus rate. “Up to $180” is the most one listing says it can pay. It is not what a typical hour pays.
Totals versus now. “900K+” and “$500M+” are totals in a company’s own words. They do not say how many people had work this month.
Listings close, and reopen under another name. The employment law listing we quoted no longer accepts applications; its page now sends lawyers to a general legal network that is open. A closed listing tells you nothing about the next one.
Scores have dates and versions. A score on version one of a test is not a score on version two.
What none of these pages can tell you, and what we could not confirm
Our argument is built from public pages, so it stops where they stop. Five gaps matter most to someone deciding whether to sign up.
Posted versus paid: what anyone was actually paid. Every rate here is a posted range or a ceiling. It matters because your decision turns on the hour you will actually be paid for, and no page we read shows one.
Registered versus active: how many experts are working. The platform figures are the companies’ own totals, with no count of people active now. The one weekly figure we found is a press report about Mercor. It matters because the size of a network tells you how many people were counted, not how likely you are to be given a project.
Ceilings versus distributions. The pages gave us ceilings, ranges and one unexplained average. It matters because a ceiling is the best case for one listing, and what you can plan on is where most paid hours fall.
What the vendor bills the lab: what buyers pay vendors. The only margin figures we found are two press reports about two companies. It matters because without the billing rate you cannot know what share of the price of your hour you are being offered.
How many hours a credentialed expert actually gets. We found caps and waiting times, and no page with hours per person. It matters because income is the rate times the hours, and the pages publish only the first.
We also could not confirm:
How often double review is used. Snorkel lists “Provenance and adjudication” as a step in building a dataset, and Mindrift says each contributor is “part of a team supervised by a Quality Assurance specialist”, but we found no page that says what share of work gets a second reviewer. The GDPval paper gives review counts for its own tasks only.
Where the Mercor cases stand. Our source is one article from April 13, 2026.
Reasoning traces as a product. We found no public page with a price for an expert’s own written reasoning.
Change over time. We read these pages on one day, so we cannot say rates rose or fell this year.
What the labs spend. No posting or official page we read gives a budget or a volume for human data.
Whether training and onboarding time is paid. We did not find it stated on the Outlier FAQ, the Mindrift FAQ or the Handshake AI page.
Who was paid for benchmark work. We did not find payment amounts, prizes or co-authorship terms on the pages we opened for FrontierMath, LegalBench, SWE-bench Verified or GDPval. The Remote Labor Index page gives project costs that it says “come directly from human professionals who completed the work”; we did not find on it what the benchmark paid.
Current scores. We could not read a dated, current score for the second version of FrontierMath or for the Remote Labor Index, so none is given.
What we think happens next
This section is our view, not a finding. Each point says what in this issue it rests on, and what would change our mind.
Does it continue? We think yes, but the work keeps moving up, and each kind of task has a short life. What it rests on: our reading in the benchmark section, that a test models start to pass has to be replaced. FrontierMath went from “under 2% of problems” in its first paper to one model at 38% on the page we read. And one company was reported to lay off 500 people from its annotation team while saying it would grow its specialist team “by 10x”. Our judgement: the demand is for whatever a model cannot yet be graded on without a person. So the work moves from labelling, to writing tasks, to grading them, to writing the test itself. It does not end as long as someone has to decide what a good answer is. What would change our mind: a lab saying in public that it has stopped buying expert judgement, or expert listings disappearing from these platforms while general listings stay.
Can many people live on it? We doubt it. What it rests on: the one platform that gives totals works out to about $556 per person counted (our calculation, above). The largest posted figures are ceilings. On one Mercor listing, passing the test put you in a pool, with a match that could come within a week or take several months. Our judgement: this stays side income for most people, and a real income for a small group whose field is wanted this quarter. The steadier job is on the other side of the table: the lab role that buys this data is posted at $270,000 to $365,000 a year. What would change our mind: any platform publishing how many people it paid last month and the median amount.
Do new fields open? We think three do, and they open late.
Fields where the standard itself is argued over. We think a rubric works when experts agree on what a good answer contains. HealthBench has 48,562 rubric criteria written by 262 physicians. Where two good experts disagree, or where the right answer changes with the country or the court, a rubric is harder to write. Our judgement: those fields need people for longer.
A profession together with a language. On the pages we read, language alone mostly posts lower: up to $40 on one platform, and $15 to $35 for one reviewer role (one platform posts linguists at up to $95). A law license posts at “$55–$150+/hour” on one platform and up to $400 on another. Our judgement: a license plus a second language is the scarce pair.
Checking a model after it is in use. GDPval’s authors report that, for its gold subset, grading one comparison “took over an hour” on average. A dataset is bought once. Checking what a deployed system did has to be done again each time the system changes. This one is our judgement only: nothing we read prices it.
The case against all of this. Labs could bring the work in-house; they already hire people to manage it. Models could grade models once a rubric exists, which would leave the expert writing rubrics and little else. And the risk to experts is real: by April 13, 2026, workers were reported to have filed six proposed class actions against Mercor after a data breach. If the first two happen, we think this market is a phase, not a profession.
What this means if you have expertise
If the argument holds, the questions worth asking are about hours, acceptance and ownership, not about the ceiling.
Read the range, not the ceiling. Ask what the rate is for your first project, and whether it is per hour or per accepted task.
Ask about hours before rate. The weekly cap, and how many hours people on the project were paid for last week.
Passing the test is not a job. Ask how many people in the pool were given work last month.
On these pages a license posts higher than a language. If you have both, look for listings that ask for both. Then compare platforms, because the platform moved the figure by a factor of two or more.
Ask what the platform keeps about you. Which documents, any recorded interview, for how long, and how to have it deleted.
Some routes offer credit, not pay. Agents’ Last Exam invites contributors “to join as co-authors” and its page does not mention payment. Decide which one you want before you start.
Your best material is your own finished work. Some requests are for real tasks with a checkable result. Before you hand one over, make sure it is yours to give and contains nothing a client or employer owns.
Best,
Kei & Kazuki
Would you like to take Glasp on the go?
With the Glasp mobile app, you can highlight and organize your favorite content anytime, anywhere. Stay productive on the move and never miss an insightful quote.
Partner With Glasp
We currently offer newsletter sponsorships. If you have a product, event, or service you’d like to share with our community of learning enthusiasts, sponsor an edition of our newsletter to reach engaged readers.







If you have done expert work for an AI lab or one of these platforms, what did the posted rate turn into once hours and accepted tasks were counted? I'd love to hear it in the replies.
The first part of this map, on voices and video:
https://read.glasp.co/p/ai-data-market-map-how-ai-companies-get-human-voices-and-video?utm_source=substack
This is a valuable and timely analysis of the growing expert-data economy behind AI development. What stands out most is the gap between attractive advertised hourly rates and the actual earning opportunities experts can expect. A high rate means little without consistent projects, transparent payment structures, and predictable working hours. I also find the discussion of who defines quality particularly important: experts contribute their knowledge, but platforms and buyers often retain control over evaluation criteria and access to work. Greater transparency around actual earnings, project allocation, and compensation would make this ecosystem fairer and more sustainable for everyone involved.