How to Measure AI Productivity Gains in a B2B Agency

TLDR: Measure AI productivity at the task level, not by counting outputs or tool logins. Compare similar client work with and without AI, include setup, review, correction, and approval time, then judge the result on quality-adjusted throughput, contribution margin, usable capacity, and client outcomes. Segment the results by task and team experience before deciding whether to scale, redesign, or stop the workflow.

Define productivity as quality-adjusted client work

Productivity in an agency is accepted client work produced with a known amount of labor, cost, and rework.

Logins, prompt counts, and drafts generated describe activity. They show that people are using a tool, not that the agency delivered more finished work or kept more margin from it. A team can draft twice as fast and still lose ground if senior review, correction, and client revisions grow to fill the gap.

The core unit is completed client work that meets the agency’s acceptance standard, per labor hour. Write that standard before the test starts.

For a monthly paid-search report, it might require correct data, a stated recommendation, and no factual edits from the account lead. Fix it in advance so nobody adjusts it after seeing the result.

Name the decision before collecting data. The test exists to choose one action for one AI-assisted workflow: scale it, redesign it, or stop it. Without that decision, measurement turns into reporting.

Published research supports measuring several things together. The UK AI Security Institute ran a randomized controlled pilot with 500 Prolific participants.

On average across the four tasks, treatment participants scored 19% more points, took 25% less time, and produced 61% more points per minute. Points per minute is a quality-adjusted throughput measure, meaning speed and quality combined in one figure.

The planning task showed no statistically significant uplift on any metric, so one task’s result should not be generalized across an agency.

Build a baseline for one repeatable agency task

Start with one recurring deliverable, such as a paid-search report, an SEO brief, or a campaign QA review, rather than the whole agency. If the task is not chosen yet, choose business tasks to automate first, then return here to build the baseline.

Choose a comparable unit of work

A baseline only works when the work being compared is similar. Before changing anything, record these fields for each unit of work:

1. Scope and complexity tier.
2. The operator and their experience level.
3. Elapsed time from assignment to delivery.
4. Labor time, counted in hours worked.
5. Number of revisions requested.
6. The acceptance criteria that apply.

Keep elapsed time and labor time separate. A report that waits two days for client approval has a long cycle but no extra production labor, and an AI test should not be blamed or credited for that wait.

Record the full human workflow

IBM’s guidance on measuring the productivity of people using AI lists five lessons: think contextually, measure key metrics against a control group, engage users with varying skill levels, measure adaptation, and consider the project’s full lifecycle.

For an agency, the lifecycle means counting setup, context gathering, production, review, correction, client approval, and delivery. In the AI-assisted version, add prompt preparation and any time spent checking output against source data.

This matters because time estimates drawn only from an AI conversation miss what happens afterward.

Anthropic states that its conversation-based method cannot see activity after the user finishes interacting with the model and may overstate productivity effects because it does not account for human work beyond the conversation. A baseline that tracks every stage avoids that blind spot.

Track a four-layer AI productivity scorecard

This four-layer scorecard asks four questions in order: is the team using the workflow, is accepted work getting out faster, is it good enough, and does the change improve margin, usable capacity, or a client outcome?

The corresponding layers are adoption and utilization, time and throughput, quality and rework, and economics and client outcomes. None should be read alone.

The AISI pilot measured quality, time, and points per minute together, which supports the same principle: a single metric can mislead. The agency must still define its own acceptance rubric, because the pilot’s scoring was built for its own tasks and not for client deliverables.

Why output quality is not productivity

On an in-house B2B SaaS content team, I built a custom Copilot agent for blog drafting. A senior copywriter repeatedly described its drafts as about 90% of finished quality. But I still approved or edited every outline, the copywriter edited the draft, product marketing tightened the messaging, and three of us approved every draft. I did not measure the new end-to-end labor time, so that quality judgment did not establish hours saved, ROI, or an agency-client result.

That experience is the measurement problem in miniature. Output quality can justify continued testing, but productivity remains unknown until the complete human workflow is timed and the accepted deliverable is counted.

Adoption and utilization

Adoption asks whether the team is using the workflow as designed. Track the share of eligible tasks run through the AI-assisted process and note where operators skip steps.

Treat these figures as diagnostic inputs. Low usage can explain a weak result, but high usage proves nothing about value.

Time and throughput

Measure labor time and cycle time separately. AI can shorten production while adding review time or approval delay, so a faster draft may not mean faster delivery.

Record hours by stage, then calculate accepted deliverables per labor hour. Report both labor time and cycle time so the team can see where the hours moved.

Counting drafts instead of accepted work measures activity, not finished client work.

Quality and rework

Use a stable rubric with a small number of criteria, scored the same way in both workflows. Track first-pass acceptance, revision count, defect severity, and client-requested rework.

Where practical, have reviewers score without knowing which workflow produced the work, because that knowledge can bias judgment. Human judgment stays inside the measurement.

How to use AI as a co-thinker rather than a blind automator is a workflow-design question. This scorecard checks whether that design held up.

Economics and client outcomes

Translate the observed change into contribution margin, usable capacity, delivery reliability, or a client outcome tied to the task, such as fewer reporting errors reaching the client.

Choose the translation before the test. Revenue belongs in the result only if the test actually measured it, and a single-task test may not.

Run a comparable two-to-four-week test

A short test can support a decision for one workflow. It cannot forecast the whole agency.

Run it for two to four weeks, long enough to capture normal variation in comparable deliverables and short enough that tools and scope stay stable. Low-volume teams should extend the window rather than pad the sample with dissimilar work.

Set a minimum number of comparable deliverables per workflow before the test starts, and extend the window until each workflow reaches it. If the team cannot reach that minimum, report the result as directional rather than decisive. A small internal test remains decision evidence for that workflow. It is not a statistically generalizable agency benchmark.

Pair similar work with and without AI

IBM recommends measuring key metrics against a control group, and the AISI pilot compared treatment participants with a control group. An agency may lack the volume or independence for a formal randomized trial, so use the closest practical version:

1. Match work units by scope tier and complexity, then assign one of each pair to the existing workflow and one to the AI-assisted workflow.
2. Alternatively, rotate the same trained operators between the two workflows week by week, so personal skill does not explain the gap.
3. Keep the earlier baseline as a reference, not the only comparison, because conditions drift.

Do not leave assignment to operator preference. An operator who chooses may give the AI version the easier work.

When staffing permits, include at least one newer and one experienced operator in both workflow conditions. If only one operator is available, report a workflow result and make no experience-level claim.

Control the variables that distort the result

Hold four things steady for the whole window: scope definition, the quality threshold, the measurement method, and the observation window.

Do not swap tools, models, or prompts mid-test. If a newer option appears during the window, schedule it as the next test instead of changing this one.

Capture exceptions as they happen. An unusual client delay, a new employee still ramping up, a scope change, or a data outage should be logged against the affected deliverable.

Report results with and without those items rather than averaging them away. A result that survives the exclusion is sturdier than one that only looks good with it.

Finally, keep the claim proportional. In the AISI pilot, the planning task showed no statistically significant uplift on any metric even though the four-task average was positive.

A strong result on paid-search reporting says nothing certain about strategy memos or client planning decks.

Write the conclusion as: this workflow, on this task, with this team, over this window.

Convert the result into margin and usable capacity

Time saved is a hypothesis until it passes through quality and cost. Anthropic cautions that conversation-level estimates may overstate productivity effects because they omit human work beyond the conversation.

Treat any model-estimated or self-reported saving as a starting guess, then replace it with observed workflow data that includes review, validation, and rework.

Calculate quality-adjusted labor cost per accepted deliverable

The formula is total labor and tool cost for the test, divided by the number of deliverables that meet the acceptance standard. Rejected or abandoned work stays in the numerator and out of the denominator.

The example below is generic. Every number is an assumption, not an observed result and not a client result.

Assume one monthly reporting cycle covering different client accounts, with 20 reports per workflow and a blended labor rate of $60 per hour. Assume these hours per report:

Stage (hours per report, all assumed)Existing workflowAI-assisted workflowSetup, prompts, context0.250.50Production2.751.25Review0.501.00Correction0.500.75Approval and delivery0.500.50Total4.504.00

Existing workflow: 20 reports at 4.5 hours is 90 hours, or $5,400 in labor. Assume no tool cost and all 20 reports are accepted. Cost per accepted report is $270.

AI-assisted workflow: 20 reports at 4.0 hours is 80 hours, or $4,800 in labor. Assume $100 in tool cost during the test and 18 of 20 reports accepted. Total cost is $4,900, so the cost per accepted report is about $272.

The AI-assisted workflow saved 10 hours, yet cost per accepted deliverable did not improve. Drafting got faster while review and correction grew, and two reports failed the standard. Hours alone would have said the workflow worked.

Decide whether saved capacity is actually usable

In this assumed 20-report comparison, production time fell by 30 hours, but setup, review, and correction rose by 20.

Assume the production hours belong to junior staff and the added review hours belong to a senior reviewer. The agency freed junior time while consuming scarce senior time, so the 10 net hours are not usable as stated.

Capacity counts only when it has a named destination, such as an audit backlog, a client onboarding, or a defined new deliverable, and the destination does not depend on the same constrained reviewer.

Once the test supports a margin or capacity figure, use it to turn AI efficiency into agency profitability rather than projecting revenue the test did not measure.

Segment results by task and team experience

An agency-wide average can hide a strong use case and a weak one. Break results out by deliverable type, complexity, operator experience, and workflow version before drawing conclusions.

A study published by NBER shows why. Erik Brynjolfsson, Danielle Li, and Lindsey R. Raymond used data from 5,179 customer support agents.

Access to the tool increased issues resolved per hour by 14% on average, including a 34% improvement for novice and lower-skilled workers, with minimal impact on experienced and highly skilled workers.

Customer support is not agency delivery. This result supports segmenting the measurement. It does not forecast any particular uplift for an agency team.

Apply the lesson with a simple table of results by operator experience and task type. Then ask three questions:

1. Does the workflow help newer team members reach the acceptance standard faster?
2. Does it narrow the gap between junior and expert output, or does it add little for experts?
3. Does it create extra review burden for senior staff that offsets the gain elsewhere?

IBM’s advice to engage users with varying skill levels points in the same direction, so include operators across experience levels in the test, not only the enthusiastic ones.

If only one segment benefits, the right move may be to scale the workflow for that segment and keep the existing process for the others. A split decision is a legitimate result.

Use decision rules to scale, redesign, or stop

Set these rules before the result is known, then apply them to the quality-adjusted numbers, not the raw hours.

1. Scale when quality holds or improves, total labor cost per accepted deliverable falls by at least the minimum meaningful difference set before the test, and the released capacity has a defined use. Scale first to similar work, and keep measuring. If the difference is smaller than that minimum, treat the result as inconclusive and rerun or redesign the workflow.
2. Redesign when time falls but rework, defects, or senior review rise. For example, a workflow that saves junior production time but consumes scarce senior review capacity has shifted the bottleneck instead of removing it. Move checks earlier, tighten the inputs, narrow the task, or change who reviews, then rerun the test.
3. Stop when the workflow cannot meet the quality threshold, or when the full cost, including setup, review, correction, and tools, exceeds the baseline.

Segment results can produce different answers for different groups. Record the decision for each segment, with the evidence behind it.

Repeat the test after any material change to the model, prompts, process, team, or client scope.

A result is evidence for the conditions under which it was measured, and those conditions change. Keep the scorecard, the rubric, and the decision on file so the next test starts from a known baseline rather than from memory.

The next step is concrete: pick one deliverable, write its acceptance standard, and record this week’s baseline.