Free field notes / Updated 2026-10-21

Concierge and Wizard of Oz Tests Explained

Test an outcome by doing the work by hand before building software. Choose a manual format, disclose human work, and measure delivery honestly.

Concierge and wizard-of-oz tests deliver an outcome manually before you build the system that might eventually automate it. A concierge test makes the hands-on service central; a wizard-of-oz test uses a simple interface while a person performs work behind it. Be honest about human involvement whenever it affects expectations, privacy, quality, timing, or the purchase decision.

Choose the manual format that answers your question

The distinction is about how the customer experiences the test, not permission to conceal relevant facts.

In a concierge test, you might sit with a community newsletter editor and format a short bulletin together. You learn their vocabulary, exceptions, and priorities while providing a useful result.

In a wizard-of-oz test, the editor might submit approved notices through a simple form and receive a formatted bulletin later. You assemble it manually. This tests whether a structured request-and-delivery process works without building a full application.

Both should describe what exists. A suitable note is: "This is a manual pilot. A person will review your request and prepare the bulletin. Automated software is not built yet."

Do not imply instant automation, private machine-only processing, or capabilities you do not have. Calling something a test does not make a misleading promise acceptable.

Match the format to the uncertain assumption

A manual test is strongest when the proposed outcome can be delivered safely and accurately by you. It is weaker when the essential value depends on real-time operation, very large scale, or a technical capability you cannot approximate honestly.

Do not use manual delivery to provide regulated financial, medical, or legal services without qualified guidance. Avoid sensitive customer data when a fictional or reduced example can answer the same question.

Work through a concrete pilot

Imagine an idea for a tool that formats short newsletters for neighborhood groups. The founder wants to know whether editors will pay for a ready-to-share bulletin and whether the required inputs are predictable.

The hypothetical pilot offer is US$40 for one short bulletin, delivered within two days, with one revision. The founder can serve three editors this week. The offer says that a person prepares each bulletin and that automated software is not built yet.

1. Ask editors about their last newsletter and current formatting process.
2. Offer the pilot only if the situation fits.
3. Collect approved notices, preferred sections, and the publication date.
4. Format the bulletin manually.
5. Ask the editor to check it before publication.
6. Record preparation time, missing inputs, revisions, and whether another event is booked.

This tests a bounded administrative outcome. The editor remains responsible for approving notices before publication; the pilot covers formatting and organization.

Disclose human work before it matters

Human involvement can affect whether someone is comfortable submitting information. A customer might willingly send approved public notices but decline to share unpublished member contact details with a manual reviewer. Explain the process before the form, not after the information arrives.

Use ordinary language: "I read each submission and create the output myself." If another person will assist, say so where relevant. Ask only for information you need, explain its purpose, and avoid copying private material into unnecessary places.

Human preparation also changes timing. Do not display "instant result" when delivery takes a day. Show an achievable window and a clear route for questions.

You do not have to explain every tool or administrative step. You do need to explain facts that could reasonably change the customer's decision, including who can see their information and what review the output receives.

Keep service scope stable enough to learn

Concierge work naturally reveals new requests. A customer asks for original articles, another wants help finding contributors, and a third wants distribution managed. These may be worthwhile ideas, but accepting everything makes the original test hard to interpret.

Keep an additions list. Finish the promised package first, then decide whether a repeated request deserves a separate offer.

Record where your personal attention creates the value. If the customer mainly pays for a reassuring conversation and ongoing adjustments, a self-service generator may remove the part they care about. Manual success does not automatically justify automation.

Ask a concrete follow-up after delivery: "Which parts of the bulletin did you publish, and what did you change?" That tells you more than asking whether the document looked professional.

Measure labor and repeat use

Suppose three hypothetical US$40 pilots produce US$120 in revenue. Preparation takes 55, 70, and 95 minutes. That is 220 minutes, or 3 hours and 40 minutes, before outreach and administration.

If materials cost US$6 total, US$114 remains before other expenses and compensation for your time. That number is not profit. Add the time spent scheduling, reviewing, and correcting work before judging the economics.

Look at the spread as well as the average. The 95-minute delivery may reveal a customer type that needs a different package. Do not assume that every slow step will be easy to automate.

A useful next signal is a second booking at a stated price. A customer praising a free pilot is different from someone choosing to pay again for the same outcome.

Know when to automate one step

Automate after you can name a stable input, repeatable process, and useful output. Begin with the step that repeatedly consumes time without requiring much judgment.

For the newsletter pilot, you might automate formatting while still reviewing the content manually. Run the revised process with a small batch and check whether quality and delivery time hold up.

Do not build an entire platform because three people enjoyed personalized help. The next question may be whether customers can supply the needed details without a long conversation. Test that transition directly.

If the manual result is not valuable, faster production rarely fixes the underlying mismatch. Return to the problem or the buyer before investing in automation.

Common mistakes

• Describing manual output as automated.
• Collecting sensitive information without explaining human access.
• Offering unlimited customization inside a fixed-price pilot.
• Ignoring revisions and communication when measuring effort.
• Assuming customers want software because they liked a service.
• Accepting more pilots than you can deliver reliably.

Use the pricing guide for a clear package, and the service validation guide to examine capacity before expanding the pilot.

Next step

Paste your idea into the free triage tool, then identify one outcome you could deliver manually and one fact customers need to know before participating. Keep the first pilot small enough to observe closely.