Concept testing and A/B testing get shoved into the same sentence because both involve options and numbers. But they observe different things.
Concept research records reactions to an idea or stimulus. A live A/B test records behavior under randomly assigned variants. If your store barely has traffic, use early research to narrow the field—then save the visitors for the question that really needs behavior.
The short answer
Key takeaways
- Concept testing is for learning and screening before exposure gets expensive.
- A/B testing is for causal comparison on a live experience with enough traffic and clean assignment.
- Don't promote separate synthetic concept runs into a fake experiment.
What each method actually sees
| Method | Observes | Strong question |
|---|---|---|
| Synthetic concept screen | Generated reactions to a stimulus | What should we revise or question? |
| Human concept test | Stated reactions from recruited people | How do people understand or rate the concepts? |
| A/B test | Behavior on randomly assigned live variants | Which variant causes a different outcome here? |
Sources: Quirk's glossary: Product concept testing, Shopify: How to run an A/B test, Shopify: Split testing for valid tests
Use concept work to protect scarce traffic
If you have four positioning ideas and 300 monthly visitors, running a four-way live test is mostly a waiting exercise. Screen the obvious weak spots first. Check comprehension. Ask relevant humans about the finalists if the decision warrants it.
That doesn't prove the remaining option will convert. It simply stops obviously confusing ideas from consuming the traffic you need for a behavioral question.
An A/B test needs more than two URLs
- Random assignment or an equivalent defensible allocation method.
- One primary outcome chosen before looking at results.
- Enough observations for the effect size that matters.
- Stable tracking, pricing, inventory, and campaign mix.
- A rule for stopping that isn't “when our favorite wins.”
Sources: Shopify: How to run an A/B test, Shopify: Split testing for valid tests
Why separate PaperPMF runs aren't A/B tests
PaperPMF tests one concept per run. Each run generates its own panel, and the workflow isn't designed as a paired, randomized concept comparison. Comparing two aggregate numbers may spark a hypothesis, but it doesn't identify a causal winner.
Use the generated reactions to improve each option. Then compare finalists with a method designed for comparison or put a clean experiment on the live store.
Sources: PaperPMF methodology
The evidence sequence
- Write distinct, factual concepts.
- Screen for confusion and weak assumptions.
- Get human feedback where lived experience matters.
- Choose one high-value live comparison.
- Read purchases, returns, and customer quality alongside conversion.
Sources: Morwitz, Steckel, and Gupta: When do purchase intentions predict sales?, Shopify: How to test a business idea
Sources and verification
Product details are based on official documentation reviewed on August 4, 2026 unless noted. Features and pricing can change; verify them with the provider before making a purchase.
- Quirk's glossary: Product concept testing — Category definition; reviewed 2026-08-03.
- Shopify: How to run an A/B test — Official ecommerce experimentation guide; reviewed 2026-08-03.
- Shopify: Split testing for valid tests — Official explanation of simultaneous variants and isolation; reviewed 2026-08-03.
- Morwitz, Steckel, and Gupta: When do purchase intentions predict sales? — Peer-reviewed research on conditions affecting the intention–purchase relationship; reviewed 2026-08-03.
- Shopify: How to test a business idea — Official staged validation guidance; reviewed 2026-08-03.
- PaperPMF methodology — First-party product documentation; reviewed 2026-08-03.
