Cover photo: Christin Hume / Unsplash.
Define the research question and eligible prompts
To measure AI answers repeatably, define the question, freeze the prompt panel and observation conditions, collect scheduled answers, preserve the raw evidence and apply the same coding rules to every eligible run. Report the resulting sample with its limitations. A single favorable response cannot establish how often a business is recommended, and a collection of convenient screenshots cannot substitute for a documented method.
This guide provides a prospective testing protocol. It does not report a completed study, sampled AI outputs or measured volatility. The example design uses ten prompts, three specified session times per day and three days: 10 × 3 × 3 equals 90 planned observations for one product or interface. That number describes workload, not statistical adequacy or market coverage. Choose a different design when the decision, available access or required confidence demands it, and document the change before collection.
Begin with a question narrow enough to answer. For example: within a fixed panel of relevant supplier-selection prompts, how consistently does the same interface mention a defined business and represent its service accurately during the observation window? This differs from asking whether AI marketing works. The narrow question defines what you collect and prevents a small descriptive exercise from turning into an unsupported claim about revenue or all users.
Choose prompts from actual buyer tasks where evidence exists: recurring sales questions, relevant site queries, support conversations or a documented service-selection process. Record the origin of each prompt without disclosing personal information. If those inputs are unavailable, label the panel as an analyst-designed convenience sample. It can still test a proposed workflow or expose an inaccurate answer, but it cannot claim to represent a measured distribution of customer demand.
Keep neutral discovery and branded accuracy panels separate. A question that names the company can test whether the answer describes it correctly; it does not test whether the company would have been discovered without that cue. Include service and geographic constraints only when they match the business. Prompts should challenge the actual offer rather than manufacture easy wins by asking for a brand you have already supplied.
Freeze conditions and observation fields
Create a short protocol document before opening the tested interface. Record the product, interface, displayed model if available, market, language, login state and relevant search or web-access setting. Some identifiers may not be exposed. Write “not exposed” rather than inferring them from the style of an answer. Save the interface version or other visible context where available, because the meaning of a product label may change during a longer observation program.
Assign stable identifiers to prompts and runs. A prompt ID refers to exact wording and constraints. A run ID identifies one scheduled attempt under recorded conditions. Keep the full wording instead of a shortened label such as “best agency,” which conceals differences that could affect interpretation. If a prompt is corrected for a spelling error or changed to reflect a new service, give the revision a new version and preserve the earlier definition.
Use fresh sessions where the interface supports them, and record any carryover you cannot exclude. Do not claim that a fresh session eliminates all personalization or hidden context. Its practical purpose is to reduce avoidable influence from the previous conversation and make the procedure more consistent. If the task specifically concerns an ongoing conversation, design that sequence explicitly instead of mixing it with isolated first-turn prompts.
The blank capture schema below separates conditions from outcomes. Conditions describe what you attempted; outcomes preserve what happened. Keep both even when a response fails, refuses or produces no useful result. A record that contains only positive coded answers cannot reveal whether missing observations were accidental, systematically excluded or caused by a product limitation. The raw evidence gives later reviewers a way to investigate rather than guess.
Decide storage and access arrangements before collection. Preserve the full answer, visible citations and a dated screenshot or export when permitted by the interface and organizational policy. Do not place confidential client details into prompts simply to make the test feel realistic. Use authorized, relevant inputs and follow the product's actual access conditions. Reproducibility is a reason to document a legitimate workflow, not to bypass restrictions.
| Field group | Required record | Reason |
|---|---|---|
| Run identity | Run ID, prompt ID, exact prompt version, scheduled and actual UTC time | Preserves planned attempts and deviations |
| Conditions | Product/interface, displayed model or not exposed, locale, login and web setting | Defines the tested environment |
| Raw evidence | Full answer, visible citations, screenshot or export reference | Supports later audit and recoding |
| Outcome | Mention, citation, recommendation, accuracy, unresolved code | Separates different answer behaviors |
| Failure/deviation | Error, refusal, interruption, changed condition, replacement rule | Prevents silent removal of unfavorable or incomplete runs |
Schedule repeats without cherry-picking
Turn the design into a schedule with a cell for every planned attempt. In the illustrative 90-observation design, each of ten prompts has three daily session times across three days. Choose the dates and times in advance, including the time zone. The sampling lattice shows planned cells with no results filled in. It is a planning aid, not evidence that any response was collected or any product behaved a particular way.
Specify the execution order. A fixed order is easy to follow, while a documented rotating or randomized order may help avoid always assigning one prompt to the first session position. Either can be appropriate for a modest descriptive exercise. Record the choice and keep it consistent. Do not change the order halfway through because the first answers look unfavorable unless a real operational problem requires a protocol amendment.
Define the failure policy. An unavailable interface, timeout or interrupted response should receive a failure code and a timestamp. Decide whether a replacement attempt is allowed, when it occurs and whether it is included in the same analysis. Preserve the original failed attempt. Repeating until a favorable answer appears and saving only that answer would measure the operator's selection behavior as much as the product's output.
Log deviations immediately. If a session happens late, the displayed model changes or the web setting is unavailable, record the difference while the evidence is fresh. A deviation does not always make the run useless, but it changes how confidently it can be compared with the intended condition. Keep a primary set that follows the protocol and a clearly marked supplementary set when that distinction helps the analysis.
For manual collection, run a small procedural rehearsal before the main schedule. Check that the capture fields, evidence storage and coding instructions work. Treat rehearsal outputs separately unless the protocol explicitly included them before collection. The rehearsal tests the workflow; it does not justify editing the prompt panel to favor the brand. Once collection begins, preserve the original plan so later changes remain visible.
Code answers with a reproducible rubric
Write the codebook before calculating a score. The minimum useful fields are brand mention, website citation, recommendation presence and recommendation accuracy for the stated requirement. Add categories only when they answer the research question. Too many loosely defined fields can slow review and create disagreements without improving the decision. Each code needs a definition, an observable decision rule and an option for genuinely unresolved cases.
Code the whole answer in context. A brand may be named in a warning, included in a neutral background sentence or offered as a possible supplier. Those are different behaviors. A website citation might support a factual explanation without recommending its owner. A recommendation might be explicit while inaccurately assigning a service or location to the business. Preserve the supporting excerpt and the verified offer fact used to judge accuracy.
Use unambiguous fictional examples to train reviewers, clearly labelled as examples rather than sampled outputs. “Apex supplies equipment” demonstrates a mention. “Consider Apex for this installation requirement” demonstrates a recommendation claim that still needs a service-fit check. A missing capability should not be invented to make the recommendation count as suitable. The codebook should allow a reviewer to explain why the answer failed or remained unresolved.
When resources allow, have another reviewer independently code a meaningful portion of the actual observations, prioritizing consequential or ambiguous cases. Compare disagreements and refine unclear rules. Record which version of the codebook applies, and recode affected earlier observations if the definition changes. Do not silently resolve every disagreement in the direction that produces a higher visibility rate. Consistency matters more than a flattering number.
Keep the coded record connected to its raw run. A spreadsheet row without a saved answer can become impossible to audit after an interface history disappears. Conversely, a folder of screenshots without stable IDs can make aggregation unreliable. The connection between scheduled run, archived evidence and outcome code is the core of the method. It allows corrections to be traced without rewriting the observation history.
Calculate rates with explicit denominators
Calculate rates only after the dataset and eligibility rules are fixed for that analysis. Mention rate is positive mention observations divided by the stated eligible observation count. Citation rate uses the corresponding site-citation outcome. Suitable recommendation rate needs relevant recommendation opportunities and an accuracy rule. Do not substitute a broad total for a narrower eligible set merely because the resulting percentage looks better or is easier to calculate.
Keep failed scheduled attempts visible beside valid answers. Report planned attempts, completed attempts, valid observations, failures and exclusions with reasons. A rate based on a small surviving subset may be mathematically correct while poorly representing the intended schedule. The analysis diagram therefore keeps failure records separate from positive counts before the denominator is chosen. It contains no measured rate because this protocol has not been run.
Evaluate repeatability at the level relevant to your question. Exact word-for-word agreement is different from consistent recommendation classification. Two answers may use different wording while naming the same suitable providers; two nearly identical answers may differ in one important service claim. Define whether you are comparing text, cited URLs, brand inclusion, suitable recommendations or factual accuracy. Avoid an “answer consistency” label that changes meaning across the report.
For citation comparisons, decide whether URL variants should be normalized to a common canonical destination or domain. Preserve the original visible URL so the transformation can be reviewed. Domain overlap and exact-page overlap answer different questions: the same publisher can be cited through different pages, and the same domain can contain conflicting or outdated material. Neither overlap measure establishes the completeness of the underlying retrieval process.
Summarize by prompt as well as across the panel. A pooled average can hide a service-specific failure if several easier prompts dominate it. Show eligible counts and outcomes for each meaningful prompt group, then explain the aggregate. Do not present repeated runs of the same ten prompts as ninety independent buyer intents. The repeated structure is useful for observing consistency under the design; it does not expand the panel's substantive coverage.
Report uncertainty and confounders
Describe the scope of uncertainty in ordinary language. The prompt panel reflects selected tasks, the observation window covers particular dates and the tested interface operates under recorded conditions. Unseen personalization, source changes and product updates may affect the answers. These limitations do not make the observations worthless. They define the circumstances in which the observations can support a decision and the questions they leave unanswered.
Separate search-result behavior from AI answer observations. Search Console's Performance documentation describes variations in search results and reporting context, but an ordinary search-results page is not a captured answer from the AI product you intended to test. Likewise, asking one model to predict another product's responses does not collect those responses. If access to the intended interface is unavailable, report the dependency instead of filling the dataset with simulations.
Do not attach a confidence interval that assumes independent random observations unless the design justifies that assumption. Repeated sessions for the same prompts share important characteristics, and an analyst-selected panel is not a probability sample of all user questions. For a modest operational study, counts, denominators, prompt-level results and a candid description of limits may communicate more accurately than a precise-looking statistical summary with an inappropriate model.
Document confounding changes. A revised service page, a new campaign, a changed model label and a different location setting can all complicate a before-and-after comparison. Record what changed and when. If several inputs moved together, describe the result as a bundled observation rather than isolating a cause you did not test. Preserve unchanged prompt cohorts where continuity matters, and label new coverage as a separate addition.
Report the result that would change your mind. If the initial hypothesis concerns inaccurate service representation, a stable set of accurate recommendations challenges that concern. If it concerns inconsistent source selection, similar coding outcomes may show that citation variation does not affect the decision. This keeps the study focused on resolving uncertainty rather than repeatedly collecting evidence until it matches a preferred marketing story.
Use findings to choose a bounded next test
End the exercise with a decision, not a claim that the measurement itself improved visibility. An inaccurate high-value service description may justify a factual correction. Repeated failure to retrieve an important page may justify an access investigation. Consistent accurate answers with few identifiable referrals may shift attention toward the landing journey or toward a different acquisition priority. Match the next action to the evidence instead of treating every finding as a reason to publish more pages.
For a subsequent intervention, state the proposed mechanism, affected assets, owner, expected observable change and review window. Keep a suitable comparison where feasible and avoid changing unrelated inputs merely to make the work look comprehensive. A before-and-after panel can provide descriptive evidence, but it should not be promoted to a causal business experiment without an appropriate design and outcome measurement.
When external support is useful, compare the measurement and implementation scope of an AI SEO service with the protocol you actually need. Supply the prompt inventory, offer facts, access limitations and decision you want the evidence to inform. Ask for raw evidence retention and a clear handoff, not just a blended score. Your organization should be able to repeat the process or understand a limitation after the engagement ends.
The practical outcome is a trustworthy record: what was planned, what actually ran, what each answer did and what remains unknown. This guide supplies that method without claiming a study has already been performed. Once genuine observations exist, publish the counts and qualifications together. That makes future comparisons useful and prevents a convenient answer from being mistaken for a reliable result.
