# AI bot requests by page role

Prepared and documentation reviewed: 8 October 2026.

Use this record to decide which important pages need an access or measurement investigation. Companion files: [blank CSV](ai-bot-pages-blank.csv) and [worked CSV](ai-bot-pages-worked.csv). The CSVs contain static values and no spreadsheet formulas. Editing a count does not update a result automatically.

The worked example is entirely synthetic. All domains, bot names, timestamps, request statuses and coverage assumptions are invented. No Clarity account, actual crawler, real log stream, configuration change, citation, referral or customer outcome was observed. The executable exercise below operates on the supplied fictional CSV only; it makes no network calls.

## Start with the question and covered population

1. Name the service-page decision, the source and who can verify it. Keep raw logs and client addresses in controlled storage; use evidence references here.
2. Record the exact hostnames, source timezones, selected period and inclusive/exclusive boundaries. Keep original timestamps before converting them. UTC in the example is an exercise convention, not a claim about all Clarity reports.
3. Establish collection coverage. Record missing domains, periods, sampled or truncated exports and processing gaps. Separately establish how much response-status and identity information is available.
4. Build an inventory of intended public pages, including pages with no requests. State eligibility and one primary role per page. Version the inventory when it changes.
5. Retain requests without converting repeat requests into new pages. This CSV uses one row per request, with a unique record ID and request_count of 1. An aggregated export needs a different, documented transformation.
6. Apply the hostname, period, content and identity scope. Keep exclusions and unknowns visible. Count unique inventory keys separately from request rows.
7. Record the supported observation and the next action with owner, acceptance evidence and status. Proposed, performed and verified are different states.

Do not copy a dashboard total into this request ledger as if it were one request. A list of requested paths is not a complete inventory of eligible pages. A user-agent label is not authentication. A 2xx status does not establish that the intended body was returned.

## Blank working record

### Scope and evidence

| Field | Your record |
| --- | --- |
| Question and business decision | |
| Data source, private export reference and extracted-at time | |
| Exact covered hostnames | |
| Integration/source confirmed by, date and evidence | |
| Original timezone and any conversion method | |
| Start inclusive and end exclusive | |
| Latest complete source time | |
| Gaps, sampling, truncation and retention limits | |
| Operator/bot purpose and identity method | |
| Host, path, content and status filters | |
| Inventory version and eligibility definition | |
| Original requested URL retained; normalization rule | |
| Clarity classification rules and overlap test, if used | |
| Unmapped records and unresolved coverage | |
| Separate referral source, period and hostname coverage | |
| Review date and source URLs | |

### Page inventory

| Exact hostname and path | One primary role / eligibility | Coverage status and evidence |
| --- | --- | --- |
| | | |

Use Service, Proof, Guide and Contact only when they fit your site; these are editorial roles. Add a reviewed role when needed. Keep unknown eligibility separate from No. Do not merge a parent path and child path into one page. Keep a secondary role in notes, not as another primary count.

### Request record

| Field | Your record |
| --- | --- |
| Unique request ID and retained private reference | |
| Exact hostname, requested path and content type | |
| Source timestamp and converted UTC timestamp if used | |
| Claimed bot / operator and purpose | |
| Identity status, method and evidence | |
| HTTP status or unknown | |
| Response body evidence where access is in question | |
| Included or excluded, with reason | |

### Decision record

| Observation / evidence / unresolved limits | Proposed action, owner and acceptance condition | Status and next review |
| --- | --- | --- |
| | | |

The CSV provides one empty template for context, inventory, request and decision records. Duplicate the relevant row type and give it a unique ID. Empty numeric and timestamp cells are unknown until supported; on non-request rows these fields are not applicable. Fill a real zero only when the source and scope establish zero. Keep the instruction row out of calculations.

## Definitions and arithmetic

- Request count is the number of eligible request rows here. Repeated requests to the same page remain distinct requests.
- Page key is the exact lowercase hostname plus case-preserved path. This example does not merge canonical targets, redirect destinations, trailing slashes or query variants. Define and validate any normalization before applying it to your own data.
- Eligible page denominator is the number of reviewed inventory pages in the stated scope with complete collection for the selected period. Show pages whose coverage is unknown beside this denominator.
- Requested-page coverage is distinct eligible page keys with at least one included request divided by eligible covered page keys, multiplied by 100.
- 2xx-page coverage is distinct eligible page keys with at least one included 2xx record divided by the same eligible covered inventory. This is a status observation, not proof of intended content, indexing or citation.
- If the eligible denominator is zero, the rate is unavailable. If collection coverage is unknown, do not manufacture zero requests. A known request with missing status stays in the request total and enters the unknown-status bucket.
- AI request share in Clarity uses all site requests as context. This dataset does not contain that denominator, so that measure is unavailable. It also contains no referral sessions or enquiries; their counts and ratios remain unavailable, not zero.

These are our worksheet definitions. We are not reproducing Clarity's proprietary classification or its pages-crawled denominator. Microsoft documents that custom categories override automatic categories, but the reviewed announcement does not establish all custom-rule conflicts or parent/child precedence. Inspect your actual project with overlapping examples before summing categories.

## Fictional worked scope

The interval is 2026-10-01T00:00:00Z inclusive to 2026-10-08T00:00:00Z exclusive. All eight inventory pages on `advisory.example` are assumed covered by a complete synthetic request stream throughout that period. One additional service page on `insights.advisory.example` has unknown coverage. No actual completeness or authentication check was performed.

The two invented bot names are ExampleSearchBot and ExampleAssistantFetch. Included records carry `assumed_verified_for_exercise`, an explicit assumption for arithmetic, not a genuine verified identity. R07 has only an unverified label and is kept outside that scope. Real use requires an appropriate source verification method and retained evidence.

| Role | Eligible covered inventory | Request records | Distinct requested pages | Distinct pages with 2xx |
| --- | ---: | ---: | ---: | ---: |
| Service | 4 | 6 | 3 | 2 |
| Proof | 1 | 1 | 1 | 1 |
| Guide | 2 | 3 | 2 | 1 |
| Contact | 1 | 2 | 1 | 1 |
| Total | 8 | 12 | 7 | 5 |

The overall requested-page fraction is 7/8, or 87.5%. Service requested-page coverage is 3/4, or 75%. Service 2xx-page coverage is 2/4, or 50%. Total 2xx-page coverage is 5/8, or 62.5%. These rates describe the invented sample only.

The 12 included records contain eight 2xx, one 3xx, two 4xx/5xx and one unknown status. Eleven of twelve have a known status. The unknown R13 still counts as a request to Contact. It is not silently counted as successful or deleted.

| Retained exclusion | Reason |
| --- | --- |
| R07 | Identity unverified. |
| R14 | SVG asset, outside the HTML page scope. |
| R15 | Exactly at the exclusive end boundary, outside the period. |
| R16 | One second before the inclusive start boundary, outside the period. |
| R17 | HTML page absent from the inventory; retain for classification review. |

R01 is exactly at the inclusive start and is included. Seventeen raw request rows minus five exclusions leave twelve. An unmapped request does not silently become an eligible page. If the reviewed inventory changes, record a new version and comparison basis.

`advisory.example/services/cloud-migration/` has zero matching requests in the declared complete synthetic stream. `insights.advisory.example/services/vendor-risk/` remains unknown and does not enter the covered denominator. The four decision records preserve this distinction, request an investigation of the security page's 403/429 responses and propose review of the unmapped URL. All actions are `proposed_not_performed`.

## Reproduce the grouping locally

Save the following code as `group-ai-bot-pages.py` beside the worked CSV. Run `python3 group-ai-bot-pages.py ai-bot-pages-worked.csv`. Python's standard library is sufficient. This code rejects unfinished or live rows intentionally: it reproduces the teaching exercise, not a general Clarity importer. Complete the blank record manually for real work, or have the responsible analyst adapt and verify its source, identity, coverage and matching rules first.

The procedure reads only the CSV path you supply and prints JSON. It does not upload data, access an account or modify the input file. Keep private exports separate from this public example.

```python
"""Reproduce the synthetic exercise only. Does not connect to any service."""
import csv
import json
import sys
from collections import Counter
from datetime import datetime


def instant(value):
    if not value.endswith('Z'):
        raise ValueError('This exercise requires explicit UTC timestamps ending Z')
    return datetime.fromisoformat(value.replace('Z', '+00:00'))


def group(rows):
    # This code intentionally rejects real/unfinished data. Adapt only after
    # establishing its collection and authentication rules independently.
    if not rows or any(r['example_or_live'] != 'synthetic' for r in rows):
        raise ValueError('Use the supplied synthetic worked file only')
    if len({r['record_id'] for r in rows}) != len(rows):
        raise ValueError('Duplicate record ID')
    periods = {(r['period_start_utc'], r['period_end_exclusive_utc']) for r in rows}
    if len(periods) != 1:
        raise ValueError('Incompatible periods')
    start_text, end_text = periods.pop()
    start, end = instant(start_text), instant(end_text)
    if end <= start:
        raise ValueError('Invalid period')
    inventory_rows = [r for r in rows if r['record_type'] == 'inventory']
    inventory = {}
    key = lambda r: (r['hostname'].lower(), r['path'])
    for r in inventory_rows:
        if key(r) in inventory:
            raise ValueError('Duplicate inventory key; resolve overlapping roles')
        if not r['page_role'] or r['eligible_page'] not in ('yes', 'no'):
            raise ValueError('Missing page role or eligibility')
        inventory[key(r)] = r
    covered = {k: r for k, r in inventory.items()
               if r['eligible_page'] == 'yes'
               and r['coverage_status'] == 'assumed_complete_for_exercise'}
    eligible_requests, excluded = [], []
    for r in (r for r in rows if r['record_type'] == 'request'):
        if r['request_count'] != '1':
            raise ValueError('One request per row required; no weighted rows')
        t = instant(r['timestamp_utc'])
        if not start <= t < end:
            reason = 'outside_period'
        elif r['content_type'] != 'text/html':
            reason = 'non_html_resource'
        elif r['identity_status'] != 'assumed_verified_for_exercise':
            reason = 'identity_unverified'
        elif key(r) not in inventory:
            reason = 'unmapped_inventory'
        elif key(r) not in covered:
            reason = 'page_coverage_unknown_or_ineligible'
        else:
            reason = 'included'
        if r['scope_result'] != reason:
            raise ValueError(f"Stored scope result differs for {r['record_id']}")
        if reason == 'included':
            eligible_requests.append(r)
        else:
            excluded.append({'id': r['record_id'], 'reason': reason})
    def bucket(r):
        s = r['http_status']
        if s == '':
            return 'unknown'
        n = int(s)
        if 200 <= n < 300:
            return '2xx'
        if 300 <= n < 400:
            return '3xx'
        if 400 <= n < 600:
            return '4xx_5xx'
        return 'other'
    roles = []
    for role in ('Service', 'Proof', 'Guide', 'Contact'):
        pages = {k for k, r in covered.items() if r['page_role'] == role}
        req = [r for r in eligible_requests if key(r) in pages]
        hit = {key(r) for r in req}
        success = {key(r) for r in req if bucket(r) == '2xx'}
        roles.append({'role': role, 'requests': len(req),
                      'pages_requested': len(hit), 'eligible_pages': len(pages),
                      'pages_with_2xx': len(success),
                      'coverage_percent': 100 * len(hit) / len(pages) if pages else None})
    hit_keys = {key(r) for r in eligible_requests}
    return {
        'notice': 'Synthetic local grouping exercise; no Clarity account or real requests inspected.',
        'period_start_utc': start_text, 'period_end_exclusive_utc': end_text,
        'inventory_pages': len(inventory), 'covered_eligible_pages': len(covered),
        'coverage_unknown_pages': [{'hostname': r['hostname'], 'path': r['path']}
                                  for r in inventory_rows if r['coverage_status'] == 'unknown'],
        'raw_request_records': sum(r['record_type'] == 'request' for r in rows),
        'eligible_request_records': len(eligible_requests),
        'distinct_eligible_pages_requested': len(hit_keys),
        'page_coverage_percent': 100 * len(hit_keys) / len(covered) if covered else None,
        'status_buckets': dict(Counter(bucket(r) for r in eligible_requests)),
        'by_role': roles, 'excluded': excluded,
        'zero_request_covered_pages': [{'hostname': h, 'path': p}
                                       for h, p in sorted(set(covered) - hit_keys)],
        'referral_sessions': None, 'qualified_enquiries': None,
        'referral_ratio': None, 'all_site_ai_request_share': None,
    }


if __name__ == '__main__':
    with open(sys.argv[1], newline='', encoding='utf-8') as source:
        result = group(list(csv.DictReader(source)))
    print(json.dumps(result, indent=2))
```

## Documentation reviewed

Reviewed 2026-10-08. Recheck product behavior and the available controls when applying the method later.

- [Page Classification announcement, September 22, 2026](https://clarity.microsoft.com/blog/bot-activity-page-classification/)
- [Bot Activity setup, metrics and limits](https://learn.microsoft.com/en-us/clarity/ai-visibility/bot-activity-overview)
- [Status, content and filtering updates, May 26, 2026](https://clarity.microsoft.com/blog/updates-to-bot-activity-in-clarity/)
- [Scrape-to-Referral coverage, August 13, 2026](https://clarity.microsoft.com/blog/scrape-to-referral-insights/)
- [Clarity FAQ and distinction between access and use](https://learn.microsoft.com/en-us/clarity/faq)

These sources support the documented product concepts. They do not substantiate any invented request or business result in this workbook.