AI Analytics Output Teardown: Spotting Hallucinated Aggregations, Date Drift, and False Joins
Objective: Given three realistic specimens of AI-generated analytics reports and charts produced by natural language queries, identify mathematical and logical defects—including unweighted average traps, time zone date drift, and fan-out join multiplication—before presenting insights to leadership.
You are the marketing operations specialist at Freshworks reviewing weekly KPI summary slides generated by an AI analytics assistant. You need to inspect each chart and narrative summary for calculation errors before the executive review.
Analyze three AI-generated analytics specimens. For each specimen, determine whether the visual summary accurately reflects the underlying data or suffers from aggregation hallucinations, date filtering mismatch, or duplicate counting.
How do you detect subtle mathematical and table join hallucinations in AI-generated analytics reports before they reach executive decision-makers?
Before you start
What you'll need
- —Basic understanding of SQL joins, aggregation functions (SUM, COUNT DISTINCT), and weighted averages
Free path (everything below is enough to finish)
Compute true weighted conversion rates and check deduplicated transaction totals
Deconstruct generated SQL queries to spot join fan-out and unweighted percentage aggregation bugs
Paid upgrades (optional, faster/deeper)
Mixpanel provides native event-stream deduplication and cohort retention visualizations.
Validate retention cohorts and event funnel calculations against raw user streams
The process
Specimens to review
Evaluate this AI-generated conversion rate calculation. Identify all mathematical flaws, aggregation defects, and false conclusions.
AI QUERY PROMPT: 'What was our average landing page conversion rate across all paid campaigns last month?' AI GENERATED SUMMARY: 'Last month, your paid marketing campaigns achieved an outstanding average landing page conversion rate of 12.4% across your 4 active landing pages.' UNDERLYING DATA TABLE: • Page A (Brand Search): 10,000 visitors, 300 conversions (Conversion Rate: 3.0%) • Page B (Generic Search): 15,000 visitors, 375 conversions (Conversion Rate: 2.5%) • Page C (Retargeting): 8,000 visitors, 320 conversions (Conversion Rate: 4.0%) • Page D (Niche Influencer Test): 100 visitors, 30 conversions (Conversion Rate: 30.0%) AI CALCULATION: (3.0% + 2.5% + 4.0% + 30.0%) / 4 = 12.375% -> Rounded to 12.4%
Specimen: synthetic, realistic
Review this cohort retention report and underlying SQL query. Identify any defects, or verify if the query logic is sound.
AI QUERY PROMPT: 'Calculate 30-day user retention for users who signed up in January 2026.'
AI GENERATED SUMMARY:
'For the January 2026 signup cohort (total 4,200 new users), 1,512 users logged into the platform between day 28 and day 30 post-signup, representing a 30-day active retention rate of 36.0%.'
UNDERLYING SQL QUERY:
```sql
WITH cohort AS (
SELECT user_id, DATE_TRUNC('month', created_at) AS signup_month
FROM users
WHERE created_at >= '2026-01-01' AND created_at < '2026-02-01'
),
retained AS (
SELECT DISTINCT c.user_id
FROM cohort c
JOIN events e ON c.user_id = e.user_id
WHERE e.event_time >= c.created_at + INTERVAL '28 days'
AND e.event_time <= c.created_at + INTERVAL '30 days'
)
SELECT
COUNT(DISTINCT c.user_id) AS total_users,
COUNT(DISTINCT r.user_id) AS retained_users,
ROUND(COUNT(DISTINCT r.user_id) * 100.0 / COUNT(DISTINCT c.user_id), 2) AS retention_pct
FROM cohort c
LEFT JOIN retained r ON c.user_id = r.user_id;
```Specimen: synthetic, realistic
Inspect the SQL query and the reported output. Identify why the AI revenue calculation is distorted and state the root-cause defect.
AI QUERY PROMPT: 'What was our total ecommerce sales revenue from email campaigns in February 2026?' AI GENERATED SUMMARY: 'In February 2026, email marketing generated $320,000 in total sales revenue across 800 customer transactions (Average Order Value: $400).' UNDERLYING SQL QUERY GENERATED BY AI: ```sql SELECT COUNT(o.order_id) AS transaction_count, SUM(o.order_total_usd) AS total_revenue FROM orders o JOIN order_items oi ON o.order_id = oi.order_id WHERE o.utm_source = 'email' AND o.order_date >= '2026-02-01' AND o.order_date < '2026-03-01'; ``` ACTUAL STORE REALITY: • Total unique email orders: 800 • Average items per order: 2.5 items • True total revenue in payment gateway: $128,000 • Actual Average Order Value: $160
Specimen: synthetic, realistic
Final deliverable
A completed 3-specimen teardown audit matrix identifying specific calculation traps (unweighted averages, table join fan-out) and corrected SQL queries.
See a reference example
AI Analytics Output Teardown for Snowflake Marketing Performance Dashboard: 1. Specimen 1 (Landing Page Conversion): Rejected. AI calculated unweighted average of rates (12.4%), ignoring traffic weighting. True weighted conversion rate is 3.06% across 33,100 visitors. 2. Specimen 2 (January Cohort Retention): Approved. SQL correctly defines cohort baseline and computes 36.0% 30-day retention with appropriate DISTINCT counts and LEFT JOIN logic. 3. Specimen 3 (Email Revenue Attribution): Rejected. Critical join fan-out defect: joining `orders` with `order_items` duplicated `order_total_usd` across line items, inflating revenue from $128,000 to $320,000 (2.5x error). Fixed by querying `orders` table directly without joining `order_items`.
Success criteria
You're done when you can:
- Spot the unweighted average error and calculate the true weighted blended conversion rate
- Confirm the cohort retention curve specimen is mathematically sound without false defects
- Identify table join fan-out causing duplicated transaction revenue
- Document the corrective prompt phrasing to fix each AI query generation failure
Key takeaway
AI analytics tools generate visually polished charts with complete confidence, but common SQL traps like unweighted percentage averages and 1-to-many table join fan-outs can inflate key metrics by 2x-3x without throwing a single database error.