# SWAPA Analytics — Claude Code Project Memory

## What This Does
Fully automated podcast, video, **and email** analytics for SWAPA (Southwest Airlines Pilots Association).

**Podcast pipeline — zero manual steps:**
1. `fetch_stats.py` runs every Monday at 9:00am via launchd
2. Hits Buzzsprout API → episode list + play counts
3. Hits Podbean API → authenticates, gets CSV download URL, downloads and parses CSV
4. Merges both into `data/podcast_stats.json`
5. Local web server (also via launchd) serves the folder at `http://localhost:8765`
6. Dashboard auto-reflects fresh data on browser reload — no manual steps ever

**Video pipeline (YouTube + Vimeo) — zero manual steps:**
1. `fetch_video_stats.py` runs every Monday at 9:15am via launchd
2. Hits YouTube Data API v3 → channel stats + per-video stats for every upload
3. Hits YouTube Analytics API (OAuth) → per-video estimated watch time, avg view duration, avg % watched
4. Hits Vimeo API → account stats + per-video stats for every upload
5. Writes everything into `data/video_stats.json`
6. Same local web server serves `swapa_video_dashboard.html`

**Known limitation — Vimeo only:** Vimeo's REST API doesn't expose watch-time/completion-rate
data on this account's plan (`live_premium`) — confirmed by testing every plausible analytics
endpoint (`/videos/{id}/analytics`, `/me/analytics`, etc.), all 404. That data only exists in
Vimeo's website Analytics tab or via an Enterprise-tier API upgrade. Vimeo stats here are
plays/likes/comments/followers/length only. YouTube, by contrast, has full watch-time data via
OAuth (see below) — added 2026-07-07.

**Email pipeline (Constant Contact) — zero manual steps:**
1. `fetch_email_stats.py` runs every Monday at 9:30am via launchd
2. Refreshes an OAuth access token (long-lived refresh token, see Credentials below)
3. Hits Constant Contact v3 `/emails` → full campaign list (id, name, status) — paginated 500/page
4. Hits Constant Contact v3 `/reports/summary_reports/email_campaign_summaries` → bulk sends/opens/clicks/bounces/optouts per campaign — paginated 500/page
5. Joins both directly on `campaign_id` (no per-campaign detail calls needed — the summary report's `campaign_id` matches the emails collection's `campaign_id` directly)
6. Writes everything into `data/email_stats.json`
7. Same local web server serves `swapa_email_dashboard.html`

**Scale note:** SWAPA's Constant Contact history goes back to 2013 — ~6,100 sent campaigns as of 2026-07.
The dashboard defaults to an "All Time" view with a toggle for "This Year" / "Last 12 Months" since the
full history is a lot of volume to parse at a glance. Bulk pagination makes the fetch fast (~40 sec for
the whole history) — do NOT reintroduce the earlier per-campaign detail-call approach (`/emails/{id}`
for each of 6000+ campaigns), which timed out. See "What Not To Do" below.

**Bookmarks:**
- `http://localhost:8765/swapa_podcast_dashboard.html`
- `http://localhost:8765/swapa_video_dashboard.html`
- `http://localhost:8765/swapa_email_dashboard.html`

---

## File Map
```
~/swapa_analytics/
├── fetch_stats.py                   ← Podcast data fetcher. Runs weekly automatically.
├── fetch_video_stats.py             ← YouTube + Vimeo data fetcher. Runs weekly automatically.
├── fetch_email_stats.py             ← Constant Contact data fetcher. Runs weekly automatically.
├── get_youtube_oauth_token.py       ← One-time script to (re-)authorize YouTube Analytics OAuth. Only run manually if the refresh token is ever revoked.
├── get_email_oauth_token.py         ← One-time script to (re-)authorize Constant Contact OAuth. Only run manually if the refresh token is ever revoked.
├── swapa_podcast_dashboard.html     ← Live podcast dashboard. View at localhost:8765.
├── swapa_video_dashboard.html       ← Live video dashboard. View at localhost:8765.
├── swapa_email_dashboard.html       ← Live email dashboard. View at localhost:8765.
├── data/
│   ├── podcast_stats.json           ← Written by fetch_stats.py. Read by podcast dashboard.
│   ├── video_stats.json             ← Written by fetch_video_stats.py. Read by video dashboard.
│   └── email_stats.json             ← Written by fetch_email_stats.py. Read by email dashboard.
├── logs/
│   ├── fetch.log / fetch_error.log              ← Weekly podcast fetch output
│   ├── video_fetch.log / video_fetch_error.log  ← Weekly video fetch output
│   ├── email_fetch.log / email_fetch_error.log  ← Weekly email fetch output
│   └── server.log / server_error.log            ← Web server output
├── CLAUDE.md                        ← This file
├── com.swapa.analytics.plist        ← launchd: podcast fetch script, Mon 9:00am
├── com.swapa.video.plist            ← launchd: video fetch script, Mon 9:15am
├── com.swapa.email.plist            ← launchd: email fetch script, Mon 9:30am
└── com.swapa.server.plist           ← launchd: web server, always-on
```

---

## Platform Architecture
Two hosting platforms. Understanding this split is essential.

| Platform | Role | API Status |
|---|---|---|
| **Buzzsprout** | App downloads (post-switch) · Combined total (pre-switch) | ✅ Live |
| **Podbean** | External downloads post-switch (Apple Podcasts, Spotify, etc.) | ✅ Live via CSV API |

**Switch date: 2026-03-23**
- **Before Mar 23:** Buzzsprout `buzzsprout_plays` = external + app combined. No split available.
- **From Mar 23 on:** Buzzsprout = app only. Podbean = external only. Both required for full picture.

**"Unknown" in Buzzsprout = SWAPA internal app.** Confirmed.

---

## JSON Field Reference
Key fields in `data/podcast_stats.json` → `episodes[]`:

| Field | Meaning |
|---|---|
| `buzzsprout_plays` | Buzzsprout total plays. App-only post-switch, combined pre-switch. |
| `app_dl` | App downloads. Populated post-switch only. Same as `buzzsprout_plays` post-switch. |
| `external_dl` | External downloads (Podbean). Populated post-switch only. Null if Podbean unavailable. |
| `total_plays` | Combined total when both available. Combined (unsplit) pre-switch. |
| `podbean_downloads` | Raw Podbean download count. Null if not yet retrieved. |
| `is_post_switch` | Boolean. True = Mar 23 or later. |
| `podbean_connected` | Top-level flag. True if any Podbean data was retrieved this run. |

---

## Credentials (never commit publicly)
- **Buzzsprout token:** `BUZZSPROUT_TOKEN` in `fetch_stats.py`
- **Buzzsprout podcast ID:** `469237`
- **Podbean Client ID:** `PODBEAN_CLIENT_ID` in `fetch_stats.py`
- **Podbean Client Secret:** `PODBEAN_CLIENT_SECRET` in `fetch_stats.py`
- **YouTube API key:** `YOUTUBE_API_KEY` in `fetch_video_stats.py` (Google Cloud Console project, YouTube Data API v3 enabled)
- **YouTube channel ID:** `UCv2ReHY98iAvmKTu31qJZTA` (SWAPA channel — NOT the personal channel `UClD7v1lcOm4Npx0HoYjLcBQ` that was tried first)
- **YouTube Analytics OAuth (watch-time data):** `YOUTUBE_OAUTH_CLIENT_ID` / `YOUTUBE_OAUTH_CLIENT_SECRET` / `YOUTUBE_OAUTH_REFRESH_TOKEN` in `fetch_video_stats.py`. Same Google Cloud project, separate "Desktop app" OAuth client, scope `yt-analytics.readonly`, consent screen published to production (not "Testing" — Testing mode expires refresh tokens after 7 days). The refresh token does not expire on its own; only re-run `get_youtube_oauth_token.py` if it's revoked at myaccount.google.com/permissions or the client secret is regenerated.
- **Vimeo access token:** `VIMEO_ACCESS_TOKEN` in `fetch_video_stats.py` (scopes: Public + Private, tied to the SWAPA Vimeo account)
- **Podbean podcast ID:** `EmAEvYtOwC6J`
- **Constant Contact:** `CLIENT_ID` / `CLIENT_SECRET` / `REFRESH_TOKEN` in `fetch_email_stats.py`. App registered at
  developer.constantcontact.com under "SWAPA Analytics Dashboard," OAuth Authorization Code flow, scopes
  `campaign_data offline_access`, refresh token type set to **Long Lived** (not Rotating) at app-registration
  time specifically so the refresh token doesn't need updating after every run. Re-run
  `get_email_oauth_token.py` only if the token is revoked at the app's My Applications page or the client
  secret is regenerated. Constant Contact's API sits behind Cloudflare, which blocks the default Python
  `urllib` User-Agent (HTTP 403 / Cloudflare error 1010) — both scripts always send a browser-like
  `User-Agent` header; don't remove it.

Podbean OAuth tokens expire after 7 days. The script always requests a fresh token — never cache it.

---

## Common Tasks

### "Refresh the data now"
```bash
cd ~/swapa_analytics && python3 fetch_stats.py
```
Expected output:
- `→ 261+ total episodes` from Buzzsprout
- Podbean token obtained, episode list fetched, post-switch episodes' stats parsed
- `✓ Written to data/podcast_stats.json`
- `Podbean data present: yes`

Then reload `http://localhost:8765/swapa_podcast_dashboard.html`.

### "Test Podbean API independently"
```bash
python3 - << 'PYEOF'
import urllib.request, urllib.parse, json
resp = json.loads(urllib.request.urlopen(
    urllib.request.Request(
        "https://api.podbean.com/v1/oauth/token",
        data=urllib.parse.urlencode({
            "grant_type": "client_credentials",
            "client_id": "baf2841c7e9e83f77b2e4",
            "client_secret": "97fc818f4a5c1b30884f7"
        }).encode()
    )
).read())
print("Token:", resp.get("access_token", "FAILED"))
PYEOF
```

### "Podbean API returned unexpected response"
Podbean has no bulk CSV/report endpoint — `/v1/analytics/podcastReach` doesn't exist
(404s with an HTML page, not JSON). Real endpoints, confirmed working as of 2026-07:
- `GET /v1/episodes?access_token=...&podcast_id=...&offset=&limit=` → paginated episode list
- `GET /v1/podcastStats/stats?access_token=...&episode_id=...&start=YYYY-MM-DD&end=YYYY-MM-DD&period=m`
  → `{"2026-06": 1622, "2026-07": 102, ...}` monthly downloads for one episode

`fetch_podbean_downloads()` in `fetch_stats.py` lists episodes, filters to
post-switch ones, and sums the monthly stats per episode_id. If Podbean changes
shape again: print the raw response in `fetch_podbean_downloads()` and adjust
the parsing there.

### "Start the web server manually"
```bash
cd ~/swapa_analytics && python3 -m http.server 8765
```
The launchd job handles this automatically. Only run manually for troubleshooting.

### "Check both launchd jobs are running"
```bash
launchctl list | grep swapa
```
Should show both `com.swapa.analytics` and `com.swapa.server`.

### "Dashboard shows Data not found"
```bash
# 1. Is the server running?
launchctl list | grep swapa.server

# 2. Does the JSON exist?
ls -la ~/swapa_analytics/data/podcast_stats.json

# 3. If JSON missing, run manually:
cd ~/swapa_analytics && python3 fetch_stats.py

# 4. If server not running:
launchctl load ~/Library/LaunchAgents/com.swapa.server.plist
```

### "Update the dashboard appearance"
Edit `swapa_podcast_dashboard.html` (or `swapa_video_dashboard.html`) directly. Each reads
from its matching JSON file via fetch(). Save the file, reload the browser — changes are
live immediately. No build step. No deployment.

### "Refresh the video data now"
```bash
cd ~/swapa_analytics && python3 fetch_video_stats.py
```
Then reload `http://localhost:8765/swapa_video_dashboard.html`.

### "Check both video launchd jobs are running"
```bash
launchctl list | grep swapa
```
Should show `com.swapa.analytics`, `com.swapa.video`, and `com.swapa.server`.

### "Manual CSV fallback (if Podbean API is down)"
```bash
python3 fetch_stats.py --podbean ~/Downloads/podbean_export.csv
```
Bypasses the Podbean API entirely. Use only if the API is unavailable.

### "Refresh the email data now"
```bash
cd ~/swapa_analytics && python3 fetch_email_stats.py
```
Takes about 40 seconds (bulk paginated API calls, ~13 pages each for campaigns and reports). Then reload
`http://localhost:8765/swapa_email_dashboard.html`.

### "Check all launchd jobs are running"
```bash
launchctl list | grep swapa
```
Should show `com.swapa.analytics`, `com.swapa.video`, `com.swapa.email`, and `com.swapa.server`.

### "Constant Contact API returns HTTP 403 / Cloudflare error 1010"
This is Cloudflare blocking the request, not a real auth failure — it triggers on Python's default
`urllib` User-Agent string. Confirm both `fetch_email_stats.py` and `get_email_oauth_token.py` are sending
a browser-like `User-Agent` header on every request (they should already — this was fixed during initial
setup on 2026-07-10). If it recurs, that header is the first thing to check.

### "Re-authorize Constant Contact (refresh token revoked)"
```bash
cd ~/swapa_analytics && python3 get_email_oauth_token.py
```
Opens a browser to Constant Contact's consent screen. After approving, paste the new refresh token into
`REFRESH_TOKEN` in `fetch_email_stats.py`.

---

## What Not To Do
- Do not move `swapa_podcast_dashboard.html` out of `~/swapa_analytics/` — the web server serves the whole folder; the dashboard must be at root level relative to `data/`
- Do not change the server port without updating the bookmark and reloading the plist
- Do not cache the Podbean OAuth token between runs — it expires in 7 days
- Do not add npm, Node, or non-stdlib Python packages — zero dependencies is a feature
- Do not fetch Constant Contact campaign details one-by-one via `/v3/emails/{campaign_id}` for every
  campaign — with ~6,100 campaigns this times out. The summary report's `campaign_id` field already
  matches the emails collection's `campaign_id` directly; join on that instead of walking each campaign's
  `campaign_activities` for an activity ID.

---

## Jason's Preferences
- Direct, minimal — no padding, no flattery
- Confirm before any destructive or irreversible action
- Simple readable code over clever abstractions
- Python stdlib only unless there is a specific unavoidable reason