Music Growth Pipeline

The Last.fm API returns only cumulative all-time stats — there is no native time series. To study whether an artist's audience size correlates with listener growth over time, you have to build the longitudinal dataset yourself by snapshotting repeatedly.

Weekly ingestion pipeline snapshots listener data for artists from the Last.fm global chart into Postgres on Neon. Artists are bucketed into size quintiles by their listener count at the start of the measurement window, so growth is compared across audience size rather than chart position. A dbt transformation layer (staging + marts + a dedicated api/ serving layer, pre-joined and pre-indexed so the app never queries the marts directly) powers both cross-sectional and longitudinal analysis. That serving layer feeds a public Next.js 15 (App Router) web app, self-hosted on EC2 behind Caddy (auto-HTTPS) at music.deanslist.dev — search, artist pages with growth charts and genre/size-band/similar-artist comparisons, leaderboards, and genre breakdowns, all reading live from Postgres through a rate-limited, read-only (app_readonly) API layer. After each weekly snapshot, dbt rebuilds the marts, a stats script writes pipeline_stats.json to GitHub, and both this portfolio card and the live site pick up the fresh data.

• → Core finding: median listener growth declines monotonically with artist size — the smallest band grows roughly 1.5× the median rate of the largest
• → Averages sit well above medians in every band — a thin tail of viral breakouts skews the mean, so the median carries the finding
• → Genre signal: EDM shows highest median growth rate; classical and metal are slowest — genre appears secondary to artist size as a growth predictor
• → Standout cases: several small artists grew 100–400% over the window — growth patterns split between viral spikes and steady week-over-week acceleration
• → Caught a real security incident before shipping: a pasted read-only connection string was actually owner-authenticated, so the "read-only" app could write — closed by hardening a role-identity check that fails closed (500) if the connected DB role isn't app_readonly
• → Migrated from Vercel to self-hosted EC2 behind Caddy: eliminated the self-fetch pattern where server components were fetching their own /api/* routes via HTTP (which needs a running server during build) — pages now query the DB directly via a shared queries module, and the API routes remain as thin wrappers for client-side consumers