Scraping Google News: the dead API, decoded URLs, and dedup

If you searched for a "Google News API" and found nothing official, you are not missing anything. Google launched a News Search API in the 2000s, deprecated it in 2011, and has never shipped a replacement. What remains is a patchwork: the news.google.com/rss feeds (undocumented, sparse, and quietly rate-limited), and the news cluster inside ordinary Search. Everything below is about working with those surfaces reliably instead of pretending an API exists.

This is deliberately not a "4 use cases, 3 steps" listicle. Google News has two problems no other source has — obfuscated redirect URLs and massive wire-syndication duplication — and most of the real work is dealing with those. We will spend the page there.

What the public surfaces actually give you

Before deciding how to pull data, it helps to know what each surface returns and where it breaks.

Problem 1: every link is a redirector, not the article

This is the gotcha that surprises everyone. A Google News result does not link to nytimes.com/…. It links to something like:

https://news.google.com/rss/articles/CBMiYWh0dHBzOi8v...0FU?oc=5

The CBMi… token is a base64-ish encoding of the publisher URL (and in the newer format, an opaque ID that must be resolved by a follow-up call). If you store these raw, your dataset is full of news.google.com rows that all look identical and none of which you can attribute to a real outlet. Two ways out:

• Follow the redirect server-side. Issue a request that does not auto-follow redirects, read the Location header (or the consent-page hop), and capture the final https:// publisher URL. Use a real User-Agent or Google bounces you to a consent interstitial.
• Decode the token. For the older CBMi format the inner URL is recoverable by base64-decoding and stripping the length prefixes. The newer format is not directly decodable and must be resolved via the redirect. Treat decoding as the fast path and redirect-following as the fallback.

Problem 2: the same story, thirty times

Run a query like "interest rate decision" and you get the identical AP or Reuters wire story republished by Yahoo, MSN, a dozen regional papers, and three aggregators. If you are counting "mentions" you will be off by an order of magnitude. Dedup that actually works:

1. Normalize the title — lowercase, strip the trailing " - Outlet Name" suffix Google appends, collapse whitespace and punctuation.
2. Cluster by similarity — a token-set ratio above ~0.9 on the normalized title is a reliable "same story" signal across outlets.
3. Keep the earliest pubDate as the canonical event and attach the rest as syndication copies rather than discarding them — the spread itself is signal.

A worked mini-example: a brand-mention monitor

Say you want every news mention of "Acme Robotics" in US English, refreshed hourly, deduped to unique stories. The pipeline is: query → resolve links → dedupe → upsert. The fetch step against a maintained Actor looks like this (swap in your own token and actor id):

curl -X POST "https://api.apify.com/v2/acts/YOUR_ACTOR_ID/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
        "query": "\"Acme Robotics\"",
        "language": "US:en",
        "maxItems": 100,
        "extractImages": false
      }'

The response rows give you title, source, link (already resolved by the Actor), snippet and an ISO publishedAt. From there you normalize titles, drop syndication duplicates by the similarity rule above, and upsert on the resolved URL so re-runs do not create dupes. Because the date is real ISO and not "3 hours ago," you can run it hourly and reliably keep only rows newer than your last watermark.

Feeding a RAG or alerting layer

Clean, attributed, deduped news rows are exactly the shape a retrieval layer wants. Two patterns we see most: (1) embed the snippet plus resolved URL into a vector store so an assistant can answer "what's the latest on X" with citations to the real outlet; (2) a simple threshold alert — if more than N distinct outlets cover the same clustered story within a window, that is a genuine spike worth a notification, whereas counting raw rows would fire on every wire republish.

Frequently asked

Is there an official Google News API in 2026?

No. The original News API was retired in 2011 and never replaced. The RSS feeds are the only first-party programmatic surface, and they are undocumented, capped at roughly 100 items, and throttle a single IP quickly. Scraping the public results is the only way to get queryable, paged news data.

Why are the article links pointing at news.google.com?

Google wraps every result in a redirector that hides the publisher URL inside a CBMi token. You must follow the redirect server-side (with a real User-Agent to avoid the consent page) or decode the token to recover the canonical source. Store both the token and the resolved URL.

How do I avoid counting the same wire story dozens of times?

Normalize the title, strip the " - Outlet" suffix, and cluster on a high title-similarity threshold or on the resolved canonical domain. Keep the earliest timestamp as the canonical event and treat the rest as syndication copies.

Disclosure: the links to Apify on this page are affiliate links. If you create a paid account through them we may earn a commission, at no extra cost to you. We recommend Apify because we build and ship Actors on it ourselves.