Reddit has an official API. After the 2023 pricing change, that stopped being good news.

If you last touched Reddit data before mid-2023, the world you remember is gone. Back then you registered a script app, pointed PRAW at it, and for historical sweeps you leaned on Pushshift. All three of those load-bearing pieces changed at once. This guide is about what the official API still does well, where it now actively gets in your way, and how scraping the public endpoints fills the gap — with the comment tree being the part everyone underestimates.

What actually changed in the API

Reddit didn't remove the API. It re-priced and re-gated it, and the details decide whether your project is viable on it:

The part people get wrong: posts are flat, comments are a tree

Most "Reddit scraper" tutorials only pull the post listing — a clean, flat table. That is the easy 20%. The value usually lives in the comments, and comments are not a list. They are a recursive structure where every node points at its parent and Reddit deliberately hides deep branches behind more stubs to keep page weight down.

To get usable data you have to (a) walk the tree depth-first, (b) expand every more stub, and (c) flatten the nesting into rows while keeping parent_id and link_id so you can reconstruct who replied to whom. A maintained actor does all of this and hands you a flat dataset with a depth field and the parent pointers intact — you skip the recursion entirely.

Three ways into a subreddit, and when to use each

Subreddit listing

Pull hot, new, top or rising from r/whatever. Best for ongoing monitoring of a community. Remember the ~1,000-item ceiling per sort: combine top?t=year with new to widen the window rather than expecting infinite scrollback.

Search

Query across all of Reddit or scoped to a subreddit for a keyword. Best when you care about a topic that crosses communities — brand mentions, a product name, a competitor. Sort by relevance for coverage or new for monitoring.

User history

Everything one account posted and commented. Best for vetting a source or studying a power-user. Treat this carefully — it is about an individual, so keep it to aggregate or public-figure analysis, not personal targeting.

Fields you actually get back

For posts: id, title, selftext, author, subreddit, score, upvote_ratio, num_comments, created_utc, permalink, url, flair, and is_self. For comments: id, body, author, score, created_utc, parent_id, link_id, and the computed depth. The two share link_id, so you can join comments back to their parent post in any spreadsheet or SQL table.

Running it without touching OAuth

Here is a sweep of two subreddits, sorted by new, that also expands the full comment tree per post:

curl -X POST "https://api.apify.com/v2/acts/renzomacar~reddit-scraper/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "subreddits": ["SaaS", "Entrepreneur"],
    "sort": "new",
    "maxPosts": 150,
    "includeComments": true,
    "maxCommentsPerPost": 200,
    "expandMoreComments": true
  }'

No token refresh loop, no per-call billing to babysit, no custom User-Agent dance. You get one JSON object per post with a nested comments[] array already expanded past the more stubs. Append &format=csv for a flat spreadsheet your analysts can pivot.

Troubleshooting the things that bite first-timers

• You only got ~1,000 posts and assumed it broke. It didn't — that's the listing ceiling. Combine sorts (top across time ranges plus new) or use search to reach older content.
• Comment counts don't match num_comments. Removed and deleted comments still increment that field. Your scraped tree will be smaller; that's correct, not a miss.
• Deleted bodies show as [removed] or [deleted]. Filter these out before sentiment analysis or they'll skew your text corpus.
• A huge thread is slow. Cap maxCommentsPerPost. The top 200 by score usually carry the signal; the long tail of one-word replies rarely changes your conclusion.
• Timestamps look wrong. created_utc is Unix epoch seconds in UTC. Convert once, up front, before grouping by day.

Disclosure: the links to Apify on this page are affiliate links. If you create a paid account through them we may earn a commission, at no extra cost to you. We recommend Apify because we build and ship the Reddit Scraper on it ourselves.