Developers Fighting to Scrape Web Data
Developers are frustrated by the difficulty and instability of accessing data from the web. They face challenges like scraping inconsistencies, platform protections, and a lack of standardized APIs, leading them to build custom tools and workarounds to automate data collection and streamline workflows. This highlights a need for more reliable and accessible data infrastructure.
SOURCES (60)
“True, maybe it's just churn over the years as well. I've mostly found it to not be able to reliably handle the node types until the specific version is mentioned.”
“Thanks for your work on this, its great! Would be wonderful to have opencode components as well.”
“Thank you for giving it a try. I just tried importing that recipe and it pulled the steps and ingredients. Are you saying the steps and ingredients were too long? There is a recipe export feature, the tutorial button explains where that can be found on the screen (it’s the 3 dots). With a small font size I’m able to fit it all on a 5x8 https://preview.redd.it/ce43qu00ycnh1.jpeg?width=1179&format=pjpg&auto=webp&s=854b28f70ea26ed29aff36190723c27b06aa4279”
“You won’t get laughed off the face of the earth. This is a pretty normal “non-dev got surprisingly far with AI/tools” situation now. The important thing is not to ask someone to “clean up 32k lines” as a vague job. Ask for a scoped technical audit. I’d break it into phases: Freeze feature work for a bit. Every new AI-built feature increases the review surface. Make sure the basics are safe: environment variables, auth/session handling, Stripe webhooks, database permissions, rate limits, input va”
“I’m building what Id call a simple price tracking tool on the side, nothing crazy, I just wanna watch a few product pages and log when stuff drops. Unfortunately, my sessions keep breaking like Ill start a scrape and halfway thru it just gets blocked or the ip changes and everything resets. I have tried reading about proxy types and almost every guide sounds the same after a while. All this is confusing, datacenter, residential, rotating, sticky... what's the actual difference that matters f”
“Manifest renders every page in a real headless browser (Playwright with system Chromium) before extraction, it's not working off raw HTML or a static fetch. That's necessary because most of the pages people want agent actions from are JS-heavy: React/Vue SPAs, elements that mount after hydration, shadow DOM components, etc. A raw-HTML approach would miss most of the interactive surface. The flow is roughly: Playwright loads the page and waits for it to settle, a set of DOM extractor scri”
“vercel has this built in - turn on deployment protection for the preview and only people you share it with can get in. also throw a noindex header on it so google doesn't pick it up while you test. that's it, no need to overthink”
“It would be so helpful to have the rich data from https://look a pedia.netlify.app/ automatically available as special fields (or tags, failing that). There are so many obfuscated or confusing names for looks having the "This looks like" field would be very helpful. Might have to match based on var name, unless hub URL is also stored locally?”
“I want to set up a Docker-run webpage archiving service. From checking around, including this Reddit, I found 4 main candidates: Linkwarden Linkding Karakeep Readeck I previously tried Linkwarden and Karakeep, which I guess are the most popular, but I cannot say that I liked any of them much (hard to say why). My requirements are to save even hard-to-save pages (like Facebook posts. Only posts, I don't care about comments) and later search them and/or reopen as html-files (I don't like p”
“Hi @marocchino — first off, thank you for sticky pull request comment . We use it in an org wide automation and it replaced a hand rolled gh api POST + search + PATCH dance. The header + update in place model is exactly right and it Just Works. Two requests, both directly useful for "agent produces a file, action posts it" pipelines like ours: 1. Skip the run (or a skip if empty input) when the input file is empty Our pipeline has a strong "silence is success" contract: a reviewer agent writes i”
“First, thanks for IVFi. It is a genuinely nice piece of software and it is the reason I went looking at the code at all. While going through master I found that output is never encoded, and I was able to reproduce three cross site scripting vectors against a stock build. I have fixed them in a fork and I am happy to open PRs here for any or all of it, or for you to just take the patches. I tried to report this privately first, but private vulnerability reporting is not enabled on this repository”
I made this lil GUI batch subtitle downloader based on subliminal a while back. Over the past couple weeks, I've made a few huge improvements as well as adding proper debian linux…
“Before Submitting [x] I searched open and closed issues and discussions for an existing request. [x] I checked whether this already exists on the dev branch or latest source. [x] I understand that maintainers want a well written issue or discussion before any code pull request. [x] This request is not a security vulnerability. Problem Some error messages may look scary to users while also exposing server details. The specific cause in this case was that a user created a once per minute automatio”
“had this exact problem with a partner's inventory system. their site was a mess and manual updates were killing me.”
“Tough problem. The hard part isn't parsing DOM. It's getting the easing and trigger points right without spitting out messy code. If you can nail clean output for GSAP or Framer Motion, that's something people would pay for”
“Can you at least describe what you are trying to accomplish? Because the web is quite powerful these days, even for more advanced stuff you have webgl and related libs like threejs for easier manipulation.”
“"Clone" is generous, I only kept the green/red strip. Every row is a feature instead of a URL. I've joined a few startups now and it's the same thing every time. No tests, one or two manual QA person, everyone just hopes. At the last place signup had been broken on web for two months before anyone noticed. Two months. Mobile still worked so signups never dropped to zero, the number just went down a bit and nobody went looking. Every endpoint returned 200 the whole time. So the”
“As a data hoarder, the auto-archiving feature caught my eye. Does it save a full html/media snapshot or just the cleaned text content?”
same, one of those things where you dont need it until you suddenly do
“this is exactly it, the DOM rewriting is the real problem not the comparison logic itself”
“Forgot: They typically use whisparr, connected to Stash app and stashdb via plugins, use prowlarr, a usenet search engine and a usenet data provider, just like I do with sonarr and radarr plus jellyfing for playing. Just ask on Discord, they are happy to talk about technical computer problems”
“What was the 1 lol? If you meant maps api from apify- yes, and it has extremely shitty filltrate”
“I've been going through a journey of anlyzing this at the raw .. meaning I have written a go language based server. It's not a framework, so I have access to very raw "metrics". And store everything in open source sqlite. Like a httpacces.log but tailored to filter out the noice, but at the same time, I like a little bit of the noise. I like watching the bot traffic. Funny thing is , the more social media and advertising posts, the more bot traffic I get. Here's my site but”
“Hi NotRexed, Thank you so much for porting HowToFishport to the web! It runs beautifully on wasm.rip, and I really appreciate your work (and huge thanks to wasm.rip too). However, I tried running the downloaded files locally, and they aren't working like the code version. The download only includes the Build folder, but it is missing the StreamingAssets and TemplateData folders. I also noticed that the index.html file on wasm.rip is different from the HTML file you posted. Could you please check”
Sounds good, obviously not yet, share the repo/access and I'll integrate
“I know these are just examples, but please add a license, e.g. CC0 so it's fine to reference and use. See: https://choosealicense.com/no permission/”
“Hand-written mocks that return happy JSON are how you ship pagination bugs. What worked is a fake server for the behaviors you actually care about: 429s, partial pages, stale fields, empty lists. Keep it behind the same adapter the real client uses so prod and fake share the request shape. Record/replay one golden path per vendor as a canary, not the whole suite, and re-record when their changelog moves. Sandbox is for the occasional manual check, never CI on every PR.”
“Hookdeck is the closest thing, free tier gives you the inbox, retries and replay without building anything. Worth trying first, though if you outgrow it you're back at the table anyway.”
“Yup! Dropping a new release here soon that you can preview on the dev branch. Most trackers should be fine with it- haven’t had anyone complain yet. I took great lengths to make sure the api requests/scraping was as respectful as possible to the trackers.”
Community node - complex web scraping with Puppeteer
“Hello again I think the API crashed again <img width="908" height="263" alt="Image" src="https://github.com/user attachments/assets/4fb9e1ce 6650 40bb bad0 ccce9acca1a5" /”
“Your current environment <details <summary The output of <code python collect env.py</code </summary </details 🐛 Describe the bug Describe the bug Chat and structured output requests return HTTP 400 or semantically invalid HTTP 200 responses for affected templates, while the Llama 3.1 control passes. To Reproduce Run this standalone Python program on the target machine after ensuring the selected model path exists and the GPU has sufficient free memory: python import json import subprocess impo”
“What was measured the file cannot be parsed, so every consumer is broken Why this matters: the file cannot be parsed, so every consumer of it is broken Reproduced by importing the module from a scratch cwd, which is why it does not show up when run from the repo root: C:\Users\viper\mind palace temp\js\github api.js:240 async createRepo(name, description = '', private = false, autoInit = true) { ^^^^^^^^^^ SyntaxError: Unexpected identifier 'createRepo' at wrapSafe (node:internal/modules/cjs/loa”
“Hiya, ive just installed it, looks good! :) is this still actively maintained? also was curious if using this will get us banned from the trackers as they have some strict rules about automation right?”
“I use it for really random things. Mostly research. Some are workflows: - scan for conferences I want to attend/speak at that are 6 weeks out. Checks flight prices and hotel costs. - competitor research. Has turned up some useful stuff. - fundraising research - keeping track of people, what they’ve posted - keeping a weekly plan based on what comes in from Slack chats or Granola. Sources it and asks for approval to add it to the weekly plan. - other random home stuff like what trash has to go ou”
“For a chrome extension manifestV3; with transformers.js and a small embedding model running in an offscreen doc - how should I go about improving the efficiency and speed of displaying the processed results when some pages in a website are tens of thousands of words of legitimate content. The library doesn't support concurrent calls to the same model session as far as I know (Session already started" / "Session mismatch") and big pages can block smaller pages and big pages can”
“I set up Hermes with it and it worked great. Have arrows that pop up when needed Great question though.”
time is too short. was editing one line and already got cut off..
“that banner looks like a brutalist album cover for a dev conference nobody asked for, i kind of love it”
“this is actually pretty clever, the lazy mode alone would save so much token bloat on big apis. 38k chars down to 1.7k is no joke been burned by auto-generated tool wrappers that fall apart the second you need auth or retry logic so its nice to see those built in from the start”
“Low and medium risk actions are not very clear to me. Plus, I don't see how this is protected from new functionality being introduced into future OS versions. Like, what if there will be a new "list directory" command? There isn't any categorization, as far as I know.”
“If you're actually spinning up real engines client-side and not just mapping JS regex to other flavors, that's a huge step up. The PHP/PCRE vs Python `re` differences alone have burned me more times than I can count, especially with lookbehinds and the way flags get handled Gonna throw a few gnarly patterns at it later tonight, curious if the Go RE2 mode actually rejects backtracking-heavy expressions or just silently falls back to something else”
“Project Name : parse-dmarc / DMARCguard Repo/Website : https://dmarcguard.io https://github.com/dmarcguardhq/dmarcguard Description : strict-RFC compliant DMARC monitoring solution to read compressed XML reports from major ESPs such as google, microsoft and others, and deliver a clean and simple dashboard with all your senders. all batteries-included inside one single-binary or docker-container, with a beautiful dashboard. I initially started this as a way to sort out my own DMARC reports after”
“i just made one property named Type for now, if they get messy, i can change the naming based on the folder”
“I am running SearXNG from a VPS, and apparently the majority of search engines blacklist IP addresses from data centers—or rather force CAPTCHAs, which is effectively the same thing. I have gone through settings.yml and disabled the big players (Google, DDG, etc.). I signed up for the Brave API, which remains free as long as you stay under 1,000 searches a month, but I really want more variety. Is there any method I can use to fix the issue? I do not want Google anyway, so it has been disabled f”
if it works, it works; static migrations are rarely pretty inside
“I use a local Claude agent (via Claude Code on a Mac cron) to scrape the key fields from all those tabs every ~15 min and dump them into a single markdown file. Keeps my working memory free and I'm only tab-switching when I find a hit. Could adapt it to your own sourcing loop pretty easy.”
“wget/HTTrack miss the JS-rendered bits because they never actually run the page. Snapshotting API JSON into the webroot (like the top comment) is the right move if your UI is a thin client over a few endpoints. If the HTML itself is assembled in JS, I'd also prerender: crawl the sitemap in a headless browser, wait for network idle, save the fully rendered HTML, then rewrite asset URLs so CSS/fonts/images are local. Quick check: after the crawl, open a couple of pages with JS disabled and see”
“Steps to Reproduce: 1. Open agents window 2. Click issue / PR 3. Select issue 4. Have the URL ready to go and try to paste 5. 🐛 Paste is not allowed”
“Hello! I found a non urgent issues in the API that makes UX much worse when working with models from web or with remote servers because we can't see current state of a ollama: is it downloading model? did it fail downloading model? is it doing inference? how much RAM/VRAM is used? Also lack of such status endpoint it is not clear what to do if connection was aborted during pull how to check the status of pull operation? Lack of this endpoint ends up in a weird UI in most projects i have seen so”