Developers Fighting to Scrape Web Data
Developers are frustrated by the difficulty and instability of accessing data from the web. They face challenges like scraping inconsistencies, platform protections, and a lack of standardized APIs, leading them to build custom tools and workarounds to automate data collection and streamline workflows. This highlights a need for more reliable and accessible data infrastructure.
SOURCES (60)
“I still see a lot of projects using joi, but i've also noticed more people moving to zod because it fits nicely with typescript.”
“The two I'm most proud of are 1. A script that uses NTFY to notify the Fractured Space (an old now unsupported multiplayer space game) community if someone has joined one of the few remaining servers so others can join in, and 2. An LLM agent that runs my entire infrastructure, installs updates, diagnoses and fixes issues, spins up and configures docker containers, and makes recommendations.”
“I want this but on a grand scale. Something where you can take a bunch of local websites and build little scrapers bespoke to each. I miss way too many good specials from my bakery haha”
Sure. I am planning to build custom extensions for these requirements.
“I think the default page is handled by the server, since some web hosts use default.html instead of index.html”
“For OP use-case that is okay, because he is not going to open anything except local stuff, his problem is client side js code escaping to the internet, and will be solved 100%. Back to the topic - Its very unlikely you will get auto-attacked by some random shitty malware, unless you target very specific content. Regular pron torrent trackers will be safe for you, because most of shit is a windows encryptors\password stealers and unlikely will target docker sandbox escaping. If very paranoic, kat”
“Thanks for the heads up. I used Claude to help reword my original post for clarity and to pass subreddit filters. The technical problem and solution are entirely my own work (the local Node.js proxy, the testing on different platforms, etc.), but I had Claude rephrase my question for better readability and to trim it down. The proxy code itself is vanilla Node.js that I wrote, and the core issue (corporate SSL inspection blocking new domains) is something I encountered and debugged myself.”
“Found some time this evening to manually test it and to set it up in Veriwasp. In my manual testing, everything was smooth and clear. I naturally knew where I should click and what to expect when I did. It was also easy to navigate and understand the flow and layout of the site. I could easily see myself using this in the future. I know you said the site is hard for bots but good news, Veriwasp handled it perfectly with everything I threw at it. I didn't go too crazy but I did a decently lon”
“Started as a project to test out Claude code, ended up continuing to use it as it is cheaper than alternatives, I can build the features I want and meets my needs for analytics across my other projects. Have just opened it up to public via production submitted by /u/ulmanau [link] [comments]”
“I am currently working on re-developing AniTroves.com and I wanna know What to show on main screen. I have configured an under construction screen but some Devs say I should give 503 error to Google but it will take some time for development to complete so I think it might not be a good idea.Any Sugestions?”
“I was losing my mind maintaining web scrapers, one tiny CSS change and the whole script dies. So I spent the last few weeks building Rosply. It doesn't care about the DOM. It literally takes a screenshot, figures out where the button is, and moves the mouse to click it. Like a human would. It runs locally on Windows. I made it a one-time $29 purchase because I'm honestly so tired of everything being a $20/mo subscription. I’ve had my first sales this week and I’m looking for more feedbac”
“Fair questions, both. Why over HTMX Different shape of problem, not a better/worse thing. HTMX is hypermedia-driven: your server renders HTML fragments, HTMX swaps them in. That's a genuinely great model, and if your backend already renders HTML, HTMX is more mature, better documented, and you should probably just use it. NoJS assumes the opposite: a JSON API and static hosting. No server-side templating, no HTML-fragment contract. You point it at /api/users and bind the result. That's w”
“Can you ask your AI why someone would want to use this over HTMX? You should also ask it why it’s bad to ignore semantic HTML and have click handlers doing navigation on buttons rather than using anchor tags.”
“I think you can vibe code a version that does similar API requests but in this case I think the paying for it is justified as it's very useful and gets the job done efficiently. I use it for bookmarking web pages from my browser in my system. It just works.”
“I was wondering if the x402 protocol works for normal meat users with a browser, rather than only being limited to AI. So I've spent couple of hours setting up a PoC, that you can try yourself. Turns out the browsers are more ready than I thought they would. And it even works relatevely out of the box. submitted by /u/morty29 [link] [comments]”
Quite interesting finds.. would love to have the daily summary!! 🫠
“honestly the proxy is the least of your worries if your whole saas leans on scraping. geonode's ok to start, i've used it, but keep your code decoupled from the provider so you can swap it in one line. never tie your reliability to one vendor. the thing that actually bit me wasn't the ip, it was the target site changing its html and my scraper just returning empty rows without erroring. you think it's running when it isn't. monitor the output, not just whether requests come b”
“browser support is only half the battle build tools cdns and image pipelines usually take much longer to catch up”
“Projects like the Ladybird browser are outright banning non-maintainer PRs [1]. I understand this but can't help to feel it goes against the underlying spirit of modern open source.Rust's Clippy has adopted a more nuanced review-PRs-to-get-yours-reviewed system [2], this impressed me and made me think of whether there are other perhaps similarly "gamified" ways to handle the LLM storm.There is also Vouch, as previously discussed on HN. [3][1] https://ladybird.org/posts/changing-how-we-develop-la”
“Happy to run your extension and fill out the SUS if the swap's still open. Mine's Recoup Radar, a Chrome extension that reads the visible text on a page you're already on (an order, a trial, a bill), spots the return-by or renewal date, and lets you save a browser reminder before it hits. Just after honest usability feedback, not a rating or review trade. Free side covers all the testing, and heads up the store listing's mid-update so the copy may look stale. https://chromewebsto”
“looks interesting, i been thinking about managing my compose stacks without jumping between three different ssh sessions all the time. the multi-host thing is what got my attention, how does it handle when one host goes offline, does the dashboard just show it as disconnected or does it start spamming alerts?”
“I put the full handler + the SQL for the profiles table in a boilerplate if it saves you time — but happy to paste the webhook snippet here if you'd rather.”
“I didn't implement it in the current version/iteration mainly coz it adds complexity on the server side and requires keeping a fingerprint DB that stays current as the browser update. The Multi Signal approach I have (33 UA patterns , header fingerprints , O(1) CIDR match , LRU caches reverse DNS) catches obvious crawlers with low overhead . TLS Fingerprinting would catch the sophisticated ones that fake everything at the HTTP Layer. When I'll be working on the next version/iteration , T”
“Good concept!! I was curious how does the Sign-In with Solana authentication work on the dashboard? Why not just use JWTs or OAuth? Also what happens when Solana rpc is slow or down? Does the whole system break?”
“I would have to research how viable a solution this is, the architecture of the current app would probably depend on the extensions being hosted on the same machine as the app since it's a self hosted website”
“We currently use Scraping Dog In our agency to provide the API service for our ranking tracker, but we’re finding their service is so unreliable. There’s always issues and absolutely zero communication to subscribers about downtime. Are there any better alternatives? Obviously we know about SERP API, but they’re much more expensive. Thanks in advance submitted by /u/JaZziZBaC [link] [comments]”
“Hi! I’ve been trying different ones lately: Apify Google SERP scraper, Serper.dev, DataForSeo and even Semrush (but too expensive) So far DataForSeo seems to be working best for me because it have domain authority and other data I can enrich for an okay pricing for my use. But again I haven’t don’t too deep of a research and I don’t know what I don’t know. submitted by /u/desssirius [link] [comments]”
“I always prefer an isolated browser profile. It gives you a more realistic view of what a new user experiences imho. -Jacob from Flowout”
“No CMS, just raw, old fashioned code in JS/Python. Honestly, it's a lot of coding, even with AI tools, and I kinda regret it sometimes.”
“I dabble in web dev but I’m more of a backend programmer. I’ve done most projects in html,css,JavaScript. I haven’t done much with JavaScript frameworks like react. I’m concerned that with a lot npm supply chain attacks about using react or other similar vulnerabilities. I’m currently working on a backend for a site/app. I’m thinking about react as its cross platform but I’m worried that it could become compromised. What are others doing now after the supply chain attacks? submitted by ”
“Firecrawl self hosting in an arm64 is difficult IMO. Its also not lightweight. If you need crawling, I suppose it might be worth it. If you mainly use it to fetch pages, the cloud services is almost free.”
“Not self hosted, but can connect to Strava and Coros: https://runalyze.com/ I just realize there is a fork on github: https://github.com/codeproducer198/Runalyze”
“Hey everyone, If you do any serious web scraping or browser automation in Node/TypeScript, you're probably familiar with Crawlee and the Apify SDK. It's a fantastic developer ecosystem, but running those scrapers at scale on a hosted cloud can get expensive fast. Plus, sometimes you just can't let sensitive scraped data or automation credentials leave your own infrastructure. So I built Crawlee Cloud . It's a self-hosted, open-source platform for running Crawlee and Apify actors”
“So you’re scraping FB marketplace listings? How are you doing that without getting banned?”
“"Claude, make a obfuscater" lol obfuscating front end code is dumb and a useless task to have done. Also, remember part of the web is also accessibility. How the fuck is this accessible code (I mean disability friendly)”
“You absolutely can. I run a team of front-end developers that build government websites without react and any real frameworks for that matter. We build toward solid web standards like native web components for many simple projects and we use to our in house built SPA framework used for rich web applications. They serve millions of active users each month. I am very much a purest. I hate dependencies on third party project and we avoid them as much as possible. Obviously we can do it for much of”
“Greetings. I am a member of Blue Rogues Development. This post is to gauge initial reactions and feedback from one of the communities we plan to visit in an upcoming software release. I'll keep this brief and on-topic. Since 2024, my team and I have been developing a new kind of software called POWBlock - A high-performance, zero-dependency, system/stack-agnostic Proof of Work service that turns any proxy or server into a defense gate against AI scrapers, bots, and small to medium DDoS attac”
“I did it. Many times. I needed something stable that I would not have to touch for years. No updates just because someone else made a mistake and introduced a security vulnerability - minimal dependencies. So I built a system with zero dependencies. It is not exactly a new experiment for me. I spent years designing and building proprietary CMS, ECS, and CRM systems for Fortune 100 companies, where relying on open source or third party dependencies was often simply not an option because of securi”
“Good catch :) thanks! Fixed and deployed: cancelling now just brings you back to the homepage with a proper message instead of that broken error page. As we say in the Netherlands.. Dankjewel Ferry”
“Fair point! Generated and vendored stuff gets filtered at ingest (vendor/build dirs, lockfiles, patterns like *.pb.go, "@generated" markers), and pages cite the files other code actually imports, so the obvious copies usually never make it in. But you're right about the deeper issue: if a generated client slips through, the validator will happily prove a claim against it. Matching code isn't the same as being the source of truth. Right now "authority"falls out of the”
“Wordpress is fine, it's essentially content editor that generates static site that gets served. The good thing is many are used to it and put up much less resistance to use it vs something custom.”
[GitHub] ★ 1/5 (v1.267.0) — I have so much hate for whoever decided the validation window was 20 seconds.
“It really depends on what you're building. Appsmith > one of the best choices for internal tools and admin panels. Budibase > great developer experience, though the self-hosted limits may be a drawback. ToolJet > solid if you need lots of integrations. Directus > excellent if your project is content- or data-driven. NocoDB > a good Airtable alternative for database-centric workflows. If you need maximum flexibility and don't mind writing code, a custom React or Angular app”
“It is still a sandbox so unfortunately not yet. There have been rumors they working on that feature”
“Does anyone know of any good technographic APIs? I run a dev consultancy and basically want to know what tech stack they currently use (ie what they use for CI/CD or security), so it can help me prepare when i talk to a prospect. I know Builtwith is the leader in this space but i feel its lacking a bit when it comes to finding tech that isnt front end or JS focused.I’ve tried using the Anthropic API but most of the time it seems to give me hallucinated results (based on the results for companies”
“I think this Anubis project is a terrible solution to the problem posed by aggressive web scrapers. Using a web browser with reasonable privacy settings has become a big loss in quality of life already, but the first time I encountered Anubis I got completely locked out of most web servers that deployed it. The situation has improved a little, but I hate that maintainers of great web services have rationalized themselves into believing that creating massive barriers to access their sites is a fa”