Developers Fighting to Scrape Web Data

“I've been working on a Likee scraper for a few months, and the latest update finally does the thing people kept asking for: you can drop in an handle or a profile URL and get the creator's videos back. Before, it only worked with share links or numeric user IDs. Here's what it returns: - A playable MP4 link when you start from a share link - Profile details like bio, follower counts, total likes, verified status and linked socials - Creator analytics covering averages, posting freque”

“Before Submitting [x] I searched open and closed issues and discussions for an existing request. [x] I checked whether this already exists on the dev branch or latest source. [x] I understand that maintainers want a well written issue or discussion before any code pull request. [x] This request is not a security vulnerability. Problem China web pages are much harder to crawl. Chinese labs are launching cheaper web search and fetch url service api calls. Desired Behavior Add them to the web searc”

“I run OddsLink, built it entirely myself from scratch. Here's how it happened. In April I was writing scripts to compare odds for my own betting strategy. Every book I added meant more code just to work out which game was which. One book says "PHI Eagles", another says "Philadelphia", a third puts the home team first. Soccer has a draw and US sports don't. One book lists five spreads lines and you have to figure out which one is the main line and many more mismatches”

“https://preview.redd.it/12joq6fgg8wh1.png?width=337&format=png&auto=webp&s=fa5088956727916f11366fe9c41d5e420948d356 I can only test it on Free and Team, need help with testing on the other tiers. https://preview.redd.it/rjsx2cnhg8wh1.png?width=288&format=png&auto=webp&s=0e736bdabec23e48e8eb346a1319295dc55d6380 https://preview.redd.it/f5khut7kg8wh1.png?width=285&format=png&auto=webp&s=5466ca5e8585d9caf5ff7015df6395470de8099e submitted by /u/Key-Acco”

submitted by /u/Tall-Badger8508 [link] [comments]

“tinyfeed is a CLI tool that generates a webpage from a collection of feeds. It's dead simple: no database, no config file, just a CLI and some HTML. This release focuses on making tinyfeed a more stable and deterministic program. A stable experience is always better for users, and it also allowed us to design a new snapshot-based test suite, so regressions are much less likely to slip through. It also included a bunch of fixes related to the `--order-by` option that was added in the previous”

“Sorry you're dealing with this, and eight years between searches is a long gap. Since your recent work was CMS landing pages, I'd build one or two small projects in React and TypeScript with tests and put them on your portfolio, so the 'Software Engineer' postings see current evidence. Also look at agencies and in-house marketing engineering teams, because your landing page and marketing site background is a real asset there. Are you open to contract work while you search, or do”

“Before Submitting [x] I searched open and closed issues and discussions for an existing request. [x] I checked whether this already exists on the dev branch or latest source. [x] I understand that maintainers want a well written issue or discussion before any code pull request. [x] This request is not a security vulnerability. Problem Open WebUI's native image generation currently supports four engines: OpenAI compatible, ComfyUI, Automatic1111, and Gemini. The OpenAI engine assumes a synchronou”

“Thank you so much for sharing! Is it possible to share a sample of what you entered for URL that failed without right error. Will help me reproduce and fix. [re: Claude CoWork for Local LLMs that runs on your Phone]”

“really cool could you do something like add an api point to this so it can be used for web scraping. another idea for increasing durability could be a low resource jev alternative that runs on a worker or something/ another on the fly categorizer to help say what goes into what slot if that makes sense. [re: I built InstaFix to make Instagram Reels playable directly in Discord]”

“I only looked at the landing page, so I can’t assess generated replies or app compatibility. Two setup notes: I’d put the own-API-key requirement beside “FREE” and the download button. Also, “Your messages stay on your Mac” reads broader than the paragraph beneath it, which says message/context prompts go to the chosen model. “Local history; prompts go to your chosen AI provider” would make that distinction clear earlier. If a landing-page exchange is useful, I work on MIRRA: AI-guided voice con”

“Before Submitting [x] I searched open and closed issues and discussions for an existing request. [x] I checked whether this already exists on the dev branch or latest source. [x] I understand that maintainers want a well written issue or discussion before any code pull request. [x] This request is not a security vulnerability. Problem We utilize inlet/outlet filter functions for numerous things but the one that impacts embeddings is our accounting. We use an outlet function to get usage data fro”

“Website needs updating, but did you check the README of the actual GitHub Repo I posted? > you bring the model you trust, or use none at all [re: Contrack 2.0 – Lightweight contacts manager and people search]”

“Hey r/webdev ! Happy Saturday. Whenever I browse inspiring websites, I always like checking what frameworks, libraries, and tools they're running under the hood. But similar extensions feel like they've gotten pretty bloated, put basic data behind paywalls, or actively collect browsing history. So I spent the last few weeks building TechReveal , a clean and lightweight alternative: 100% Private: Runs entirely in your local browser sandbox. Zero telemetry, no data collection, no accounts.”

“We need to find root cause and fix it, so everything is always nicely formatted (we should also check previous releases for similar issues, and carefully investigate all failed cases). We need to download all logs and data related about the issue to this repository, make sure we compile that data to ./docs/case studies/issue {id} folder, and use it to do deep case study analysis (also make sure to search online for additional facts and data), in which we will reconstruct timeline/sequence of eve”

“Hi everyone! A while ago, I was looking for a good open-source URL shortener to self host it on my home server for work. But every solution I found had its own disadvantages like outdated UI, inactive or archived repositories, missing features, and so on. So I decided to take the ideas I liked most from different projects and combine them into a single docker application which is now called ShortSwitch. Somehow, it turned into three modules: Links : A url shortener with QR codes, expiration date”

“Description One common workflow I have is to have an agent monitor my job runs, and repeatedly update an HTML report for me. Marimo lends itself super well to this type of workflow. unfortunately, marimo doesn't expose the "download as HTML" functionality to agents via the MCP or via the CLI. As such, I can't actually use marimo as my main driving interface for agents, even though I'd actually really like to. Are you willing to submit a PR? (You must receive approval from the team before submitt”

“Before you start [x] I searched existing issues and this is not a duplicate. [x] I tested against the latest release or main . [x] This is not a security vulnerability (those go through SECURITY.md, not a public issue). obscura version or commit 0.2.4 (official linux x86 64 release, render build); source checked at main bd95294 OS and architecture Linux x86 64 (Debian trixie container under gVisor; Ubuntu 24.04 host) How are you using obscura? CLI (fetch / serve / scrape / mcp) Affected area CLI”

“I got tired of two things: uploading my data to the cloud for processing, paid tools, and the "paste your data into an LLM" advice. So I wrote a tool where the browser is the compute node. DuckDB-WASM runs in the tab. Files are picked with the File System Access API, copied into OPFS, and registered with DuckDB as a filesystem handle, so Parquet is read by range requests instead of being loaded into wasm memory. Server side is one Rust binary that serves static files. That is the whole”

“Thanks, this is really useful, especially the price reader. I looked at what happened: your plans load in the browser after the page opens, and my reader doesn't run JavaScript, so it saw your FAQ but none of the prices. The message should have said that instead of "no plan with a fixed price". And an error about "that page" after you pasted text is just a wrong message. Both are on my list. You're right that those three lines fight each other. What I mean is that the”

“well done Francesco, nice to see this as an outcome of our Hackathon. one interesting simpler feature to add would be to add a Pauli and HSV decompositions, like the RFI one that you already did. I can pass you some code that uses standard libraries to do this.”

“For a research project in the cultural sphere, we’re looking for someone to help us scrape unstructured data from across the web, clean it, integrate it with our own data sources, and ultimately connect everything into one reliable dataset. On top of that dataset, you’ll build a multimodal embedding space to uncover relationships and patterns within the data. Experience with web scraping, data pipelines, data engineering and multimodal embeddings is essential, and you should be comfortable worki”

“Event type Speaker - User Group, Meetup, Local events Description How about moving API integrations to services that work and are reusable without requiring a robot, instead of embedding them into robotic processes? In this session…”

“the API docs is completly broken please fix it The reference under /docs/api renders unstyled: Scalar's buttons, menus and headings show as raw browser defaults (grey "Open Menu", "Download OpenAPI Document" buttons, bulleted links). A visual fix: /omni:visual fix shows today beside four or five variations, applies the one picked, and opens one pull request that closes this issue. 🦸 Omni man by Omni Loop ©”

“I've seen that "browser-only" caveat buried in so many tools and it's always the thing that bites you when you're actually trying to test seriously. Putting it front and center before someone even starts typing seems like such an obvious move, but half the builders I know forget until someone loses a whole session. The shared starter project idea is clever too. Takes the guesswork out of whether the tester just didn't have a good use case or the tool itself fell flat. I”

“Many of them are very specific and might not apply for these containers. I'd guess by not having them reachable with a public URL you could already ignore quite a few. This is actually two separate reasons why a CVE might not matter. just because library X has a CVE, and an application uses library X, doesn't mean the application is exposed to the CVE. The majority of library CVEs depend on the application using the library in a particular way. Keeping the network endpoint private protec”

Hi everyone, I know many open source tools exist as well many paid ones for Google Maps leads scraping. But none of them are proper complete product (I believe). Still you need…

I asked Claude Opus 5.5 Medium about this and discovered that https://gist.github.com is the way. I also reported the following to security@pastebin.com : Since at least October 14, 2026, loading https://pastebin.com/ redirects…

“without a link to the actual app i cant say, but my experience was that setup for openclaw took a day, and it has grown in capabilities as i have needed it. [re: I got tired of AI tools that upload everything and charge monthly, so I built one that runs 100% offline on your own machine and can actually operate your PC]”

“Hi all, I've been building Senapati, a reputation API for IP addresses, domains, URLs and email addresses. The problem I kept hitting: an IP or domain gets blocked somewhere, and the answer to "why?" is spread across dozens of blocklists with different formats. A single risk score doesn't help when you need to know who to ask for delisting. What it does: One call checks an indicator against public blocklist feeds, DNS blocklists and threat-intel engines, and returns every engin”

“Hi all, I've been building Senapati, a reputation API for IP addresses, domains, URLs and email addresses. The problem I kept hitting: an IP or domain gets blocked somewhere, and the answer to "why?" is spread across dozens of blocklists with different formats. A single risk score doesn't help when you need to know who to ask for delisting. What it does: One call checks an indicator against public blocklist feeds, DNS blocklists and threat-intel engines, and returns every engin”

submitted by /u/furysamurai72 [link] [comments]

“All those APIs are good ones, there are apis in the market available that are dedicatedly build for getting scraped data from social platfroms.. Supadata, SocialKit, scrapecreators.. that offer better pricing then the ones that are mentioned here. Worth testing them out :) [re: YouTube data APIs for title, comments, and transcripts (2026)]”

Hey everyone, Whenever I or my dev friends take up freelance projects, clients send over 10-page agreements filled with heavy legal English. Most of us just scroll to the bottom and sign…

“Sim! Meus únicos plugins são do Git (pra sincronizar via GitHub) e o Templater (que cria arquivos novos baseados em templates pré selecionados (atualmente configurado apenas para arquivos .md) 😃 [re: Minha primeira tentativa de Dashboard com o Canvas]”

“As a backend guy who got pulled into frontend work the same way, the fastest jump in quality came from just copying patterns from sites I liked. Not the code, but the spacing, font sizes, button styles, how they handle empty states. You start to build an eye for what looks off. For code quality, pick a component library and stick to it instead of hand rolling everything. Keeps the JSX cleaner and saves you from the classic backend dev sin of writing a 200 line CSS file that fights itself. Also a”

ShowOnce: record a browser task once, get an n8n workflow (open source)

“I run IndieTools. One realistic case for your checker is a link whose status changes legitimately: our Free listing's website link is NoFollow by default and becomes DoFollow while an optional website badge is verified. A monitor needs to record the current state and the check date, rather than treating the directory's link type as a permanent property. For the weekly email, I'd want three outcomes kept distinct: the link was found, the page was successfully fetched and the link was”

“情境 我在 iSH 裡跑了一個常駐的小服務（127.0.0.1 上的 HTTP 服務，讓預覽面板裡的 HTML 頁面呼叫）。App 被系統關掉、下次再打開後，這個服務就不在了，要等手動在終端機或透過 agent 跑一次指令才會回來。crond 這類常駐程式也是一樣。 目前看到的狀況 稍微看了一下程式碼（main @ b4c0661），理解如下，如果有誤還請指正： 開機流程在 ensureKernelBooted() ： installIfNeeded() → ISHKernel.shared.boot(withRootPath:) → applyDefaultMountOverlay() ，之後沒有執行使用者腳本的步驟。 https://github.com/OpenMinis/OpenMinis/blob/b4c0661d5631ebab4d1a2e6f3fd4c805d4030a6c/src/ios/Agent/Chat/AIChatViewModel%2BMisc.swift L12 L50 boot 裡是 become first process() 後註冊各個 offload”

submitted by /u/Emotional_Essay5771 [link] [comments]

“Open source it ! [re: Claude CoWork for Local LLMs that runs on your Phone]”

“Thanks! Yeah, the margins really are quite different. The job scraper mostly reads JSON feeds, so it's almost free to run, while the audit starts an actual Chrome browser for every page. Regarding the 10k question, the audit doesn't crawl a whole site; it only checks the pages you give it, and each page is charged as it finishes. So the margin per page holds whether it's 10 pages or 10,000. The cap I mentioned is on memory; Lighthouse needs about a full CPU core, so I locked the memo”

“What do you guys do? Would something like OpenRouter work here? I've heard weird things about Merge and I'd personally recommend something more developer-focused instead of sales-led [re: Anyone with experience with Merge.dev? i will not promote]”

“It's an open-source CLI that copies blocks, txs, receipts and traces from your nodes into Avro, exactly as the node returned them, on disk, S3, Kafka or Pulsar. You do the analysis with your own tools. 0.9.0 adds Kafka output and notifications, backpressure (much lower memory on backfills), a quick verify mode, fetch timeouts and delayed archival. https://github.com/emeraldpay/dshackle-archive submitted by /u/splix [link] [comments]”

“New release of Dshackle Archive v0.9, now with Kafka support. It's an open-source CLI that copies blocks, txs, receipts and traces from your nodes into Avro, exactly as the node returned them, on disk, S3, Kafka or Pulsar. You do the analysis with your own tools. 0.9.0 adds Kafka output and notifications, backpressure (much lower memory on backfills), a quick verify mode, fetch timeouts and delayed archival. https://github.com/emeraldpay/dshackle-archive submitted by /u/splix [li”

“That's sad that no one doing some Notion extension in browser or something to "bring back" this function... [re: Change model lost function]”

“The search speed caught my eye, php stuff always feels like wading through mud after a while [re: Nextcloud/filebrowser alternative in GO]”

“Right now it only checks the HTTP status, for both websites and APIs, so a 200 that shows an error page would slip through. Keyword checks are planned for the MVP: you set a phrase the page must contain, and if it's missing, the site counts as down and you get alerted, with a recovery notification when it's back. I'll update you here once it's live. [re: Made an uptime tracker bc i literally didn't know my own app was down lol]”

“I've been experimenting with turning websites into these ridiculous little mechanical contraptions. You know, ramps, springs, bells, things bouncing around for absolutely no reason. I figured websites were getting a little too serious anyway. I guess you could technically use it to make a marketing video. But honestly, I just wanted to watch things bounce around. It's free to play with if you're curious: Thunkroom submitted by /u/TanukiTrix [link] [comments]”

“Before Submitting [x] I searched open and closed issues and discussions for an existing report. [x] I reproduced this bug on the latest release AND on the current dev branch, right before submitting this report. I did not just check an old version or rely on a check from days ago. [x] I understand that maintainers want a well written issue before any code pull request. [x] This is not a security vulnerability. Installation Method Docker Open WebUI Version v0.12.0 Operating System Alma 10 Browser”

“Hi Makers, I am taking on 2–3 quick scenario builds this week with same-day turnaround: • Lead Intake to Google Sheets: Capture incoming website forms, Type form, or Calendly bookings cleanly into formatted Google Shee…”

“Hi all, I am looking for a reasonable sync calendars rest api for commercial use. Can someone suggest something? Thank you! submitted by /u/HerouDev [link] [comments]”

“I've been running OpenAltFinder for almost 16 months now, and received quite a bit of positive feedback. The directory has also grown substantially. Over the last few months I've been receiving more and more submissions that were AI-assisted/vibe-coded.Personally I'm a bit skeptical (both quality wise, and longevity wise), but I also want to give everyone a chance to showcase what they've build. I had already added an "AI-assisted" label, but right now the majority of submissions include that la”

“my guy there is a free github that does this. and everyone is pumping this here at least 3-4 times per week. there are gazilions of similar websites and apps. for real at least ask the clanker for something else. Not 2005 anymore. [re: What I learned pricing a utility SaaS where people use it once and leave]”

“Making people paste in an API key is a pretty quick way to filter out anyone who isn't already technical, which might be skewing your idea of who actually needs this. The people who'd benefit most from quick capture without friction are exactly the ones who'd bounce off a setup step like that. Validation at this stage is less about whether the idea is good and more about whether anyone keeps using it after a week. Watch retention before you think about pricing, because a note-taking”

“For me I vibe coded my own UI because I didn't really like the UI of OpenWebUI Search provider? SearXNG mostly. However before I set that up exa + tavily + firecrawl free tier was enough for me, I just switched for privacy mostly. Model? Qwen 3.8 27b Splash Q4 64k context. Now that does require about 28 GB of VRAM, so you actually need decent hardware. [re: What are you doing for a reliable, well designed local model based AI search?]”

“This isn't the place to ask what to build. This is the place when you have an idea and don't have the time or knowledge to build it yourself. [re: Python scripts, web scrapers, Telegram bots, small automations]”

“Hi, and first off: thank you for OptChat. The summary tree take on persistent memory is a genuinely fresh design, and the care in the implementation shows — the RPC/print guards, the socket liveness profile leases, and restricting join mode to print are all exactly the right calls. The request: I would love to use OptChat with Pi Web (pi web.dev), which is my only interface to Pi. Being able to run OptChat's remembered, endless conversations inside Pi Web would be an amazing experience. Today th”

“User story: Find where a function is used Persona: Alice, a developer planning a refactor, on a laptop Severity: major Tested commit: 618f6f88adcf91fb524c2cd08c637d2ec511b544 ( 618f6f8 , "Merge pull request 147 from arihant2math/claude/next fix p88") Related (closed): 95 ("Code search: search file contents in a repository and across the instance"). Code search is still not reachable from the app at this commit. Steps to reproduce Signed in as u p1 13 : 1. Open http://127.0.0.1:3080/acme/atlas (a”