RAG Text Scraper

Extracts clean article text from a list of URLs and saves as .txt files.

The AI Text Scraper is a powerful tool designed for developers, Vibe Coders, Product Managers, and Researchers who need to build high-quality text datasets for Retrieval-Augmented Generation (RAG) systems. Tired of manually cleaning up ads, headers, and other clutter from web articles? This extension automates the entire process, allowing you to turn a list of URLs into clean, ready-to-use .txt files with just one click. ✨ KEY FEATURES ✨ **Time Save! Bulk & Single Page Scraping** No more copying and pasting individual files into separate word doc, cleaning the data and saving as a .txt file! **More Time Save! Intelligent Content Extraction:** Powered by Mozilla's Readability.js library, the extension intelligently removes ads, banners, and navigation menus to isolate the core article content. **The Best Time Save! AI-Powered Cleaning:** Take your data quality to the next level. Connect your own API key to use powerful language models (Google Gemini, OpenAI GPT, or Anthropic Claude) to fix paragraphing, remove duplicate sentences, and eliminate any remaining artifacts. --- 👤 WHO IS THIS FOR? 👤 AI Developers, & Vibe Coders: Quickly build and expand knowledge bases for your RAG applications. Data Scientists: Efficiently gather and preprocess large text corpora for analysis and model training. Product Managers: Rapidly create a proof-of-concept or MVP for an AI feature by sourcing a clean, initial dataset without needing an engineering team. . Researchers & Students: Collect and archive articles and online sources for academic work without the noise. --- ⚙️ **HOW IT WORKS** ⚙️ The extension uses a two-stage process: 1. **Extraction:** It first uses Readability.js to find the main content of a webpage. 2. **AI Cleaning (Optional):** If you enable the AI feature, the extracted text is then sent to your chosen AI provider with a specific prompt to perform a final, high-fidelity cleanup, ensuring the output is perfect for ingestion into a vector database. Get started in seconds. Configure your settings, paste your URLs, and start building your dataset today!