Portfolio
GitHub

Python library

AiScrapeLite

An LLM-guided web scraper you can pip install

You give it a URL and describe the data you want in plain English. It opens the page in a real browser, lets a language model decide what to click or type, and hands back what you asked for as JSON. You don't write selectors for each site.

View Source See how to use it
  • Python
  • Playwright
  • Groq
  • markdownify
  • asyncio

How it works

A loop to get there, then one extraction

The model never sees raw HTML while it navigates. It gets a short numbered list of what's on the page and answers with actions that point at those numbers.

  1. 1

    Open the page

    Playwright launches Chromium, goes to the URL and waits for network activity to settle.

  2. 2

    List what can be used repeats

    A script inside the page collects every visible link, button, input, dropdown and text area and numbers them, so the model sees lines like [4] INPUT: "Search". Hidden elements are skipped.

  3. 3

    Ask the model what to do repeats

    The list and your goal go to Llama 3.3 70B on Groq. It replies with up to three actions, like filling element 4 and pressing Enter, or with an empty list when the data is already on screen.

  4. 4

    Carry the actions out repeats

    Each action runs against its numbered element: click, fill, press Enter, select, scroll or wait. Steps 2 to 4 repeat until the model has nothing left to do or max_steps runs out.

  5. 5

    Turn the page into markdown

    Scripts, styles, navigation and footers are stripped out, and what's left becomes markdown, so the model reads the content instead of the markup around it.

  6. 6

    Extract and check

    The model pulls out what the goal asked for as JSON. The library strips any code fences the model wrapped around it, parses the result and confirms every field in required_fields is present.

Usage

Install and run

Install

$ pip install git+https://github.com/Pr45H4nt/aiscrapelite.git
$ playwright install chromium

# .env
GROQ_API_KEY=your_groq_api_key_here

From the command line

$ aiscrapelite "https://news.ycombinator.com" \
    "get the top 5 post titles"

From Python, with required fields

import asyncio
from aiscrapelite.scraper import scrape

async def main():
    result = await scrape(
        url="https://github.com/trending",
        goal="get the top 3 trending repositories with name, description, and stars",
        required_fields=["name", "description", "stars"]
    )
    print(result)

asyncio.run(main())

Pages that need a search or a click first

result = await scrape(
    url="https://reddit.com",
    goal="search for 'python' and get the first 3 post titles",
    max_steps=3
)

What comes back

{
    "valid": bool,
    "data": dict or list,
    "errors": list
}

Details

Good to know

  • The browser window stays visible while it runs, so you can watch each step happen.
  • If an element disappears between planning and acting, that action is skipped and the rest still run.
  • Planning calls use a low temperature (0.1), which keeps the chosen actions consistent from run to run.
  • Extraction sends the first 6,000 characters of the page's markdown, since the content that matters is usually near the top.
  • When the model's reply isn't valid JSON, valid is false and data holds the raw text so you can see what went wrong.

Read the code

Six small modules, one for each part: the page snapshot, running actions, the LLM calls, markdown conversion, extraction and the main loop.

View on GitHub