The developer utilized a local 7B/8B parameter LLM, specifically the Qwen model, to automate data collection from major classifieds websites. The approach bypasses anti-bot systems and WAF protections by simulating human behavior through a browser-based agent rather than standard network requests.
The agent uses the Playwright library to control the system-installed Microsoft Edge browser in a visible mode (headless=False). By using flags like --start-maximized and --focus, along with the page.bring_to_front() method, the script ensures the browser window remains active and visible, mimicking a legitimate user session.
To overcome obfuscated frontend code with dynamic CSS classes, the agent parses the DOM tree using semantic text markers. It identifies items by locating price indicators and unique item ID patterns within the URL. For scattered data, the model performs a full text dump of the page to correlate information.
The system includes an 8-second delay after page loading to allow security scripts to execute. During testing, the agent demonstrated autonomy by handling a 429 Too Many Requests error and identifying discrepancies between product titles and descriptions in the DOM text, providing warnings to the user.
