- name
- llms-txt-sniffer
- description
- Locate and utilize AI-friendly documentation index files (llms.txt, llms-full.txt) or sitemap.xml. Use when encountering documentation URLs (containing /docs/, /api-reference/, /guides/, or platform subdomains) to instantly map sites and reduce token burn.
- metadata
- {"openclaw": {"requires": {"bins": ["python3", "curl"]}, "version": "1.3.1"}}
- argument-hint
- [url]
- allowed-tools
- Bash, Read
llms-txt-sniffer: The Smart Document Radar
This skill streamlines documentation ingestion by locating the most AI-optimized version of a site's content.
🧠 Why llms.txt?
It provides a high-density, Markdown-based index designed for LLMs to map entire sites instantly and save tokens.
🚀 Discovery Strategy (Two-Stage)
Stage 1: Quick Jump Probes (Instructional)
- URL + /llms.txt: Probe
{input_url}/llms.txtusingcurl -I. - Domain Root: Probe
https://{domain}/llms.txtusingcurl -I.
Stage 2: Advanced Sniffing (Tool-based)
If Stage 1 fails, run the companion sniffer script located in this skill's directory: python3 sniffer.py $ARGUMENTS
📜 Behavioral Rules
- User-Initiated Only: Only invoke this skill when the user explicitly provides a documentation URL. Do not autonomously scan domains.
- Switch to High-Speed Mode: Once an index is found, prioritize its links over manual scraping.
- Index Summary: Always present a brief structure overview.
- Fallback: Use
sitemap.xmlparser results ifllms.txtis missing.