Skillget

genpark-robots-txt-crawl-delay-policy-evaluator-skill

mcp
Security Audit
Warn
Health Warn
  • No license — Repository has no license file
  • Description — Repository has a description
  • Active repo — Last push 0 days ago
  • Low visibility — Only 7 GitHub stars
Code Pass
  • Code scan — Scanned 6 files during light audit, no dangerous patterns found
Permissions Pass
  • Permissions — No dangerous permissions requested

No AI report is available for this listing yet.

SUMMARY

RFC 9309 robots.txt parser with path permissions, Disallow/Allow rules, and crawl delay compliance

README.md

genpark-robots-txt-crawl-delay-policy-evaluator-skill

Agent Skill implementing RFC 9309 Robots.txt Permission & Rate Delay Evaluation in 100% Python standard library.

Architectural Flow

flowchart TD
    Robots["robots.txt File"] --> Parser["Directive Parser (Disallow / Allow / Crawl-delay)"]
    Path["Target URL Path"] --> Matcher["Prefix Path Matcher"]
    Parser & Path --> Matcher
    Matcher --> Allow{"Matches Allow Rule?"}
    Allow -->|Yes| Permit["Permit Crawl Request"]
    Allow -->|No| Disallow{"Matches Disallow Rule?"}
    Disallow -->|Yes| Block["Block Autonomous Agent Access"]
    Disallow -->|No| Permit

Reviews (0)

No results found