Skillget

genpark-robots-txt-crawl-delay-policy-evaluator-skill

mcp
Guvenlik Denetimi
Uyari
Health Uyari
  • No license — Repository has no license file
  • Description — Repository has a description
  • Active repo — Last push 0 days ago
  • Low visibility — Only 7 GitHub stars
Code Gecti
  • Code scan — Scanned 6 files during light audit, no dangerous patterns found
Permissions Gecti
  • Permissions — No dangerous permissions requested

Bu listing icin henuz AI raporu yok.

SUMMARY

RFC 9309 robots.txt parser with path permissions, Disallow/Allow rules, and crawl delay compliance

README.md

genpark-robots-txt-crawl-delay-policy-evaluator-skill

Agent Skill implementing RFC 9309 Robots.txt Permission & Rate Delay Evaluation in 100% Python standard library.

Architectural Flow

flowchart TD
    Robots["robots.txt File"] --> Parser["Directive Parser (Disallow / Allow / Crawl-delay)"]
    Path["Target URL Path"] --> Matcher["Prefix Path Matcher"]
    Parser & Path --> Matcher
    Matcher --> Allow{"Matches Allow Rule?"}
    Allow -->|Yes| Permit["Permit Crawl Request"]
    Allow -->|No| Disallow{"Matches Disallow Rule?"}
    Disallow -->|Yes| Block["Block Autonomous Agent Access"]
    Disallow -->|No| Permit

Yorumlar (0)

Sonuc bulunamadi