Skip to main content

robots.txt

  1. 在publuc目录下
  2. 通过<IP>/robots.txt访问 # Block all crawlers for / 防止爬虫爬取的地方 # Allow all crawlers 允许爬虫爬取的地方
# Block all crawlers for /accounts  
  
# Allow all crawlers  
User-agent: *  
Allow: /