HQY 一个和谐有爱的空间

解决屏蔽流氓蜘蛛抓取，如MJ12bot 、DotBot 、BLEXBot 、PetalBot 、DataForSeoBot

1、robots文件屏蔽

User-agent: MJ12bot

Disallow: /

User-agent:DotBot

Disallow: /

User-agent:BLEXBot

Disallow: /

User-agent:PetalBot

Disallow: /

User-agent:DataForSeoBot

Disallow: /

2、Nginx等服务器规则屏蔽

if ($http_user_agent ~*  （MJ12bot|DotBot|BLEXBot|PetalBot|DataForSeoBot） )

{

    return 403;

}

注意：在前期如果流量少，可以设置下防火墙，屏蔽掉轰炸式蜘蛛，蜘蛛来访也可以增加收录率，所以根据自己需求来屏蔽

if ($http_user_agent ~* (YandexBot|spbot|DnyzBot|Researchscan|semrushbot|yahoo|AhrefsBot|DotBot|Uptimebot|MJ12bot|MegaIndex.ru|ZoominfoBot|Mail.Ru|SeznamBot|BLEXBot|ExtLinksBot|aiHitBot|Barkrowler)){

    return 403;

}

推荐本站淘宝优惠价购买喜欢的宝贝:

本文链接：https://www.hqyman.cn/post/11009.html 非本站原创文章欢迎转载，原创文章需保留本站地址！

休息一下~~