openvibers/openvibe.codes Commits

git.openvibe.codes: Meta's AI crawler gets robots.txt and nothing else (#255)

OP OpenVibers committed 11 days ago · 70440c72da · 1 file, +8 −0

meta-externalagent made 97% of the forge's requests (about two a second,
every commit of every repository; 8% of a core in Forgejo) past the default
robots.txt's wildcard rules. Forgejo's custom robots.txt now names it with
Disallow: /, and until it reads that, nginx answers it 403 everywhere but
/robots.txt. Browsers and facebookexternalhit (link previews) are
unaffected. Installed on the host (backup .bak-20261102-meta) and reloaded.

Changes

ddiff --git a/deploy/nginx/openvibe.codes.conf b/deploy/nginx/openvibe.codes.conf
iindex fc85211..6e5a1cc 100644
---- a/deploy/nginx/openvibe.codes.conf
++++ b/deploy/nginx/openvibe.codes.conf
@@@ -154,6 +154,14 @@ server {
      # Forgejo's own Prometheus endpoint stays off the internet (app.ini leaves [metrics] at its default, off).
      location ~* ^/metrics(/|$) { return 404; }
  
++    # Meta's AI crawler (meta-externalagent) walked every commit of every repository past robots.txt's wildcard rules:
++    # 97% of this site's requests, about two a second, and 8% of a core in Forgejo (2026-11-02). robots.txt now names
++    # it (Disallow: /, in Forgejo's custom/public/robots.txt); until it reads that, it gets robots.txt and nothing else.
++    set $meta_crawler 0;
++    if ($http_user_agent ~* "meta-externalagent") { set $meta_crawler 1; }
++    if ($uri = /robots.txt) { set $meta_crawler 0; }
++    if ($meta_crawler) { return 403; }
++
      location / {
          client_max_body_size 512m;
          proxy_request_buffering off;