From 6783f0122bca630b432a92aee29a9aa7c0f53daf Mon Sep 17 00:00:00 2001 From: mandar1jn Date: Wed, 12 Oct 2022 11:30:20 +0200 Subject: Update robots.txt The old robots.txt didn't disallow any pages and didn't specifically allow any. Its contents had no effect. This removes all the useless content and adds a Sitemap statement to direct web crawlers to the sitemap --- robots.txt | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/robots.txt b/robots.txt index 406a190..345303f 100644 --- a/robots.txt +++ b/robots.txt @@ -1,6 +1,3 @@ # robotstxt.org/ -User-agent: Twitterbot -Disallow: - -User-agent: * +Sitemap: https://raylib.com/sitemap.xml -- cgit v1.2.3