mirror of
https://github.com/RooVetGit/Roo-Code.git
synced 2026-09-07 08:26:51 +00:00
5861d8562a
- Add robots.txt file to guide search engine crawlers - Replace static sitemap.xml with dynamic sitemap.ts for all pages - Enhance metadata with keywords, Open Graph, and Twitter Cards - Add comprehensive structured data (Organization, WebSite, SoftwareApplication) - Add metadata to enterprise page - Include documentation for Google Search Console setup These changes will help the Roo Code website appear in Google search results when users search for 'roo code'.
27 lines
No EOL
396 B
Text
27 lines
No EOL
396 B
Text
User-agent: *
|
|
Allow: /
|
|
|
|
# Sitemap
|
|
Sitemap: https://roocode.com/sitemap.xml
|
|
|
|
# Crawl-delay for polite crawling
|
|
Crawl-delay: 1
|
|
|
|
# Allow all major search engines
|
|
User-agent: Googlebot
|
|
Allow: /
|
|
|
|
User-agent: Bingbot
|
|
Allow: /
|
|
|
|
User-agent: Slurp
|
|
Allow: /
|
|
|
|
User-agent: DuckDuckBot
|
|
Allow: /
|
|
|
|
# Block common bot paths that aren't needed
|
|
Disallow: /api/
|
|
Disallow: /_next/
|
|
Disallow: /admin/
|
|
Disallow: *.json$ |