​

Section 1: Brand Communities and Content Discoverability by Search Engines

​

1.1 SEO Attributes for Enhanced Discoverability

​

1.2 SEO Compatibility and Benchmarking with PageSpeed Insights

​

Section 2: Brand Communities Discoverability by LLMs + Content Crawlability

​

2.1 Content Crawlability on Search Consoles

• Question: Is all content in the communities helping SEO/crawlable, or only certain types of videos (i.e. just User Generated Content - UGC)?
• Answer: All content within the communities is crawlable. After white labeling the brand will submit the sitemap of community, group, post, and profile pages to search consoles viz. Google,Bing, Brave and DuckDuckGo for crawling, provided that the robots.txt is configured correctly.

​

Detailed steps for sitemap submission to major search consoles:

1. ​

   Google Search Console

• a. Go to Google Search Console and sign in with a Google account.
• b. Add your website as a property by entering your domain and verifying ownership (e.g., via DNS record, HTML file upload, or Google Analytics).
• c. Once verified, navigate to “Sitemaps” under the “Indexing” section in the sidebar.

• d. Enter the URL of your sitemap (e.g., https://example.com/sitemap.xml) and click “Submit.”
• e. Google will process the sitemap and show its status (e.g., success, errors).

2. ​

   Bing Webmaster Tools

• a. Go to Bing Webmaster Tools and sign in with a Microsoft account.

• b. Navigate to “Sitemaps” and click on “Submit Sitemap”

• c. Add your website by entering the URL and click “Submit.”

• d. Bing will validate and process the sitemap.

2.1 By Importing from Google Search Console

• a. Import Your Sites from GSC : On the Welcome page, click the “Import” button available under “Import your sites from GSC” option.

• b. Allow Permissions : On the next page, Bing would request your confirmation to fetch data from your Google Search Console and import them to Bing Webmaster Tools. Click “Continue”.

• c. Next, Google would request you to authenticate and then request your permission to let Bing access the Google Search Console data of your verified properties. Click “Allow”.

• d. Select Your Sites to Import : Now Bing would fetch all the verified websites available in your Google Search Console account, and here you can select the sites that you want to import. You can also notice from the below image, Bing was able to fetch the sitemap from your Google Search Console account. Select the sites to be imported and then click the “Import” button.

• e. Check Imported Sitemap : Once Bing has successfully imported your website’s data, you’ll see a success notice, as shown below.

• f. Click “Done”, and now you can confirm if Bing has imported your sitemap, by navigating to the “Sitemaps” section from the left sidebar.

3. ​

   Brave

• a. Go to Brave Submit URL.

• b. Add your site.
• c. Click Submit.

4. ​

   DuckDuckGo

• a. DuckDuckGo primarily relies on other search engines’ indexes (especially Bing) for website discovery. There is no dedicated sitemap submission portal.
• b. To ensure your site is indexed by DuckDuckGo, submit your sitemap to Bing Webmaster Tools (as detailed above). DuckDuckGo will discover your site through Bing’s index.
• c. Additionally, ensure your site is linked from other reputable websites to aid discovery.

​

2.2 Offsite Optimization

• Question: How else can our community content positively impact offsite optimization related to SEO and traffic building?
• Answer: The existing framework supports offsite optimization by facilitating backlinking and social media engagement, aligning with industry standards for enhancing SEO and driving traffic.

​

2.3 Technical Aspects for LLM SEO (LEO)

​

2.3.1 Crawlability of Pages

• Question: Is the content crawlable by LLM?
• Answer: Yes, all six types of pages are crawlable:

​

2.3.2 OpenAI Crawlers

• Reference: OpenAI Official Documentation
• Recommendation: The client’s webmaster should update the website’s robots.txt file according to OpenAI’s official documentation. It is recommended to enable crawling for all OpenAI user agents to maximize LLM SEO and discoverability.
• Robots.txt Information: For more information on robots.txt, refer to Google’s documentation.

​

2.3.2.1 Steps to Configure “robots.txt” for OpenAI:

• 1. User-agent: Specify the OpenAI user agent (e.g., GPTBot, ChatGPT-User).
• 2. Disallow: Block OpenAI from accessing certain paths or the entire site.
• 3. Allow: Explicitly allow OpenAI to access specific paths.

**User-agent**: *  
**Disallow**: /private/  
**User-agent**: GPTBot  
**Allow**: /

​

2.3.3 Perplexity Crawlers

• Reference: Perplexity Official Documentation
• Recommendation: The client’s webmaster should update the website’s robots.txt file as per Perplexity’s official documentation to ensure optimal LLM SEO and discoverability.

​

2.3.3.1 Steps to Configure robots.txt for Perplexity:

1. User-agent: Identify Perplexity’s user agent (e.g., PerplexityBot).
2. Disallow: Prevent Perplexity from accessing certain URLs or the entire site.
3. Allow: Specifically permit Perplexity to access designated URLs.

**User-agent**: *  
**Disallow**: /admin/  
**User-agent**: PerplexityBot  
**Allow**: /

​

2.4 llms.txt File

• Reference: LLMs-TXT Official Documentation
• Discover websites using llms.txt: llms.txt directory

• LLMs.txt Explained | TDS Archive
• LLMs.txt Explained | Towards Data Science

​

2.4.1 Steps to Implement /llms.txt:

1. Create the /llms.txt file: Create a new markdown file named llms.txt at the root directory of your website.
2. Add basic information: Include a brief introduction about your website and its purpose.
3. Provide guidance to LLMs: Explain how you would like LLMs to interact with your content (e.g., prioritize certain sections, avoid specific areas).
4. Link to detailed markdown files: If you have more detailed information, create additional markdown files and link to them from /llms.txt. This can include site maps, content guidelines, etc.
5. Publish the file: Ensure the llms.txt file is accessible via a web browser by navigating to yourdomain.com/llms.txt.
6. Maintain the file: Regularly update the file to reflect any changes in your website structure or content strategy.

​

2.5 llms-full.txt File

• It should contain all the detailed content in one place, allowing LLMs to access all the information without needing to navigate to multiple pages.

1. Create the /llms-full.txt file: Create a new markdown file named llms-full.txt at the root directory of your website.
2. Add detailed basic information: Include an extensive introduction about your website and its purpose, covering all aspects relevant to LLMs.
3. Provide comprehensive guidance to LLMs: Explain in detail how you would like LLMs to interact with your content, including specific instructions for different types of content, sections, and potential issues to avoid.
4. Link to extensive markdown files: Create detailed markdown files with comprehensive information, and link to them from /llms-full.txt. This can include site maps, content guidelines, technical specifications, and any other relevant documentation.
5. Publish the file: Ensure the /llms-full.txt file is accessible via a web browser by navigating to yourdomain.com/llms-full.txt.
6. Maintain the file: Regularly update the file to reflect any changes in your website structure, content strategy, or technical specifications.

​

2.6 Submitting /llms.txt and /llms-full.txt to sitemap.xml

​

2.6.1 Steps to Submit /llms.txt and /llms-full.txt to sitemap.xml:

1. Create or update your sitemap.xml file:

• If you already have a sitemap.xml file, locate it in the root directory of your website.
• If not, create a new XML file named sitemap.xml in the root directory.

2. Add <url> entries for /llms.txt and /llms-full.txt:

• Open your sitemap.xml file and add a <url> element for each file, similar to how you would for any other page on your site.
• Example:

• a. Replace “https://yourwebsite.com/llms.txt” and “https://yourwebsite.com/llms-full.txt” with the actual URLs of your files.
• b. The <lastmod> tag should contain the date of the last modification of the file.
• c. The <changefreq> tag indicates how often the file is expected to change.
• d. The <priority> tag indicates the relative importance of this URL compared to other URLs on your site.

3. Submit your sitemap.xml to search engines:

• Submit the updated sitemap.xml to search consoles through their respective webmaster tools. This process is detailed in Section 2.1.

• e. Improved Discoverability for LLMs: By including these files in your sitemap, you explicitly inform LLMs about their existence and location. This can help LLMs find and process these files more efficiently.
• f. Better Understanding of your Intent: Including these files in the sitemap signals to LLMs that you consider these files to be an important part of your site’s structure and that you want them to be accessed and understood.
• g. Enhanced LLM SEO: While the direct impact on traditional search engine rankings may be minimal, providing clear guidance to LLMs can contribute to a more comprehensive and accurate understanding of your website’s content, which may indirectly improve your site’s overall LLM SEO (LEO).

​

2.7 Best Practices

​

2.7.1 Generative Engine Optimization (GEO)

• Key Principle: Ensure content is well-structured and engaging to facilitate easy access, interpretation, and prioritization by AI systems.
• Structured Curation: Organize communities and groups logically to aid LLMs and search engines in understanding and navigating the content.
• Reference: https://searchengineland.com/generative-engine-optimization-strategies-446723#h-1-generative-ai-research-and-analysis

​

2.7.2 Answer Engine Optimization (AEO)

• Answer Engine Optimization (AEO) focuses on optimizing content to appear as direct answers in search engines, AI-driven assistants, and voice search results.
• The llms-full.txt file can significantly support AEO efforts by providing LLMs with detailed, structured information that facilitates the extraction of direct answers.
• It makes it easier for LLMs to understand and utilize your information when generating answers to user queries. This can improve your chances of appearing as a featured snippet or direct answer in search results, ultimately enhancing your website’s visibility and attracting more targeted traffic.
• Reference: https://surferseo.com/blog/answer-engine-optimization/