Introduction

This article addresses the question of how content in a brand community is discoverable by search engines and Large Language Models (LLMs), and how this process contributes to SEO. It provides information to ensure that brand communities are optimized for both traditional web crawlers and AI-driven discovery.

​

Section 1: Brand Communities and Content Discoverability by Search Engines

We follow standard practices for SEO, including optimizing URLs, titles, and descriptions, as well as incorporating relevant keywords.

​

1.1 SEO Attributes for Enhanced Discoverability

The attached sheet details the attributes at the community, group, post, and profile levels that contribute to SEO and greater discoverability by search engines - SEO : Genuin Page Meta Attributes - Sephora

Table 1. Pagewise meta attributes mapping

Above Table 1 outlines SEO and social metadata attributes across different community page types for optimized sharing and indexing.

Table 2. Pagewise Attributes Classification

​

1.2 SEO Compatibility and Benchmarking with PageSpeed Insights

PageSpeed Insights (PSI) results for our community pages highlight strong SEO compatibility. Key metrics such as Largest Contentful Paint (LCP), and Cumulative Layout Shift (CLS) are within the optimal ranges, demonstrating that the platform is built with SEO in mind. These results serve as a benchmark and a testament to our commitment to providing an SEO-friendly environment for brand communities.

Mobile Web Analysis for sephora.begenuin.com :

Desktop Web Analysis for sephora.begenuin.com :

Also a strong Domain Authority (DA) and a robust backlinking strategy will significantly enhance SEO performance.

​

Section 2: Brand Communities Discoverability by LLMs + Content Crawlability

​

2.1 Content Crawlability on Search Consoles

• Question: Is all content in the communities helping SEO/crawlable, or only certain types of videos (i.e. just User Generated Content - UGC)?

• Answer: All content within the communities is crawlable. After white labeling the brand will submit the sitemap of community, group, post, and profile pages to search consoles viz. Google,Bing, Brave and DuckDuckGo for crawling, provided that the robots.txt is configured correctly.

​

Detailed steps for sitemap submission to major search consoles:

1. ​

   Google Search Console

• a. Go to Google Search Console and sign in with a Google account.

• b. Add your website as a property by entering your domain and verifying ownership (e.g., via DNS record, HTML file upload, or Google Analytics).

• c. Once verified, navigate to “Sitemaps” under the “Indexing” section in the sidebar.

• d. Enter the URL of your sitemap (e.g., https://example.com/sitemap.xml) and click “Submit.”

• e. Google will process the sitemap and show its status (e.g., success, errors).

2. ​

   Bing Webmaster Tools

• a. Go to Bing Webmaster Tools and sign in with a Microsoft account.

• b. Navigate to “Sitemaps” and click on “Submit Sitemap”

• c. Add your website by entering the URL and click “Submit.”

• d. Bing will validate and process the sitemap.

2.1 By Importing from Google Search Console

Bing Webmaster Tools also lets you import any existing verified properties from Google Search Console. So if your website is already verified at Google Search Console, then you can import your site to Bing Webmaster Tools with the following steps.

• a. Import Your Sites from GSC : On the Welcome page, click the “Import” button available under “Import your sites from GSC” option.

• b. Allow Permissions : On the next page, Bing would request your confirmation to fetch data from your Google Search Console and import them to Bing Webmaster Tools. Click “Continue”.

• c. Next, Google would request you to authenticate and then request your permission to let Bing access the Google Search Console data of your verified properties. Click “Allow”.

• d. Select Your Sites to Import : Now Bing would fetch all the verified websites available in your Google Search Console account, and here you can select the sites that you want to import. You can also notice from the below image, Bing was able to fetch the sitemap from your Google Search Console account. Select the sites to be imported and then click the “Import” button.

• e. Check Imported Sitemap : Once Bing has successfully imported your website’s data, you’ll see a success notice, as shown below.

• f. Click “Done”, and now you can confirm if Bing has imported your sitemap, by navigating to the “Sitemaps” section from the left sidebar.

3. ​

   Brave

• a. Go to Brave Submit URL.

• b. Add your site.

• c. Click Submit.

4. ​

   DuckDuckGo

• a. DuckDuckGo primarily relies on other search engines’ indexes (especially Bing) for website discovery. There is no dedicated sitemap submission portal.

• b. To ensure your site is indexed by DuckDuckGo, submit your sitemap to Bing Webmaster Tools (as detailed above). DuckDuckGo will discover your site through Bing’s index.

• c. Additionally, ensure your site is linked from other reputable websites to aid discovery.

​

2.2 Offsite Optimization

• Question: How else can our community content positively impact offsite optimization related to SEO and traffic building?

• Answer: The existing framework supports offsite optimization by facilitating backlinking and social media engagement, aligning with industry standards for enhancing SEO and driving traffic.

​

2.3 Technical Aspects for LLM SEO (LEO)

​

2.3.1 Crawlability of Pages

• Question: Is the content crawlable by LLM?

• Answer: Yes, all six types of pages are crawlable:

a) Community Page

b) Group Page

c) Post Page

d) User Profile

e) Brand Profile

f) Subdomain Home Page

​

2.3.2 OpenAI Crawlers

OpenAI uses web crawlers (“robots”) and user agents to gather data for its products. Webmasters can control how their sites interact with OpenAI by using robots.txt tags. Each setting is independent.

• Reference: OpenAI Official Documentation

• Recommendation: The client’s webmaster should update the website’s robots.txt file according to OpenAI’s official documentation. It is recommended to enable crawling for all OpenAI user agents to maximize LLM SEO and discoverability.

• Robots.txt Information: For more information on robots.txt, refer to Google’s documentation.

​

2.3.2.1 Steps to Configure “robots.txt” for OpenAI:

The following robots.txt directives can be used to manage OpenAI’s access to your site:

• 1. User-agent: Specify the OpenAI user agent (e.g., GPTBot, ChatGPT-User).
• 2. Disallow: Block OpenAI from accessing certain paths or the entire site.
• 3. Allow: Explicitly allow OpenAI to access specific paths.

Example robots.txt configuration:

**User-agent**: *  
**Disallow**: /private/  
**User-agent**: GPTBot  
**Allow**: /

This configuration blocks all user agents from the /private/ directory, but allows the GPTBot to access the rest of the site, where GPTBot will use the data to feed their foundational AI models.

​

2.3.3 Perplexity Crawlers

Perplexity uses web crawlers (“robots”) and user agents to collect and index information from the internet. Webmasters can manage how their sites interact with Perplexity using robots.txt tags. Changes may take up to 24 hours to be reflected.

• Reference: Perplexity Official Documentation

• Recommendation: The client’s webmaster should update the website’s robots.txt file as per Perplexity’s official documentation to ensure optimal LLM SEO and discoverability.

​

2.3.3.1 Steps to Configure robots.txt for Perplexity:

These robots.txt directives can control how Perplexity interacts with your site:

1. User-agent: Identify Perplexity’s user agent (e.g., PerplexityBot).

2. Disallow: Prevent Perplexity from accessing certain URLs or the entire site.

3. Allow: Specifically permit Perplexity to access designated URLs.

Example robots.txt for Perplexity:

**User-agent**: *  
**Disallow**: /admin/  
**User-agent**: PerplexityBot  
**Allow**: /

In this example, all bots are blocked from the /admin/ section, while PerplexityBot is allowed to crawl the rest of the website.

​

2.4 llms.txt File

To provide LLM-friendly content, as per industry standards adding an /llms.txt markdown file to websites. This file offers background information, guidance, and links to detailed markdown files.

• Reference: LLMs-TXT Official Documentation

• Discover websites using llms.txt: llms.txt directory

Additional References:

• LLMs.txt Explained | TDS Archive

• LLMs.txt Explained | Towards Data Science

​

2.4.1 Steps to Implement /llms.txt:

1. Create the /llms.txt file: Create a new markdown file named llms.txt at the root directory of your website.

2. Add basic information: Include a brief introduction about your website and its purpose.

3. Provide guidance to LLMs: Explain how you would like LLMs to interact with your content (e.g., prioritize certain sections, avoid specific areas).

4. Link to detailed markdown files: If you have more detailed information, create additional markdown files and link to them from /llms.txt. This can include site maps, content guidelines, etc.

5. Publish the file: Ensure the llms.txt file is accessible via a web browser by navigating to yourdomain.com/llms.txt.

6. Maintain the file: Regularly update the file to reflect any changes in your website structure or content strategy.

​

2.5 llms-full.txt File

According to ideal practices, adding an /llms-full.txt markdown file to websites is recommended. This file offers detailed background information, comprehensive guidance, and extensive links to markdown files.

• It should contain all the detailed content in one place, allowing LLMs to access all the information without needing to navigate to multiple pages.

2.5.1 Steps to Implement /llms-full.txt:

1. Create the /llms-full.txt file: Create a new markdown file named llms-full.txt at the root directory of your website.

2. Add detailed basic information: Include an extensive introduction about your website and its purpose, covering all aspects relevant to LLMs.

3. Provide comprehensive guidance to LLMs: Explain in detail how you would like LLMs to interact with your content, including specific instructions for different types of content, sections, and potential issues to avoid.

4. Link to extensive markdown files: Create detailed markdown files with comprehensive information, and link to them from /llms-full.txt. This can include site maps, content guidelines, technical specifications, and any other relevant documentation.

5. Publish the file: Ensure the /llms-full.txt file is accessible via a web browser by navigating to yourdomain.com/llms-full.txt.

6. Maintain the file: Regularly update the file to reflect any changes in your website structure, content strategy, or technical specifications.

​

2.6 Submitting /llms.txt and /llms-full.txt to sitemap.xml

While /llms.txt and /llms-full.txt are not traditional web pages meant for direct user consumption, including them in your sitemap can provide clearer guidance to LLMs about the availability and purpose of these files. This can improve LLM discoverability and understanding of your site’s content.

​

2.6.1 Steps to Submit /llms.txt and /llms-full.txt to sitemap.xml:

1. Create or update your sitemap.xml file:

• If you already have a sitemap.xml file, locate it in the root directory of your website.

• If not, create a new XML file named sitemap.xml in the root directory.

2. Add <url> entries for /llms.txt and /llms-full.txt:

• Open your sitemap.xml file and add a <url> element for each file, similar to how you would for any other page on your site.

• Example:

NOTE:

• a. Replace “https://yourwebsite.com/llms.txt” and “https://yourwebsite.com/llms-full.txt” with the actual URLs of your files.

• b. The <lastmod> tag should contain the date of the last modification of the file.

• c. The <changefreq> tag indicates how often the file is expected to change.

• d. The <priority> tag indicates the relative importance of this URL compared to other URLs on your site.

3. Submit your sitemap.xml to search engines:

• Submit the updated sitemap.xml to search consoles through their respective webmaster tools. This process is detailed in Section 2.1.

2.7.2 Impact of Submitting /llms.txt and /llms-full.txt:

• e. Improved Discoverability for LLMs: By including these files in your sitemap, you explicitly inform LLMs about their existence and location. This can help LLMs find and process these files more efficiently.

• f. Better Understanding of your Intent: Including these files in the sitemap signals to LLMs that you consider these files to be an important part of your site’s structure and that you want them to be accessed and understood.

• g. Enhanced LLM SEO: While the direct impact on traditional search engine rankings may be minimal, providing clear guidance to LLMs can contribute to a more comprehensive and accurate understanding of your website’s content, which may indirectly improve your site’s overall LLM SEO (LEO).

​

2.7 Best Practices

​

2.7.1 Generative Engine Optimization (GEO)

Generative Engine Optimization (GEO) involves optimizing content to rank higher in AI-driven search engines that synthesize information from multiple sources.

• Key Principle: Ensure content is well-structured and engaging to facilitate easy access, interpretation, and prioritization by AI systems.

• Structured Curation: Organize communities and groups logically to aid LLMs and search engines in understanding and navigating the content.

• Reference: https://searchengineland.com/generative-engine-optimization-strategies-446723#h-1-generative-ai-research-and-analysis

​

2.7.2 Answer Engine Optimization (AEO)

• Answer Engine Optimization (AEO) focuses on optimizing content to appear as direct answers in search engines, AI-driven assistants, and voice search results.

• The llms-full.txt file can significantly support AEO efforts by providing LLMs with detailed, structured information that facilitates the extraction of direct answers.

• It makes it easier for LLMs to understand and utilize your information when generating answers to user queries. This can improve your chances of appearing as a featured snippet or direct answer in search results, ultimately enhancing your website’s visibility and attracting more targeted traffic.

• Reference: https://surferseo.com/blog/answer-engine-optimization/

All the community pages are inherently friendly to both traditional search engines and LLMs. By implementing the technical configurations outlined and adhering to the recommended best practices, you can maximize discoverability, improve SEO performance, and effectively engage with AI-driven search.