Skip to content

Discover how to easily access the Net Work page list

The XML sitemap remains the most reliable way to get a comprehensive view of the URLs published on a site. On Net Work, this file…

Femme professionnelle naviguant sur la liste des pages d'un site web professionnel sur un ordinateur portable au bureau

The XML sitemap remains the most reliable way to obtain a comprehensive view of the URLs published on a site. On Net Work, this file is accessible at a standardized address, but its use goes beyond simple consultation in a browser.

XML Sitemap and robots.txt: Two Complementary Files for Mapping Net Work

An XML sitemap lists the URLs that a publisher wants search engines to explore. It does not guarantee indexing or ranking, but it provides a structured inventory.

The robots.txt file governed by RFC 9309 (published in September 2022 by the IETF) formalizes the Robots Exclusion Protocol. The Sitemap: directive that is included is a complementary convention, widely supported by Google and Bing, but it is not part of the core of this RFC. In other words, declaring the sitemap in robots.txt helps cooperative crawlers locate it, without constituting an access control mechanism.

To directly access the list of Net Work pages, simply add the path /sitemap/ to the main domain. Depending on the CMS configuration, this entry point returns either a single XML file or an index of sitemaps divided by content type (articles, pages, categories).

What the Sitemap Does Not Show

URLs blocked by a Disallow directive in robots.txt or marked noindex generally do not appear in the sitemap. Orphan pages, meaning those that do not receive any internal links, may also be absent if the sitemap generator relies solely on internal linking.

We recommend cross-referencing the sitemap with a complete crawl of the site to detect these discrepancies. An SEO spider tool scans the actual link structure and reveals the URLs that the sitemap ignores.

Man working remotely consulting the complete list of pages of a professional network on a large desktop screen

Lastmod Tag and Google Ping: What Has Changed Since 2023

Google has deprecated the sitemap ping via HTTP. The old notification endpoint is no longer effective. The recommended method now is to declare the sitemap in Google Search Console, and then to provide an exact lastmod field in each XML entry.

A common pitfall: regenerating the sitemap updates the lastmod date of all URLs, even those whose content has not changed. Google expects a date corresponding to a significant content change, not just the simple reconstruction of the file. A lastmod that is systematically reset ends up being ignored by the crawler.

IndexNow as a Supplement, Not a Substitute

The IndexNow protocol allows for quick notification to Bing and participating engines that a URL has been modified. We observe a recurring confusion: IndexNow does not guarantee the crawling or indexing of the notified page. It speeds up potential consideration, without replacing the declaration of the sitemap in webmaster tools.

For a site like Net Work, combining the submission of the sitemap in Search Console, the Sitemap: directive in robots.txt, and the integration of IndexNow covers the three main discovery channels.

Utilizing the List of Net Work Pages for an Internal Linking Audit

Consulting the sitemap in a browser displays a raw XML file, which is difficult to read. Several approaches make the analysis usable.

  • Import the sitemap into a spreadsheet (Google Sheets, Excel) via the XML import function to filter URLs by directory, last modified date, or declared update frequency.
  • Load the sitemap URL into a dedicated SEO crawler, which compares the listed pages with those actually discovered by the crawl and reports on orphaned pages or redirects.
  • Use the search operator site:net-work.fr in Google to compare the number of indexed results with the number of URLs present in the sitemap, and identify any indexing discrepancies.

The gap between listed pages and indexed pages serves as a direct indicator of the technical health of the site. A sitemap containing URLs with a 404 error or a 301 redirect sends a negative signal to crawling bots.

Two colleagues consulting together the list of pages of a professional network on a tablet in a coworking space

Cleaning the Sitemap: Entries to Remove

We recommend removing from the sitemap URLs that return an HTTP code other than 200. Paged pages without unique content, empty tag archives, and URLs with tracking parameters have no place in this file.

A clean sitemap, limited to canonical pages and up to date, improves the crawl budget allocated by engines. On a medium-sized site like Net Work, this impact remains modest, but a noise-free sitemap facilitates diagnosis during a technical audit.

Robots.txt and AI Crawlers: A Distinction Not to Be Overlooked

RFC 9309 specifies that robots.txt constitutes a request addressed to cooperative robots. A non-cooperative crawler, particularly some language model training bots, may ignore these directives without technical consequence. The file does not protect a list of confidential pages.

For publishers who wish to control access for GPTBot or other AI agents, the only reliable approach remains to combine the Disallow directive in robots.txt with server-side authentication mechanisms. The sitemap exposes the complete structure of the site: any URL that appears there is by definition intended to be discovered.

Before publishing a sitemap, we recommend checking that it does not contain URLs for administration, staging pages, or internal resources. This simple reflex avoids offering a detailed site map to unwanted agents.

Accessing the list of a site’s pages via its XML sitemap is not limited to opening a URL. The file becomes an audit tool when cross-referenced with a crawl, monitoring the consistency of the lastmod, and cleaning up outdated entries. On Net Work, this approach remains accessible without advanced technical skills, provided one knows what to look for in the file.

Discover how to easily access the Net Work page list