OpenCloudModelBot

purpose

Reading public cloud documentation, slowly.

OpenCloudModel is an academic research project that compares how cloud providers name and describe their services. OpenCloudModelBot reads their public documentation, keeps a dated copy of each page, and records the links between pages.

It does not log in, fill in forms, run scripts, or follow links outside the paths listed below.

identity

How to recognise it

Every request carries these headers. It never presents itself as a browser.

Other software can send the same User-Agent. To tell the real crawler apart, check where the request came from, as described under verify.

behaviour

How it behaves

• robots
  It reads robots.txt and obeys the rules for OpenCloudModelBot, including Crawl-delay.
• pace
  One request at a time to a host, waiting the larger of the Crawl-delay and the pace set for the source below.
• backoff
  On a 429 or 5xx it doubles its delay for that host, up to 600 s, and waits at least as long as a Retry-After header asks. Asked to wait longer than 600 s, it stops and tries again on a later run. A failed request gets at most 2 retries.
• refresh
  For pages it already has, it sends If-None-Match and If-Modified-Since, or skips them when the sitemap lastmod shows no change.
• limits
  It asks for markdown where a site offers it, sends no cookies, and reads at most 64 MiB per response.

control

How to block it

Add this to robots.txt to stop it reading a site, or use a narrower Disallow path to stop it reading part of one:

Or write to crawler@opencloudmodel.org, and the source is removed.

sources

What it reads

• azure-learn

  vendor

  azure

  licence

  CC-BY-4.0

  hosts

  learn.microsoft.com

  paths

  /en-us/azure/

  starts from

  https://learn.microsoft.com/_sitemaps/azure_en-us_1.xml
https://learn.microsoft.com/_sitemaps/azure_en-us_2.xml
https://learn.microsoft.com/_sitemaps/azure_en-us_3.xml
https://learn.microsoft.com/_sitemaps/azure_en-us_4.xml
https://learn.microsoft.com/_sitemaps/azure_en-us_5.xml
https://learn.microsoft.com/_sitemaps/azure_en-us_6.xml
https://learn.microsoft.com/_sitemaps/azure_en-us_7.xml

  pace

  at least 1.0 s between requests

  links

  followed up to 3 steps from a sitemap, only within the paths above

• gcp-docs

  vendor

  gcp

  licence

  CC-BY-4.0

  hosts

  docs.cloud.google.com

  paths

  /

  skipped

  /(java|nodejs|ruby|python|dotnet|php|go|cpp)/docs/reference
/(reference|rest|rpc)/

  starts from

  https://docs.cloud.google.com/sitemap.xml

  pace

  at least 1.0 s between requests

  links

  followed up to 3 steps from a sitemap, only within the paths above

• aws-docs

  vendor

  aws

  licence

  CC-BY-SA-4.0

  hosts

  docs.aws.amazon.com

  paths

  /

  skipped

  ^https://docs\.aws\.amazon\.com/[a-z]{2}_[a-z]{2}/
^https://docs\.aws\.amazon\.com/(AWSCloudFormation|cdk|whitepapers|solutions|prescriptive-guidance|hands-on|code-library|sdkref|sdk-for-[^/]+)/
/(APIReference|apireference|api|cli|relnotes|TemplateReference)/
/example_[^/]*\.html$
/(doc-history|document-history|glossary)\.html$

  starts from

  https://docs.aws.amazon.com/sitemap_index.xml

  pace

  at least 5.0 s between requests

  links

  followed up to 3 steps from a sitemap, only within the paths above

  Kept for private research only: page text from this source is not published; only URLs and data derived from them are.

repositories

What it reads from git

These sources are cloned from their public repositories rather than crawled, at the latest commit, which is recorded with each page.

• azure-git

  vendor

  azure

  licence

  CC-BY-4.0

  repositories

  https://github.com/MicrosoftDocs/azure-docs.git
https://github.com/MicrosoftDocs/azure-compute-docs.git
https://github.com/MicrosoftDocs/azure-aks-docs.git
https://github.com/MicrosoftDocs/azure-management-docs.git
https://github.com/MicrosoftDocs/azure-monitor-docs.git

verify

Where it runs from

Every request comes from one of these addresses:

A genuine request passes two checks:

• reverse DNS
  The requesting address resolves to a name under crawler.opencloudmodel.org, and that name resolves back to the same address.
• address list
  The address falls in a range listed at https://opencloudmodel.org/crawler/ips.json.