The Robot Text as a Service API enables developers to keeps track of rules for web crawlers or SEO tools on different websites. Developers can use GET or POST requests to check resources to see whether they can be crawled using a given user agent. The alpha release of this API is being offered for free.
The following is a list of sample source code snippets that matched your search term. Source code snippets are chunks of source code that were found out on the Web that you can cut and paste into your own source code. Whereas most of the sample source code we've curated for our directory is for consuming APIs, we occasionally find something interesting on the API provider side of things. If you know of some sample source code that would be of interest to the ProgrammableWeb community, we'd like to know about it. Be sure to check our guidelines for making contributions to ProgrammableWeb.