The Torrere reader

What the request with Torrere/1.0 in its user agent is, and how to refuse it.

Torrere is a recipe app. It turns a photo of a recipe card, a link or pasted text into steps you can cook from. When someone using it pastes the address of a recipe page, Torrere reads that one page on their behalf, so the recipe on it can be cooked from without copying it out by hand.

What it does

What it does not do

Refusing it

Refuse it by name and it stays away. In robots.txt:

User-agent: Torrere
Disallow: /

Torrere asks your robots.txt for that rule before it reads a page, and again if a redirect takes it somewhere else. It keeps the answer for an hour, so adding the rule stops it the same morning. A rule naming Torrere wins over anything in your User-agent: * group, in both directions: close the door to everyone and open it for Torrere, and Torrere comes in.

One thing to know, because it is the honest version: if your robots.txt cannot be read at all, a server error or a timeout rather than a rule, Torrere reads the page. A crawler about to make thousands of requests should back off there. This is one person opening one link, and failing them because a file was briefly down helps nobody. A rule Torrere can read is always honoured.

Or block the user agent at your edge. Either way, people who paste a link to your site are told it could not be read, and are not shown your content.

Questions

Write to support@torrere.com. One person reads it.