Documents & Resources
Welcome to the BotGuard Knowledge Base
BotGuard Knowledge Base
BotGuard Rules Editor Explained
You have most likely noticed a great decrease in bot/parasitic traffic while using our service with the built-in Core Rules. However, while monitoring your logs via our dashboard, you might still be seeing some visitors whose behavior might seem strange to you. This is where our Custom Rules engine comes into play, and it could help you to get rid of those unwanted visitors for good.
The main two purposes of the Custom Rules engine are:
- Whitelist services and clients that shouldn’t be blocked, but, unfortunately, they are
- Block services and clients that still get access to your site, although they shouldn’t
But, of course, you're free to use it any other way you see fit for your own specific needs.
Here are a few considerations you should keep in mind when making your own Custom Rule:
- Custom Rules have priority over Core Rules and Rulesets.
- Each rule may contain more than one condition.
- When you add more than one condition to the rule, they would be treated with the logical operator ‘AND’, not ‘OR’. This means that the rule will be triggered only when ALL the listed conditions are met.
- There is no option to edit a Custom Rule. However, you could add a new rule and then remove the old one.
Let’s take a look at the match types you can use for making Custom Rules:
- Is / Is not - this one’s simple, it means a strict match
- Contains / Not contains - a partial match to the rule string
- Greater than / Less than - applicable to number values
- Is any of - a strict match to a number of possible values
- Matches to expression - matches a regular expression
We’ve included explanations for each of the categories of conditions listed below:
Internet Service Provider and Autonomous System Number conditions are pretty much the same. In the first case, you should enter the ISP name, in the second case, you enter the AS number. Please use numbers only, do not include a prefix with AS/ASN abbreviations.
User-Agent means the corresponding HTTP header, which is sent by any software that makes a request to your server. You could use this condition to block/allow requests from various browsers, like Google Chrome, Mozilla Firefox (or their mobile versions), CLI tools like wget or curl, programming languages libraries, like Urllib or Requests in Python, Go Http Client, libwww-perl, and so on. You can also try exploring the world of various User-Agents to get more examples via special databases. We recommend checking out whatismybrowser.com or useragentstring.com.
Referrer URL. A browser sends the location of where a link to your site was clicked in a special ‘Referer’ HTTP Header. One common Custom Rule, which might be useful for you, is to block visitors with an empty ‘Referer’ header. When the ‘Referer’ header is empty, it means that the client made a direct request to the site, and didn’t follow any link or perform any search request prior to the visit. Most of the time it’s fine, because it means the user typed your site address from memory, but sometimes it’s an indicator of bot traffic.
HTTP Header condition does about the same as the two headers above, but you could use any HTTP header here, even non-standard ones.
IP Address. This one is quite clear. Our common advice is to whitelist administrator/developer/QA IP addresses - these people often use custom software and send unusual requests to the site, which might be blocked otherwise. Please be sure to whitelist your site’s uptime monitor IP address. This condition supports CIDR notation to list IP subnets, like 18.104.22.168/24. You can also easily list different subnets in a single rule using the ‘Is any of’ match type (separating each subnet with a comma followed by a space, for example: first, second, third, etc).
URI Path condition allows you to grant or deny requests that were sent to certain URLs on your site. You should omit the scheme and domain name to use it. For example, a bot is sending requests to URL ‘https://www.example.net/some/path/resource.html’, in which case you would use ‘/some/path/resource.html’ for the rule. Commonly used with the ‘Match to expression’ match type to cover multiple files and directories via Regular Expressions.
Country. Used to restrict access based on geography.
Domain. Our service automatically protects the subdomains for your main domain name. But sometimes you might need to apply a Custom Rule to some specific subdomains, this is where you should use this condition.