In addition to good bots (user-agents) , such as GoogleBot, which scan the site for indexing in search engines, there are also malicious bots whose purpose is to collect statistics on the site for third parties, parse prices for goods and services, search for hidden pages, vulnerabilities, etc.
Malicious bots usually do not carry a payload for site owners, but they can cause many problems, such as generating an unreasonable load on the hosting account and spamming the site, so in order to prevent and anticipate such situations, the virtual hosting control panel blocks malicious bots by default to the site.
If necessary, use the following steps to unblock or block a malicious bot that is not on the default list on the site:
1. Open the cp.cityhost.ua control panel and go to hosting management.
2. Go to the settings of the desired site and click the Security button
3. In the Blocked user-agents section, click the Edit button
4. In the window that opens, you can manage the list of bots that will not have access to the site.
You can add your own bot by clicking on the field at the end of the list, then enter the name of the bot (its user-agent) and apply the changes with the Save button. From this moment on, the bot does NOT have access to the site, you can make sure of this by executing the curl command from the example below.
The character case does not play a role when writing - writing
TeStBoT will work the same. Allowed characters
Az 0-9 / . , ( ) : ; + - (пробіл) . The correct bot user-agent can be obtained from the visit log of your site, or at user-agents.net/bots
Opposite the name (user-agent) of the bot, click on the cross icon and apply the changes with the Save button. From this moment on, the bot HAS access to the site, you can make sure of this by executing the curl command from the example below.
If you have accidentally or intentionally cleared the bot list, you can restore the default values and enable bot blocking by clicking the "Restore Defaults" button.
You can check the blocking status of a specific bot (user-agent) by executing the curl command through any terminal, for example, in WEB-SSH on the hosting.
Run the following command after substituting your own data:
curl -I "https://domain.zone" -A "user-agent"
https://domain.zone ? your site specifying the protocol
user-agent ? bot ID
If the bot is blocked,
curl: (56) Recv failure: Connection reset by peer or
curl: (92) HTTP/2 stream 0 was not closed cleanly will be returned. If the bot is not blocked, a response like
HTTP/1.1 200 OK or any other positive status will be returned.
All question categories