Script to process apache log file to fight spammers / DDOS attackers
https://www.zyxware.com/sites/default/files/styles/user_image/public/pictures/zyxlogo.png?itok=NCFkMd_w
BY webmaster
7 years ago
Drupal-Technical

One of the challenges with working with dynamic websites is that you have to keep fighting malicious users who regularly sap your server capacity with rogue crawling of your site. To do this you would have to monitor and analyze the traffic patterns on the server regularly. You would definitely want to do this when you have load spikes on the server and you wish to find out the IP addresses, user agents and the specific URLs which resulted in these spikes. This is all the more relevant in Drupal sites where a rogue bot can take down the site when proper DDOS mechanisms are not set in place.

The latest copy of the script can be downloaded from

https://github.com/zyxware/misc-utils/tree/master/ls-httpd

You can copy the script to /usr/local/bin or into some folder which is in your $PATH variable on the server. Remember to configure the script with the path to your apache access log. You can update the default value of the variable log_file to wherever your apache log file is located. Also do note that the script was written for the specific log file format used in our servers. You might want to tweak the awk parameters if your apache log file uses a different format.

Alternatively you can use the following as your apache log file format in apache.conf

LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined

and then ensure that your log format is set up as combined in your virtualhost configuration

CustomLog ${APACHE_LOG_DIR}/access.log combined

The following are some example usage patterns

ls-httpd url 1000

will find top URLs in the last 1000 access log entries

ls-httpd ip 1000

will find top IPs in the last 1000 access log entries

ls-httpd agent 1000

will find top user agents in the last 1000 access log entries

ls-httpd url 17:

will find top URLs from 17:00:00 to 17:59:59

ls-httpd url 17:2

will find top URLs from 17:20:00 to 17:29:59

ls-httpd url 17:21

will find top URLs from 17:21:00 to 17:21:59

ls-httpd url 17

will find top URLs in the last 17 access log entries :-)

Hope you find this useful.


RELATED ARTICLE

/themes/custom/zyxpro_light/images/placeholder.png
close

on 01st January 2008 / by webmaster
We have added a few new modules to our site recently. Drupal Technical StumbleThis Sean (not verified) access_time 31 Jul 2021 - 11:45 Hi there, I uploaded the files, enabled the modules to use SU, and now I see the link, but there is no icon... How can I fix this? Thanks, Sean webmaster access_time 31 Jul 2021 - 11:45 In reply to Icon doesn't show by Sean (not verified) What do you see when you view the source? Search for stumblethis_button and you should be able to see the code for the image and the URL. Then you should be able to troubleshoot from there. Juicy Couture Addict (not verified) access_time 31 Jul 2021 - 11:45 thanks for the post. would love to hear more of you. by the way, drupal's really popular nowadays as it has easy and fast features. you agree with me? thanks. Pagination Current page 1 Page 2 Page 3 Page 4 Next page Next › Last page Last »
/themes/custom/zyxpro_light/images/placeholder.png
close

on 16th January 2008 / by webmaster
Most webmasters do not realize this, but a lot of the content on lot of websites can be accessed from multiple URLs. A simple example would be where www.example.com and example.com leads to the same page. This is a fatal mistake in Search Engine Optimization and search engines penalize you for duplicate content. The correct configuration would be where the above two urls will lead you to the same page but example.com will redirect you to www.example.com with a 301 (Moved permanently) status which will not result in search engines penalizing the page. It is very easy to configure 301 redirects using Apache .htaccess file and the process is the same for a Drupal installation also. Drupal Technical Apache htaccess SEO Anonymous (not verified) access_time 31 Jul 2021 - 11:45 Hello. I'm trying to make example.com show as www.example.com, and I'm running into difficulties. I'm on Apache 2.0 and using the following lines in my httpd.conf file: RewriteEngine on RewriteCond %{HTTP_HOST} ^xxxxxxxxxx\.com$ [NC] RewriteRule ^(.*)$ http://www.xxxxxxxxxx.com/$1 [L,R=301] When I go to http://xxxxxxxxxx.com, I get http://www.xxxxxxxxxx.com (as expected). However, when I go to http://xxxxxxxxxx.com/node/1 (it's a Drupal site), I get a 404 thrown and the URL changes to xxxxxxxxxx.com/var/www/drupal/node/1. Same thing with www.xxxxxxxxxx.com/node/1. Any suggestions? I want to run without Drupal's .htaccess file (instead incorporating these calls into my httpd.conf file). webmaster access_time 31 Jul 2021 - 11:45 In reply to Rewrite including filesystem path by Anonymous (not verified) I think the problem is with the base path which results in the redirection to /var/www/ part. The best approach I would think is to start with drupal htaccess and then strip out parts and move to httpd SNVC (not verified) access_time 31 Jul 2021 - 11:45 This is definitely a good guide. Thanks for this. wellyson access_time 31 Jul 2021 - 11:45 This is really nice and helpful. Pagination Current page 1 Page 2 Page 3 Page 4 Next page Next › Last page Last »
/themes/custom/zyxpro_light/images/placeholder.png
close

on 08th February 2008 / by webmaster
Attachment Size slashdotit.tar.gz 173.55 KB Similar to the StumbleThis module that we had created earlier we have created another bookmarking Drupal Technical Slashdotit Shlomi Fish (not verified) access_time 31 Jul 2021 - 11:45 Hi! I'm interested in this module, but Drupal 6.2 complains that it's incompatible with it. Can you please update it or tell me how? Regards, Shlomi Fish shlomifish.org webmaster access_time 31 Jul 2021 - 11:45 In reply to Can you update it for Drupal-6.x? by Shlomi Fish (not verified) Sure we are working on this currently and will post an update soon. Also we will post this module on www.drupal.org as well Cheers Anoop John Team Zyxware Anonymous (not verified) access_time 31 Jul 2021 - 11:45 Hello! Nice Article very interesting, thanks Pagination Current page 1 Page 2 Page 3 Page 4 Next page Next › Last page Last »