Programming Forums
Page 1 of 2 1 2 >
Show 40 post(s) from this thread on one page

Programming Forums (http://www.programmingforums.org/forumindex.php)
-   Perl (http://www.programmingforums.org/forum21.html)
-   -   Count number of occourences of a IP address in a log file. (http://www.programmingforums.org/showthread.php?t=14770)

lynxus Dec 19th, 2007 3:39 AM
Count number of occourences of a IP address in a log file.
 
Hi guys, i have a log file that normally is many many many IP's in it.
However sometimes i get attacked by a single IP and it slows down the system. The log file shows this IP more than any other.

What i was thinking of was a script that could check teh file, if it has shed loads of the same IP it woudl do something ( mail me ) i can do the mail part, but the actual looking and working with part im stuck on.

Has anyone got any idea how to do the below?

Check file.log to see if an IP is repeated lets say 1000 times or more.
If so then do Blaaaaa. Else do nothing.

I cant get it to look for an IP ie 1.1.1.1 as it changes, so somehow it will need to be able to identify what it looks like?


Any info would be GReeeeeaaaat.

-Graham

Sane Dec 19th, 2007 12:24 PM
Re: Count number of occourences of a IP address in a log file.
 
You want something called a hash table. In other languages, this data type has other synonyms. In Perl, it's an "associative array". In Python, it's a "dictionary". Etc etc.

If you plan on using Perl, read up on how to use an associative array. Then you will implement pseudocode that looks something like this:

:
for each ip address in the log file
    see if the ip address exists in the hash table
    if the ip address does exist
        then increment its value by +1
    if the ip address does not exist
        then make a new entry in the hash table for the ip address
        set its starting value to 1

if any entry in the hash table goes over 1000. then do something about it.

You will also need to think of a way to incorporate time into account, since someone could view your page 1000 times over the span of a month, and your program would consider it an attack.

Unless your log file already flushes daily. Then you're set.

lynxus Dec 20th, 2007 6:50 AM
Re: Count number of occourences of a IP address in a log file.
 
Cool, thanks ill take a look.

yeah the logfile rotates, so it purges old entrys at the bottom. So hopefully it shouldnt give any faulse posivives however even those wouldnt be to much of a problem as some monitoring of it is better then none :)

Thanks
again

mbd Dec 21st, 2007 10:10 AM
Re: Count number of occourences of a IP address in a log file.
 
perl is too powerful to loop through a log file counting up instances of different ip addresses. just use something like this.

:
  1. #!/usr/bin/perl
  2.  
  3. $log = "1.1.1.1 2.2.2.2 3.3.3.3 2.2.2.2 1.1.1.1 1.1.1.1";
  4. while ($log =~ m/(\d\.\d\.\d\.\d)/g)
  5. {
  6.         $ip = $1;
  7.         $count = ($log =~ s/$1/$1/g);
  8.         print "$ip $count\n";
  9. }

Sane Dec 21st, 2007 10:24 AM
Re: Count number of occourences of a IP address in a log file.
 
An O(N^2) algorithm? What if the file is 100kb?

Even if it compared 10,240 bytes per second. It would still take 12 days to run it only once.

:
print (100*1024)**2/10240.0/3600.0/24.0

That's one of the worst implementations for such a simple problem I've ever come across.


An O(N) algorithm, such as using an associative array, is just as simple to implement, and will take 10 seconds under the exact same circumstances...

:
print (100*1024)/10240.0

Sane Dec 21st, 2007 11:13 AM
Re: Count number of occourences of a IP address in a log file.
 
And in case you think 100kb is unreasonably large for a log file... a request every 13 seconds can fill that up in less than one day. :icon_neutral:

:
print ( 3600*24 ) / ( 100*1024/16.0 )

mbd Dec 21st, 2007 12:08 PM
Re: Count number of occourences of a IP address in a log file.
 
it is a delicate balance between whos time is more valuable. this is not a great example of it seeing that your solution would be rather simple to implement also, but:

my time * my salaray > (my cpu's time ^ 2) * my cpu's salaray

i wager that my algorithm could count a 1MB file in less than ten minutes under the worst case of no ip address duplicates.

Sane Dec 21st, 2007 12:12 PM
Re: Count number of occourences of a IP address in a log file.
 
Quote:
Originally Posted by mbd (Post 138614)
i wager that my algorithm could count a 1MB file in less than ten minutes under the worst case of no ip address duplicates.
Prove me wrong. :) I'll take you on that wager.

mbd Dec 21st, 2007 2:51 PM
Re: Count number of occourences of a IP address in a log file.
 
post a 1MB log file

Jessehk Dec 21st, 2007 4:59 PM
Re: Count number of occourences of a IP address in a log file.
 
1 Attachment(s)
I wrote a quick C++ program using the Boost.Random libraries to generate a text file with 100,000 IP addresses.

Have fun. :)


All times are GMT -5. The time now is 2:17 PM.
Page 1 of 2 1 2 >
Show 40 post(s) from this thread on one page

Powered by vBulletin® Version 3.7.0, Copyright ©2000 - 2008, Jelsoft Enterprises Ltd.
Copyright ©2007 DaniWeb® LLC