Programming Forums > Script Programming > Perl

lynxus · Dec 19th, 2007, 3:39 AM

Hi guys, i have a log file that normally is many many many IP's in it.
However sometimes i get attacked by a single IP and it slows down the system. The log file shows this IP more than any other.

What i was thinking of was a script that could check teh file, if it has shed loads of the same IP it woudl do something ( mail me ) i can do the mail part, but the actual looking and working with part im stuck on.

Has anyone got any idea how to do the below?

Check file.log to see if an IP is repeated lets say 1000 times or more.
If so then do Blaaaaa. Else do nothing.

I cant get it to look for an IP ie 1.1.1.1 as it changes, so somehow it will need to be able to identify what it looks like?

Any info would be GReeeeeaaaat.

-Graham

Sane · Dec 19th, 2007, 12:24 PM

You want something called a hash table. In other languages, this data type has other synonyms. In Perl, it's an "associative array". In Python, it's a "dictionary". Etc etc.

If you plan on using Perl, read up on how to use an associative array. Then you will implement pseudocode that looks something like this:

(Toggle Plain Text)

for each ip address in the log file
    see if the ip address exists in the hash table
    if the ip address does exist
        then increment its value by +1
    if the ip address does not exist
        then make a new entry in the hash table for the ip address
        set its starting value to 1

if any entry in the hash table goes over 1000. then do something about it.

You will also need to think of a way to incorporate time into account, since someone could view your page 1000 times over the span of a month, and your program would consider it an attack.

Unless your log file already flushes daily. Then you're set.

lynxus · Dec 20th, 2007, 6:50 AM

Cool, thanks ill take a look.

yeah the logfile rotates, so it purges old entrys at the bottom. So hopefully it shouldnt give any faulse posivives however even those wouldnt be to much of a problem as some monitoring of it is better then none

Thanks
again

mbd · Dec 21st, 2007, 10:10 AM

perl is too powerful to loop through a log file counting up instances of different ip addresses. just use something like this.

 perl Syntax (Toggle Plain Text) 
#!/usr/bin/perl
 
$log = "1.1.1.1 2.2.2.2 3.3.3.3 2.2.2.2 1.1.1.1 1.1.1.1";
while ($log =~ m/(\d\.\d\.\d\.\d)/g)
{
	$ip = $1;
	$count = ($log =~ s/$1/$1/g);
	print "$ip $count\n";
}
#!/usr/bin/perl

$log = "1.1.1.1 2.2.2.2 3.3.3.3 2.2.2.2 1.1.1.1 1.1.1.1";
while ($log =~ m/(\d\.\d\.\d\.\d)/g)
{
	$ip = $1;
	$count = ($log =~ s/$1/$1/g);
	print "$ip $count\n";
}

Sane · Dec 21st, 2007, 10:24 AM

An O(N^2) algorithm? What if the file is 100kb?

Even if it compared 10,240 bytes per second. It would still take 12 days to run it only once.

(Toggle Plain Text)

print (100*1024)**2/10240.0/3600.0/24.0

That's one of the worst implementations for such a simple problem I've ever come across.

An O(N) algorithm, such as using an associative array, is just as simple to implement, and will take 10 seconds under the exact same circumstances...

(Toggle Plain Text)

print (100*1024)/10240.0

Sane · Dec 21st, 2007, 11:13 AM

And in case you think 100kb is unreasonably large for a log file... a request every 13 seconds can fill that up in less than one day.

(Toggle Plain Text)

print ( 3600*24 ) / ( 100*1024/16.0 )

mbd · Dec 21st, 2007, 12:08 PM

it is a delicate balance between whos time is more valuable. this is not a great example of it seeing that your solution would be rather simple to implement also, but:

my time * my salaray > (my cpu's time ^ 2) * my cpu's salaray

i wager that my algorithm could count a 1MB file in less than ten minutes under the worst case of no ip address duplicates.

Sane · Dec 21st, 2007, 12:12 PM

Quote:

Originally Posted by mbd

i wager that my algorithm could count a 1MB file in less than ten minutes under the worst case of no ip address duplicates.

Prove me wrong.

I'll take you on that wager.

mbd · Dec 21st, 2007, 2:51 PM

post a 1MB log file

Jessehk · Dec 21st, 2007, 4:59 PM

I wrote a quick C++ program using the Boost.Random libraries to generate a text file with 100,000 IP addresses.

Have fun.

Dec 19th, 2007, 3:39 AM	#1
lynxus Newbie Join Date: Dec 2007 Posts: 4 Rep Power: 0	Count number of occourences of a IP address in a log file. Hi guys, i have a log file that normally is many many many IP's in it. However sometimes i get attacked by a single IP and it slows down the system. The log file shows this IP more than any other. What i was thinking of was a script that could check teh file, if it has shed loads of the same IP it woudl do something ( mail me ) i can do the mail part, but the actual looking and working with part im stuck on. Has anyone got any idea how to do the below? Check file.log to see if an IP is repeated lets say 1000 times or more. If so then do Blaaaaa. Else do nothing. I cant get it to look for an IP ie 1.1.1.1 as it changes, so somehow it will need to be able to identify what it looks like? Any info would be GReeeeeaaaat. -Graham

Dec 19th, 2007, 12:24 PM	#2
Sane Programming Guru Join Date: Apr 2005 Posts: 1,799 Rep Power: 5	Re: Count number of occourences of a IP address in a log file. You want something called a hash table. In other languages, this data type has other synonyms. In Perl, it's an "associative array". In Python, it's a "dictionary". Etc etc. If you plan on using Perl, read up on how to use an associative array. Then you will implement pseudocode that looks something like this: (Toggle Plain Text) for each ip address in the log file see if the ip address exists in the hash table if the ip address does exist then increment its value by +1 if the ip address does not exist then make a new entry in the hash table for the ip address set its starting value to 1 if any entry in the hash table goes over 1000. then do something about it. for each ip address in the log file see if the ip address exists in the hash table if the ip address does exist then increment its value by +1 if the ip address does not exist then make a new entry in the hash table for the ip address set its starting value to 1 if any entry in the hash table goes over 1000. then do something about it. You will also need to think of a way to incorporate time into account, since someone could view your page 1000 times over the span of a month, and your program would consider it an attack. Unless your log file already flushes daily. Then you're set.

Dec 20th, 2007, 6:50 AM	#3
lynxus Newbie Join Date: Dec 2007 Posts: 4 Rep Power: 0	Re: Count number of occourences of a IP address in a log file. Cool, thanks ill take a look. yeah the logfile rotates, so it purges old entrys at the bottom. So hopefully it shouldnt give any faulse posivives however even those wouldnt be to much of a problem as some monitoring of it is better then none Thanks again

Dec 21st, 2007, 10:10 AM	#4
mbd Programmer Join Date: Nov 2007 Posts: 86 Rep Power: 1	Re: Count number of occourences of a IP address in a log file. perl is too powerful to loop through a log file counting up instances of different ip addresses. just use something like this. perl Syntax (Toggle Plain Text) #!/usr/bin/perl $log = "1.1.1.1 2.2.2.2 3.3.3.3 2.2.2.2 1.1.1.1 1.1.1.1"; while ($log =~ m/(\d\.\d\.\d\.\d)/g) { $ip = $1; $count = ($log =~ s/$1/$1/g); print "$ip $count\n"; } #!/usr/bin/perl $log = "1.1.1.1 2.2.2.2 3.3.3.3 2.2.2.2 1.1.1.1 1.1.1.1"; while ($log =~ m/(\d\.\d\.\d\.\d)/g) { $ip = $1; $count = ($log =~ s/$1/$1/g); print "$ip $count\n"; }

Dec 21st, 2007, 10:24 AM	#5
Sane Programming Guru Join Date: Apr 2005 Posts: 1,799 Rep Power: 5	Re: Count number of occourences of a IP address in a log file. An O(N^2) algorithm? What if the file is 100kb? Even if it compared 10,240 bytes per second. It would still take 12 days to run it only once. (Toggle Plain Text) print (1001024)2/10240.0/3600.0/24.0 print (1001024)*2/10240.0/3600.0/24.0 That's one of the worst implementations for such a simple problem I've ever come across. An O(N) algorithm, such as using an associative array, is just as simple to implement, and will take 10 seconds under the exact same circumstances... (Toggle Plain Text) print (1001024)/10240.0 print (1001024)/10240.0 Last edited by Sane; Dec 21st, 2007 at 10:41 AM.*

Dec 21st, 2007, 11:13 AM	#6
Sane Programming Guru Join Date: Apr 2005 Posts: 1,799 Rep Power: 5	Re: Count number of occourences of a IP address in a log file. And in case you think 100kb is unreasonably large for a log file... a request every 13 seconds can fill that up in less than one day. (Toggle Plain Text) print ( 360024 ) / ( 1001024/16.0 ) print ( 360024 ) / ( 1001024/16.0 )

Dec 21st, 2007, 12:08 PM	#7
mbd Programmer Join Date: Nov 2007 Posts: 86 Rep Power: 1	Re: Count number of occourences of a IP address in a log file. it is a delicate balance between whos time is more valuable. this is not a great example of it seeing that your solution would be rather simple to implement also, but: my time * my salaray > (my cpu's time ^ 2) * my cpu's salaray i wager that my algorithm could count a 1MB file in less than ten minutes under the worst case of no ip address duplicates.

Dec 21st, 2007, 2:51 PM	#9
mbd Programmer Join Date: Nov 2007 Posts: 86 Rep Power: 1	Re: Count number of occourences of a IP address in a log file. post a 1MB log file

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Thread Tools
Show Printable Version Email this Page
Display Modes
Switch to Linear Mode Hybrid Mode Switch to Threaded Mode

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Jumping to a specific line number in a text file	Adeil	C++	4	Nov 5th, 2007 9:54 AM
PigLattin Converter, count number of words used in dictionary.	MrSmiley	Python	2	Oct 17th, 2005 4:47 PM
After execution - Error cannot locate /Skin File?	wchar	Visual Basic	1	Mar 5th, 2005 9:04 PM
airport Log program using 3D linked List : problem reading from file	gemini_shooter	C++	0	Mar 2nd, 2005 4:12 PM
Specific file address...	Monkey_features	Delphi	0	Feb 8th, 2005 5:05 PM

Programming Forums > Script Programming > Perl

Count number of occourences of a IP address in a log file.