Programming Forums > Community Forums > Coder's Corner Lounge

titaniumdecoy · May 28th, 2008, 5:35 PM

I have a 100MB file with (I suspect) millions of lines of text in it. I need a way to select 100 random lines and place them in another file. I don't want to spend a lot of time writing a Python script to do this, and I'm not sure how well such a script would operate on such a large file. Does anyone know of a Unix command that could do this? If not, how would you go about this? Thanks.

iEngage · May 28th, 2008, 5:56 PM

well for whatever language you are writing in, just get the number of lines in the file, then have your program randomly generate a number between 1 and the number of lines, and have it read that line number.

pseudo Syntax (Toggle Plain Text)

numberOfLines = getNumOfLines(file.txt);
 
lineNumber = randomNumber(1,numberOfLines);
 
read(file.txt,lineNumber);

obviously depending on the language it may take more code than that... that's not even a real language

mbd · May 28th, 2008, 9:38 PM

im not sure why you could not do this yourself, but here ya go

 perl Syntax (Toggle Plain Text) 
#!/usr/bin/perl
 
@lines = <STDIN>;
for (1 .. (($ARGV[0] < $#lines + 1) ? $ARGV[0] : $#lines + 1))
{
	print splice @lines, rand() % ($#lines + 1), 1;
}
#!/usr/bin/perl

@lines = <STDIN>;
for (1 .. (($ARGV[0] < $#lines + 1) ? $ARGV[0] : $#lines + 1))
{
	print splice @lines, rand() % ($#lines + 1), 1;
}

titaniumdecoy · May 29th, 2008, 12:54 AM

Thanks. I'm impressed by how short the program is in Perl.

andro · May 29th, 2008, 9:50 PM

cat file.txt | awk 'BEGIN {srand()} {print rand() "\t" $0}' | sort -n | cut -f2- | tail -n 100

May 28th, 2008, 5:35 PM	#1
titaniumdecoy Expert Programmer Join Date: Nov 2005 Posts: 843 Rep Power: 3	Quickly select random lines I have a 100MB file with (I suspect) millions of lines of text in it. I need a way to select 100 random lines and place them in another file. I don't want to spend a lot of time writing a Python script to do this, and I'm not sure how well such a script would operate on such a large file. Does anyone know of a Unix command that could do this? If not, how would you go about this? Thanks.

May 28th, 2008, 5:56 PM	#2
iEngage Newbie Join Date: May 2008 Location: teh interwebz Posts: 22 Rep Power: 0	Re: Quickly select random lines well for whatever language you are writing in, just get the number of lines in the file, then have your program randomly generate a number between 1 and the number of lines, and have it read that line number. pseudo Syntax (Toggle Plain Text) numberOfLines = getNumOfLines(file.txt); lineNumber = randomNumber(1,numberOfLines); read(file.txt,lineNumber); numberOfLines = getNumOfLines(file.txt); lineNumber = randomNumber(1,numberOfLines); read(file.txt,lineNumber); obviously depending on the language it may take more code than that... that's not even a real language __________________ iEngage

May 28th, 2008, 9:38 PM	#3
mbd Programmer Join Date: Nov 2007 Posts: 86 Rep Power: 1	Re: Quickly select random lines im not sure why you could not do this yourself, but here ya go perl Syntax (Toggle Plain Text) #!/usr/bin/perl @lines = <STDIN>; for (1 .. (($ARGV[0] < $#lines + 1) ? $ARGV[0] : $#lines + 1)) { print splice @lines, rand() % ($#lines + 1), 1; } #!/usr/bin/perl @lines = <STDIN>; for (1 .. (($ARGV[0] < $#lines + 1) ? $ARGV[0] : $#lines + 1)) { print splice @lines, rand() % ($#lines + 1), 1; }

May 29th, 2008, 12:54 AM	#4
titaniumdecoy Expert Programmer Join Date: Nov 2005 Posts: 843 Rep Power: 3	Re: Quickly select random lines Thanks. I'm impressed by how short the program is in Perl.

May 29th, 2008, 9:50 PM	#5
andro Professional Programmer Join Date: Oct 2005 Location: California Posts: 294 Rep Power: 3	Re: Quickly select random lines cat file.txt \| awk 'BEGIN {srand()} {print rand() "\t" $0}' \| sort -n \| cut -f2- \| tail -n 100 __________________ http://www.kevinherron.com/

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Random "Letter" Generator	kewlgeye	Java	19	May 13th, 2008 9:52 AM
random Numbers in openGL	csrocker101	C++	5	Apr 24th, 2007 8:02 PM
time Delays and Random functions	Markphaser	C++	17	Feb 21st, 2006 3:48 AM
Random Number & Average Problem	Hadrurus	Java	6	Aug 15th, 2005 1:08 PM
non repeating random number generation	gencor45	C#	2	Feb 9th, 2005 12:11 AM

Programming Forums > Community Forums > Coder's Corner Lounge

Quickly select random lines