Programming Forums - View Single Post - Let's try efficiency

jim mcnamara · Aug 23rd, 2006, 4:43 PM

Suppose that we have a really big number file, say 20 MB of numbers in a
single column. It has duplicates. Let's call this file bigfile.

You are given another file, which is much smaller, say 20000 lines. We'll
call it smallfile. It is also a list of numbers, with no duplicates.

Requirements Statement:
Create a new file based on the data in bigfile.
1. the new file will contain no lines found in smallfile
2. the new file will have no duplicates.

This code snippet, while it works, will take a large number of operations: file I/O's & searches:

(Toggle Plain Text)

while read number
do
	grep -v "$number" bigfile > newfile
	mv newfile bigfile
done < smallfile
sort -u bigfile > newfile

(Toggle Plain Text)

grep -f -v smallfile bigfile | sort -u > newfile

is a possibility. If your grep doesn't barf on more than 2048000 bytes in the -f file (XOPEN limit)

But we're in another forum... and we're trying something.

Soo... based on the name of the forum (sed & awk in case you forgot) how
would you create a nice efficient chunk of code that meets the Requirements
Statement above? Efficient means the least number of passes thru bigfile.
Say 2-3 maybe. Forget grep & sort.

In other words, you have to create a resultset from bigfile "minus"
smallfile that is a unique list.

Go for it. And think associative arrays.

Aug 23rd, 2006, 4:43 PM	#1
jim mcnamara Hobbyist Programmer Join Date: Jun 2005 Location: New Mexico Posts: 228 Rep Power: 4	Let's try efficiency - a coding challenge Suppose that we have a really big number file, say 20 MB of numbers in a single column. It has duplicates. Let's call this file bigfile. You are given another file, which is much smaller, say 20000 lines. We'll call it smallfile. It is also a list of numbers, with no duplicates. Requirements Statement: Create a new file based on the data in bigfile. 1. the new file will contain no lines found in smallfile 2. the new file will have no duplicates. This code snippet, while it works, will take a large number of operations: file I/O's & searches: (Toggle Plain Text) while read number do grep -v "$number" bigfile > newfile mv newfile bigfile done < smallfile sort -u bigfile > newfile while read number do grep -v "$number" bigfile > newfile mv newfile bigfile done < smallfile sort -u bigfile > newfile (Toggle Plain Text) grep -f -v smallfile bigfile \| sort -u > newfile grep -f -v smallfile bigfile \| sort -u > newfile is a possibility. If your grep doesn't barf on more than 2048000 bytes in the -f file (XOPEN limit) But we're in another forum... and we're trying something. Soo... based on the name of the forum (sed & awk in case you forgot) how would you create a nice efficient chunk of code that meets the Requirements Statement above? Efficient means the least number of passes thru bigfile. Say 2-3 maybe. Forget grep & sort. In other words, you have to create a resultset from bigfile "minus" smallfile that is a unique list. Go for it. And think associative arrays.