Programming Forums > Script Programming > Bash / Shell Scripting

gaurivk · Jun 28th, 2010, 7:03 AM

Hello, first time posting here.

My problem is following. I have 524 files each containing 13 columns (say file001.dat to file524.dat) I have another file with 524 lines (attrib.dat). This file contains 4 attributes of each of 524 files. What I need to do is, read attributes from attrib.dat and print 3 of it it as columns in corresponding file###.dat. Let's say file001.dat contains following (let's restrict to 4 cols):
39.759 23 1.532 -0.3957
38.935 24 0.786 -0.6945
38.111 25 -0.301 -0.03586
37.286 26 0.793 0.6567
36.462 27 1.309 1.166

There are 523 more such files. Now attrib.dat will contain 524 lines with cols like:
101.031 -2.616 52.4255 0.0008
|
|
99.7730 2.75080 109.584 0.00133

What I want as output is:
101.031 -2.616 52.4255 39.759 23 1.532 -0.3957
101.031 -2.616 52.4255 38.935 24 0.786 -0.6945
101.031 -2.616 52.4255 38.111 25 -0.301 -0.03586
101.031 -2.616 52.4255 37.286 26 0.793 0.6567
101.031 -2.616 52.4255 36.462 27 1.309 1.166
|----------------------------| |-------------------------------------|
first row from attrib.dat first file

This would be for file001.dat, and every file###.dat needs to be modified.

Is it better to do it in a program? Can I do it as a shell script? I was hoping something like this:
FILELIST=`ls file*.dat`

while f in $FILELIST && read line
do
awk -v var="$line" '{print var,"\t", $0}' $f
done < attrib.dat

Of course, it doesn't work and it is geared to print all of 4 cols in attrib.dat.

Btw, I am quite a noob for awk, and only reasonably good at shell scripting. So ANY help is appreciated.

Thanks.

L7Sqr · Jun 28th, 2010, 9:30 AM

Certainly this could be done in the shell alone. I find that tasks like this are better suited to some scripting language, however. Perl, Python, Ruby, etc. are very robust for handling text and, in most cases, much easier to maintain then a set of piped calls to cut, awk, grep, ...
For example, something simple in Ruby might look like:

(Toggle Plain Text)

 $ cat attrib.dat
101.031 -2.616 52.4255 0.0008

 $ cat file001.dat
39.759 23 1.532 -0.3957
38.935 24 0.786 -0.6945
38.111 25 -0.301 -0.03586
37.286 26 0.793 0.6567
36.462 27 1.309 1.166

 $ ./doit.rb < attrib.dat
101.031 -2.616 52.4255 39.759 23 1.532 -0.3957
101.031 -2.616 52.4255 38.935 24 0.786 -0.6945
101.031 -2.616 52.4255 38.111 25 -0.301 -0.03586
101.031 -2.616 52.4255 37.286 26 0.793 0.6567
101.031 -2.616 52.4255 36.462 27 1.309 1.166

And the script looks like:

 ruby Syntax (Toggle Plain Text) 
#!/usr/bin/env ruby
 
 
$stdin.each_with_index do |stuff, line|
    prefix = stuff.chomp.split /\s+/
    prefix.pop
    File.open("file%03d.dat" % [line+1]).each do |row|
        tail = row.split /\s+/
        puts (prefix + tail).join(' ')
    end
end
#!/usr/bin/env ruby


$stdin.each_with_index do |stuff, line|
    prefix = stuff.chomp.split /\s+/
    prefix.pop
    File.open("file%03d.dat" % [line+1]).each do |row|
        tail = row.split /\s+/
        puts (prefix + tail).join(' ')
    end
end

DaWei · Jun 28th, 2010, 10:06 AM

Are you sure you meant to prepend the first line of ATTRIB to each of the other files (as you show in your example), or did you mean to prepend each line of ATTRIB to a different file, as I inferred from your question?

gaurivk · Jun 28th, 2010, 10:10 AM

I actually don't even know what Ruby is (apart from the fact that it's some sort of scripting language). So I woul have to get to know it and learn what this script means. But if it's this simple in Ruby, I will definitely give it a try. I hope there are other solutions which are within my grasp. But still, thanks for your reply.

Quote:

Originally Posted by L7Sqr

Certainly this could be done in the shell alone. I find that tasks like this are better suited to some scripting language, however. Perl, Python, Ruby, etc. are very robust for handling text and, in most cases, much easier to maintain then a set of piped calls to cut, awk, grep, ...
For example, something simple in Ruby might look like:

(Toggle Plain Text)

 $ cat attrib.dat
101.031 -2.616 52.4255 0.0008

 $ cat file001.dat
39.759 23 1.532 -0.3957
38.935 24 0.786 -0.6945
38.111 25 -0.301 -0.03586
37.286 26 0.793 0.6567
36.462 27 1.309 1.166

 $ ./doit.rb < attrib.dat
101.031 -2.616 52.4255 39.759 23 1.532 -0.3957
101.031 -2.616 52.4255 38.935 24 0.786 -0.6945
101.031 -2.616 52.4255 38.111 25 -0.301 -0.03586
101.031 -2.616 52.4255 37.286 26 0.793 0.6567
101.031 -2.616 52.4255 36.462 27 1.309 1.166

And the script looks like:

 ruby Syntax (Toggle Plain Text) 
#!/usr/bin/env ruby
 
 
$stdin.each_with_index do |stuff, line|
    prefix = stuff.chomp.split /\s+/
    prefix.pop
    File.open("file%03d.dat" % [line+1]).each do |row|
        tail = row.split /\s+/
        puts (prefix + tail).join(' ')
    end
end
#!/usr/bin/env ruby


$stdin.each_with_index do |stuff, line|
    prefix = stuff.chomp.split /\s+/
    prefix.pop
    File.open("file%03d.dat" % [line+1]).each do |row|
        tail = row.split /\s+/
        puts (prefix + tail).join(' ')
    end
end

gaurivk · Jun 28th, 2010, 10:12 AM

You inferred right. So line 1 (3 columns out of 4) would prepend to file001.dat, line 2 would prepend to file002.dat ... line 524 would prepend to file524.dat.

Quote:

Originally Posted by DaWei

Are you sure you meant to prepend the first line of ATTRIB to each of the other files (as you show in your example), or did you mean to prepend each line of ATTRIB to a different file, as I inferred from your question?

DaWei · Jun 28th, 2010, 11:15 AM

This is pseudo code. Use any language you're comfortable with.

(Toggle Plain Text)

Open ATTRIBS for reading
For each line in attribs file
    open file TEMP for writing
    open file TARGET for reading
    read LINE, discarding newline
    For each line in target file
        read LINE2, retaining newline
        write LINE to TEMP, adding space
        write LINE2 to temp
    End For
    Close TARGET
    Close TEMP
    Delete TARGET (or rename as a safety backup, if desired)
    Rename TEMP to TARGET
End For
Close ATTRIBS

L7Sqr · Jun 28th, 2010, 12:23 PM

@DaWei:
Your solution doesn't address the need to drop (possibly arbitrary) columns from the attrib file. To be fair, mine is not particularly flexible either. Something in my gut tells me that combination of required columns is not fixed and will lead to problems down the line in either case. Though, that could just be last night's Fogo De Chao talking...

@gaurivk:
I posted Ruby as that is what I am most familiar with. My point is that you might want to consider facilities other than just the shell to help you in your task. If you dont know a scripting language yet, using this as an opportunity to learn one would benefit you in the future.

DaWei · Jun 28th, 2010, 1:03 PM

L7, you're correct. The requirements document is a tad lacking.

Restricting oneself to some left subset of columns from the attrib file would be trivial. Restricting oneself to some dynamically specified subset of columns would be only a tad more difficult.

I shot for the language-agnostic approach because I have no idea of the range of capabilities of the OP. More complex pick-and-choose operations are obviously going to depend on the capabilities of the chosen language.

Incidentally, I downloaded Ruby and began to play with it a couple of years ago. Right then my hard drive crashed. With no real impetus (as in: work), I haven't continued with it.

Patsie · Jun 28th, 2010, 3:18 PM

Hi all,

I'd like to sugges the following piece of awk for the OP.
for each line in attrib.dat read the line from the corresponding file###.dat and prepend the first 3 columns of attrib.dat to it.

(Toggle Plain Text)

awk '{
  file = sprintf("file%03d.dat", NR);
  getline line < file;
  printf("%s %s %s %s\n", $1, $2, $3, line) > file;
  close(file);
}' < attrib.dat

DaWei · Jun 28th, 2010, 6:11 PM

He wants to prepend to ALL the lines in the dat file, not just the first.

Jun 28th, 2010, 7:03 AM	#1
gaurivk Newbie Join Date: Jun 2010 Posts: 3 Rep Power: 0	Turn rows into columns Hello, first time posting here. My problem is following. I have 524 files each containing 13 columns (say file001.dat to file524.dat) I have another file with 524 lines (attrib.dat). This file contains 4 attributes of each of 524 files. What I need to do is, read attributes from attrib.dat and print 3 of it it as columns in corresponding file###.dat. Let's say file001.dat contains following (let's restrict to 4 cols): 39.759 23 1.532 -0.3957 38.935 24 0.786 -0.6945 38.111 25 -0.301 -0.03586 37.286 26 0.793 0.6567 36.462 27 1.309 1.166 There are 523 more such files. Now attrib.dat will contain 524 lines with cols like: 101.031 -2.616 52.4255 0.0008 \| \| 99.7730 2.75080 109.584 0.00133 What I want as output is: 101.031 -2.616 52.4255 39.759 23 1.532 -0.3957 101.031 -2.616 52.4255 38.935 24 0.786 -0.6945 101.031 -2.616 52.4255 38.111 25 -0.301 -0.03586 101.031 -2.616 52.4255 37.286 26 0.793 0.6567 101.031 -2.616 52.4255 36.462 27 1.309 1.166 \|----------------------------\| \|-------------------------------------\| first row from attrib.dat first file This would be for file001.dat, and every file###.dat needs to be modified. Is it better to do it in a program? Can I do it as a shell script? I was hoping something like this: FILELIST=`ls file*.dat` while f in $FILELIST && read line do awk -v var="$line" '{print var,"\t", $0}' $f done < attrib.dat Of course, it doesn't work and it is geared to print all of 4 cols in attrib.dat. Btw, I am quite a noob for awk, and only reasonably good at shell scripting. So ANY help is appreciated. Thanks.

Jun 28th, 2010, 9:30 AM	#2
L7Sqr Professional Programmer Join Date: Jun 2005 Location: here Posts: 440 Rep Power: 8	Re: Turn rows into columns Certainly this could be done in the shell alone. I find that tasks like this are better suited to some scripting language, however. Perl, Python, Ruby, etc. are very robust for handling text and, in most cases, much easier to maintain then a set of piped calls to cut, awk, grep, ... For example, something simple in Ruby might look like: (Toggle Plain Text) $ cat attrib.dat 101.031 -2.616 52.4255 0.0008 $ cat file001.dat 39.759 23 1.532 -0.3957 38.935 24 0.786 -0.6945 38.111 25 -0.301 -0.03586 37.286 26 0.793 0.6567 36.462 27 1.309 1.166 $ ./doit.rb < attrib.dat 101.031 -2.616 52.4255 39.759 23 1.532 -0.3957 101.031 -2.616 52.4255 38.935 24 0.786 -0.6945 101.031 -2.616 52.4255 38.111 25 -0.301 -0.03586 101.031 -2.616 52.4255 37.286 26 0.793 0.6567 101.031 -2.616 52.4255 36.462 27 1.309 1.166 $ cat attrib.dat 101.031 -2.616 52.4255 0.0008 $ cat file001.dat 39.759 23 1.532 -0.3957 38.935 24 0.786 -0.6945 38.111 25 -0.301 -0.03586 37.286 26 0.793 0.6567 36.462 27 1.309 1.166 $ ./doit.rb < attrib.dat 101.031 -2.616 52.4255 39.759 23 1.532 -0.3957 101.031 -2.616 52.4255 38.935 24 0.786 -0.6945 101.031 -2.616 52.4255 38.111 25 -0.301 -0.03586 101.031 -2.616 52.4255 37.286 26 0.793 0.6567 101.031 -2.616 52.4255 36.462 27 1.309 1.166 And the script looks like: ruby Syntax (Toggle Plain Text) #!/usr/bin/env ruby $stdin.each_with_index do \|stuff, line\| prefix = stuff.chomp.split /\s+/ prefix.pop File.open("file%03d.dat" % [line+1]).each do \|row\| tail = row.split /\s+/ puts (prefix + tail).join(' ') end end #!/usr/bin/env ruby $stdin.each_with_index do \|stuff, line\| prefix = stuff.chomp.split /\s+/ prefix.pop File.open("file%03d.dat" % [line+1]).each do \|row\| tail = row.split /\s+/ puts (prefix + tail).join(' ') end end __________________ "...and though our kids are blessed their parents let them shoulder all the blame." - The Quiet Things That No One Ever Knows [BrandNew] My Blog

Jun 28th, 2010, 10:06 AM	#3
DaWei Resident Grouch Join Date: Jun 2005 Posts: 8,368 Rep Power: 17	Re: Turn rows into columns Are you sure you meant to prepend the first line of ATTRIB to each of the other files (as you show in your example), or did you mean to prepend each line of ATTRIB to a different file, as I inferred from your question? __________________ Contributor's Corner: Politically Incorrect DaWei on Pointers

Jun 28th, 2010, 11:15 AM	#6
DaWei Resident Grouch Join Date: Jun 2005 Posts: 8,368 Rep Power: 17	Re: Turn rows into columns This is pseudo code. Use any language you're comfortable with. (Toggle Plain Text) Open ATTRIBS for reading For each line in attribs file open file TEMP for writing open file TARGET for reading read LINE, discarding newline For each line in target file read LINE2, retaining newline write LINE to TEMP, adding space write LINE2 to temp End For Close TARGET Close TEMP Delete TARGET (or rename as a safety backup, if desired) Rename TEMP to TARGET End For Close ATTRIBS Open ATTRIBS for reading For each line in attribs file open file TEMP for writing open file TARGET for reading read LINE, discarding newline For each line in target file read LINE2, retaining newline write LINE to TEMP, adding space write LINE2 to temp End For Close TARGET Close TEMP Delete TARGET (or rename as a safety backup, if desired) Rename TEMP to TARGET End For Close ATTRIBS __________________ Contributor's Corner: Politically Incorrect DaWei on Pointers

Jun 28th, 2010, 12:23 PM	#7
L7Sqr Professional Programmer Join Date: Jun 2005 Location: here Posts: 440 Rep Power: 8	Re: Turn rows into columns @DaWei: Your solution doesn't address the need to drop (possibly arbitrary) columns from the attrib file. To be fair, mine is not particularly flexible either. Something in my gut tells me that combination of required columns is not fixed and will lead to problems down the line in either case. Though, that could just be last night's Fogo De Chao talking... @gaurivk: I posted Ruby as that is what I am most familiar with. My point is that you might want to consider facilities other than just the shell to help you in your task. If you dont know a scripting language yet, using this as an opportunity to learn one would benefit you in the future. __________________ "...and though our kids are blessed their parents let them shoulder all the blame." - The Quiet Things That No One Ever Knows [BrandNew] My Blog

Jun 28th, 2010, 1:03 PM	#8
DaWei Resident Grouch Join Date: Jun 2005 Posts: 8,368 Rep Power: 17	Re: Turn rows into columns L7, you're correct. The requirements document is a tad lacking. Restricting oneself to some left subset of columns from the attrib file would be trivial. Restricting oneself to some dynamically specified subset of columns would be only a tad more difficult. I shot for the language-agnostic approach because I have no idea of the range of capabilities of the OP. More complex pick-and-choose operations are obviously going to depend on the capabilities of the chosen language. Incidentally, I downloaded Ruby and began to play with it a couple of years ago. Right then my hard drive crashed. With no real impetus (as in: work), I haven't continued with it. __________________ Contributor's Corner: Politically Incorrect DaWei on Pointers

Jun 28th, 2010, 3:18 PM	#9
Patsie Professional Programmer Join Date: Jun 2010 Posts: 450 Rep Power: 3	Re: Turn rows into columns Hi all, I'd like to sugges the following piece of awk for the OP. for each line in attrib.dat read the line from the corresponding file###.dat and prepend the first 3 columns of attrib.dat to it. (Toggle Plain Text) awk '{ file = sprintf("file%03d.dat", NR); getline line < file; printf("%s %s %s %s\n", $1, $2, $3, line) > file; close(file); }' < attrib.dat awk '{ file = sprintf("file%03d.dat", NR); getline line < file; printf("%s %s %s %s\n", $1, $2, $3, line) > file; close(file); }' < attrib.dat

Jun 28th, 2010, 6:11 PM	#10
DaWei Resident Grouch Join Date: Jun 2005 Posts: 8,368 Rep Power: 17	Re: Turn rows into columns He wants to prepend to ALL the lines in the dat file, not just the first. __________________ Contributor's Corner: Politically Incorrect DaWei on Pointers

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Converting Columns to Rows	Awker	Sed and Awk	1	Mar 24th, 2012 4:00 AM
Transposing rows to columns	Blivo	Sed and Awk	1	Mar 13th, 2012 8:19 AM
Delete non-satisfying rows of data from matrix	isix6	C++	9	Dec 21st, 2008 8:26 AM
Problem adding/removing rows to a form.	blasterstudios	JavaScript and Client-Side Browser Scripting	11	May 5th, 2006 3:44 PM
How can I disable particular rows in a datagrid?	HappyTomato	C#	2	Aug 25th, 2005 3:48 PM

Programming Forums > Script Programming > Bash / Shell Scripting

Turn rows into columns