Programming Forums
User Name Password Register
 

RSS Feed
FORUM INDEX | TODAY'S POSTS | UNANSWERED THREADS | ADVANCED SEARCH

Reply
 
Thread Tools Display Modes
Old Jun 28th, 2010, 7:03 AM   #1
gaurivk
Newbie
 
Join Date: Jun 2010
Posts: 3
Rep Power: 0 gaurivk is on a distinguished road
Turn rows into columns

Hello, first time posting here.

My problem is following. I have 524 files each containing 13 columns (say file001.dat to file524.dat) I have another file with 524 lines (attrib.dat). This file contains 4 attributes of each of 524 files. What I need to do is, read attributes from attrib.dat and print 3 of it it as columns in corresponding file###.dat. Let's say file001.dat contains following (let's restrict to 4 cols):
39.759 23 1.532 -0.3957
38.935 24 0.786 -0.6945
38.111 25 -0.301 -0.03586
37.286 26 0.793 0.6567
36.462 27 1.309 1.166

There are 523 more such files. Now attrib.dat will contain 524 lines with cols like:
101.031 -2.616 52.4255 0.0008
|
|
99.7730 2.75080 109.584 0.00133

What I want as output is:
101.031 -2.616 52.4255 39.759 23 1.532 -0.3957
101.031 -2.616 52.4255 38.935 24 0.786 -0.6945
101.031 -2.616 52.4255 38.111 25 -0.301 -0.03586
101.031 -2.616 52.4255 37.286 26 0.793 0.6567
101.031 -2.616 52.4255 36.462 27 1.309 1.166
|----------------------------| |-------------------------------------|
first row from attrib.dat first file

This would be for file001.dat, and every file###.dat needs to be modified.

Is it better to do it in a program? Can I do it as a shell script? I was hoping something like this:
FILELIST=`ls file*.dat`

while f in $FILELIST && read line
do
awk -v var="$line" '{print var,"\t", $0}' $f
done < attrib.dat

Of course, it doesn't work and it is geared to print all of 4 cols in attrib.dat.

Btw, I am quite a noob for awk, and only reasonably good at shell scripting. So ANY help is appreciated.

Thanks.
gaurivk is offline   Reply With Quote
Old Jun 28th, 2010, 9:30 AM   #2
L7Sqr
Professional Programmer
 
Join Date: Jun 2005
Location: here
Posts: 440
Rep Power: 10 L7Sqr will become famous soon enough
Re: Turn rows into columns

Certainly this could be done in the shell alone. I find that tasks like this are better suited to some scripting language, however. Perl, Python, Ruby, etc. are very robust for handling text and, in most cases, much easier to maintain then a set of piped calls to cut, awk, grep, ...
For example, something simple in Ruby might look like:
 $ cat attrib.dat
101.031 -2.616 52.4255 0.0008

 $ cat file001.dat
39.759 23 1.532 -0.3957
38.935 24 0.786 -0.6945
38.111 25 -0.301 -0.03586
37.286 26 0.793 0.6567
36.462 27 1.309 1.166

 $ ./doit.rb < attrib.dat
101.031 -2.616 52.4255 39.759 23 1.532 -0.3957
101.031 -2.616 52.4255 38.935 24 0.786 -0.6945
101.031 -2.616 52.4255 38.111 25 -0.301 -0.03586
101.031 -2.616 52.4255 37.286 26 0.793 0.6567
101.031 -2.616 52.4255 36.462 27 1.309 1.166

And the script looks like:
ruby Syntax (Toggle Plain Text)
  1. #!/usr/bin/env ruby
  2.  
  3.  
  4. $stdin.each_with_index do |stuff, line|
  5. prefix = stuff.chomp.split /\s+/
  6. prefix.pop
  7. File.open("file%03d.dat" % [line+1]).each do |row|
  8. tail = row.split /\s+/
  9. puts (prefix + tail).join(' ')
  10. end
  11. end
__________________
"...and though our kids are blessed their parents let them shoulder all the blame."
- The Quiet Things That No One Ever Knows [BrandNew]

My Blog
L7Sqr is offline   Reply With Quote
Old Jun 28th, 2010, 10:06 AM   #3
DaWei
Resident Grouch
 
DaWei's Avatar
 
Join Date: Jun 2005
Posts: 8,368
Rep Power: 19 DaWei will become famous soon enoughDaWei will become famous soon enough
Re: Turn rows into columns

Are you sure you meant to prepend the first line of ATTRIB to each of the other files (as you show in your example), or did you mean to prepend each line of ATTRIB to a different file, as I inferred from your question?
__________________
Contributor's Corner:
Politically Incorrect
DaWei on Pointers
DaWei is offline   Reply With Quote
Old Jun 28th, 2010, 10:10 AM   #4
gaurivk
Newbie
 
Join Date: Jun 2010
Posts: 3
Rep Power: 0 gaurivk is on a distinguished road
Re: Turn rows into columns

I actually don't even know what Ruby is (apart from the fact that it's some sort of scripting language). So I woul have to get to know it and learn what this script means. But if it's this simple in Ruby, I will definitely give it a try. I hope there are other solutions which are within my grasp. But still, thanks for your reply.

Quote:
Originally Posted by L7Sqr View Post
Certainly this could be done in the shell alone. I find that tasks like this are better suited to some scripting language, however. Perl, Python, Ruby, etc. are very robust for handling text and, in most cases, much easier to maintain then a set of piped calls to cut, awk, grep, ...
For example, something simple in Ruby might look like:
 $ cat attrib.dat
101.031 -2.616 52.4255 0.0008

 $ cat file001.dat
39.759 23 1.532 -0.3957
38.935 24 0.786 -0.6945
38.111 25 -0.301 -0.03586
37.286 26 0.793 0.6567
36.462 27 1.309 1.166

 $ ./doit.rb < attrib.dat
101.031 -2.616 52.4255 39.759 23 1.532 -0.3957
101.031 -2.616 52.4255 38.935 24 0.786 -0.6945
101.031 -2.616 52.4255 38.111 25 -0.301 -0.03586
101.031 -2.616 52.4255 37.286 26 0.793 0.6567
101.031 -2.616 52.4255 36.462 27 1.309 1.166

And the script looks like:
ruby Syntax (Toggle Plain Text)
  1. #!/usr/bin/env ruby
  2.  
  3.  
  4. $stdin.each_with_index do |stuff, line|
  5. prefix = stuff.chomp.split /\s+/
  6. prefix.pop
  7. File.open("file%03d.dat" % [line+1]).each do |row|
  8. tail = row.split /\s+/
  9. puts (prefix + tail).join(' ')
  10. end
  11. end
gaurivk is offline   Reply With Quote
Old Jun 28th, 2010, 10:12 AM   #5
gaurivk
Newbie
 
Join Date: Jun 2010
Posts: 3
Rep Power: 0 gaurivk is on a distinguished road
Re: Turn rows into columns

You inferred right. So line 1 (3 columns out of 4) would prepend to file001.dat, line 2 would prepend to file002.dat ... line 524 would prepend to file524.dat.

Quote:
Originally Posted by DaWei View Post
Are you sure you meant to prepend the first line of ATTRIB to each of the other files (as you show in your example), or did you mean to prepend each line of ATTRIB to a different file, as I inferred from your question?
gaurivk is offline   Reply With Quote
Old Jun 28th, 2010, 11:15 AM   #6
DaWei
Resident Grouch
 
DaWei's Avatar
 
Join Date: Jun 2005
Posts: 8,368
Rep Power: 19 DaWei will become famous soon enoughDaWei will become famous soon enough
Re: Turn rows into columns

This is pseudo code. Use any language you're comfortable with.
Open ATTRIBS for reading
For each line in attribs file
    open file TEMP for writing
    open file TARGET for reading
    read LINE, discarding newline
    For each line in target file
        read LINE2, retaining newline
        write LINE to TEMP, adding space
        write LINE2 to temp
    End For
    Close TARGET
    Close TEMP
    Delete TARGET (or rename as a safety backup, if desired)
    Rename TEMP to TARGET
End For
Close ATTRIBS
__________________
Contributor's Corner:
Politically Incorrect
DaWei on Pointers
DaWei is offline   Reply With Quote
Old Jun 28th, 2010, 12:23 PM   #7
L7Sqr
Professional Programmer
 
Join Date: Jun 2005
Location: here
Posts: 440
Rep Power: 10 L7Sqr will become famous soon enough
Re: Turn rows into columns

@DaWei:
Your solution doesn't address the need to drop (possibly arbitrary) columns from the attrib file. To be fair, mine is not particularly flexible either. Something in my gut tells me that combination of required columns is not fixed and will lead to problems down the line in either case. Though, that could just be last night's Fogo De Chao talking...

@gaurivk:
I posted Ruby as that is what I am most familiar with. My point is that you might want to consider facilities other than just the shell to help you in your task. If you dont know a scripting language yet, using this as an opportunity to learn one would benefit you in the future.
__________________
"...and though our kids are blessed their parents let them shoulder all the blame."
- The Quiet Things That No One Ever Knows [BrandNew]

My Blog
L7Sqr is offline   Reply With Quote
Old Jun 28th, 2010, 1:03 PM   #8
DaWei
Resident Grouch
 
DaWei's Avatar
 
Join Date: Jun 2005
Posts: 8,368
Rep Power: 19 DaWei will become famous soon enoughDaWei will become famous soon enough
Re: Turn rows into columns

L7, you're correct. The requirements document is a tad lacking.

Restricting oneself to some left subset of columns from the attrib file would be trivial. Restricting oneself to some dynamically specified subset of columns would be only a tad more difficult.

I shot for the language-agnostic approach because I have no idea of the range of capabilities of the OP. More complex pick-and-choose operations are obviously going to depend on the capabilities of the chosen language.

Incidentally, I downloaded Ruby and began to play with it a couple of years ago. Right then my hard drive crashed. With no real impetus (as in: work), I haven't continued with it.
__________________
Contributor's Corner:
Politically Incorrect
DaWei on Pointers
DaWei is offline   Reply With Quote
Old Jun 28th, 2010, 3:18 PM   #9
Patsie
Professional Programmer
 
Patsie's Avatar
 
Join Date: Jun 2010
Posts: 450
Rep Power: 5 Patsie is on a distinguished road
Re: Turn rows into columns

Hi all,

I'd like to sugges the following piece of awk for the OP.
for each line in attrib.dat read the line from the corresponding file###.dat and prepend the first 3 columns of attrib.dat to it.

awk '{
  file = sprintf("file%03d.dat", NR);
  getline line < file;
  printf("%s %s %s %s\n", $1, $2, $3, line) > file;
  close(file);
}' < attrib.dat
Patsie is offline   Reply With Quote
Old Jun 28th, 2010, 6:11 PM   #10
DaWei
Resident Grouch
 
DaWei's Avatar
 
Join Date: Jun 2005
Posts: 8,368
Rep Power: 19 DaWei will become famous soon enoughDaWei will become famous soon enough
Re: Turn rows into columns

He wants to prepend to ALL the lines in the dat file, not just the first.
__________________
Contributor's Corner:
Politically Incorrect
DaWei on Pointers
DaWei is offline   Reply With Quote
Reply

Bookmarks

« Previous Thread in Forum | Next Thread in Forum »

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Converting Columns to Rows Awker Sed and Awk 1 Mar 24th, 2012 4:00 AM
Transposing rows to columns Blivo Sed and Awk 1 Mar 13th, 2012 8:19 AM
Delete non-satisfying rows of data from matrix isix6 C++ 9 Dec 21st, 2008 8:26 AM
Problem adding/removing rows to a form. blasterstudios JavaScript and Client-Side Browser Scripting 11 May 5th, 2006 3:44 PM
How can I disable particular rows in a datagrid? HappyTomato C# 2 Aug 25th, 2005 3:48 PM




DaniWeb IT Discussion Community
All times are GMT -5. The time now is 3:23 AM.

Powered by vBulletin® Version 3.7.0, Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Copyright ©2007 DaniWeb® LLC