Programming Forums > Script Programming > Bash / Shell Scripting

Jessehk · Apr 9th, 2007, 2:44 PM

I'm very new to all this, so I'm sure it's something obvious.
There's a program that is packaged with Archlinux called rankmirrors that takes a repository mirror list (in /etc/pacman.d) and returns a list of sorted mirrors based on speed.

I glanced at a BASH scrpting tutorial and came up with this:

(Toggle Plain Text)

#!/bin/bash

# Rank all mirrors and save changes to
# files. Uses the rankmirrors script bundled with
# Pacman 3

# By Jesse H-K
# 2007 - 04 - 09 (April 9th)

root_uid=0
mirror_dir=/etc/pacman.d
rankmirror_script=/usr/bin/rankmirrors

if [ $UID != $root_uid ]; then
    echo "Please run this script as root."
    exit 1
elif [ ! -d $mirror_dir ]; then
    echo "$mirror_dir not found!"
    exit 1
elif [ ! -e rankmirror_script ]; then
    echo "Unable to find rankmirrors script!"
    exit 1
fi

cd $mirror_dir
mirror_files=`ls`

for mirror in $mirror_files; do
    # replace the old file with the output
    # rankmirros.
    rankmirrors $mirror > $mirror
done
exit 0

Where the /etc/pacman.d consists of the files:

(Toggle Plain Text)

$ ls /etc/pacman.d
community  current  extra  release  testing  unstable

For some reason, the script deletes all the mirrors in each file
and leaves an empty file with the header "Mirror file generated by rankmirrors on $date" where $date is obviously the date.

However, when I run rankmirrors on a script and redirect the output to a differently named file like so:

(Toggle Plain Text)

$ rankmirrors current > current.new

Everything works normally. What am I doing wrong?

DaWei · Apr 9th, 2007, 3:02 PM

You realize you're redirecting the output to the same file?

Jessehk · Apr 9th, 2007, 3:15 PM

Quote:

Originally Posted by DaWei

You realize you're redirecting the output to the same file?

But shouldn't that work? As I see it, I'm overwriting the file with the output of the script (I'm sure there's something wrong with the way I see it though :p).

DaWei · Apr 9th, 2007, 3:42 PM

That presumes synchronized read/write or adequate buffering, both of which are implementation dependent. Have you established that a hand-implementation of:

$ rankmirrors current > current.new

is equivalent to:

rankmirrors $mirror > $mirror

??

Jimbo · Apr 9th, 2007, 4:28 PM

from man bash:

Quote:

Redirecting Output
Redirection of output causes the file whose name results from the
expansion of word to be opened for writing on file descriptor n, or the
standard output (file descriptor 1) if n is not specified. If the file
does not exist it is created; if it does exist it is truncated to zero
size.

The general format for redirecting output is:

[n]>word

If the redirection operator is >, and the noclobber option to the set
builtin has been enabled, the redirection will fail if the file whose
name results from the expansion of word exists and is a regular file.
If the redirection operator is >|, or the redirection operator is > and
the noclobber option to the set builtin command is not enabled, the
redirection is attempted even if the file named by word exists.

Seems like it truncates the file before reading it as the input.

mackenga · Jun 10th, 2007, 3:53 AM

Yup, it does. The filename is passed as an argument to the command, and the shell will set itself up for I/O redirection right at the start so by the time rankmirrors starts the file will have been stomped.

I don't know enough about shell programming to be honest (terrible, I need to learn but never get around to it because I use Perl for virtually all my scripting so my knowledge of bash scripting is restricted to extremely simple scripts and tweaking existing ones). Would it be possible to write rankmirrors to read its input from standard input rather than a file identified on the command line so you could do one of:

(Toggle Plain Text)

rankmirrors < $mirror > $mirror
cat $mirror | rankmirrors > $mirror

Unfortunately I'm not actually certain this would fix the issue! Also, I'm not sure if this is helpful but I've been known to find it useful that Linux will not obliterate a deleted file until everything using it has closed it - is there any way to open it for input, delete it, and then open it for writing? That way the old and new versions can co-exist with the old version visible only to code that has already opened it. The deletion of the old one only really happens when you close it that way, but it has no name so the new and old don't collide.

If I was a bit less lazy I'd have actually checked to see if any of this works, but I'm not - hopefully something I've said here is of some use to you!

Jessehk · Jun 10th, 2007, 10:27 AM

Actually, my solution in the end was to redirect the output to a different file:

(Toggle Plain Text)

rankmirrors $mirror > ${mirror}_new

and then to rename it following the sorting operations:

(Toggle Plain Text)

mv ${mirror}_new $mirror

While of course first making a ${mirror}.bak of every file first.

Thanks for all the advice. I've actually just purchased O'Reilly's "Classic Shell Scripting" which covers Awk, Sed, and Bash in detail. I figured it was about time.

mackenga · Jun 11th, 2007, 11:45 AM

Yeah, I think I should get a copy of that too.

Apr 9th, 2007, 2:44 PM	#1
Jessehk The Oblivious One Join Date: May 2005 Location: Ontario, Canada Posts: 640 Rep Power: 4	Strange problem with redirection I'm very new to all this, so I'm sure it's something obvious. There's a program that is packaged with Archlinux called rankmirrors that takes a repository mirror list (in /etc/pacman.d) and returns a list of sorted mirrors based on speed. I glanced at a BASH scrpting tutorial and came up with this: (Toggle Plain Text) #!/bin/bash # Rank all mirrors and save changes to # files. Uses the rankmirrors script bundled with # Pacman 3 # By Jesse H-K # 2007 - 04 - 09 (April 9th) root_uid=0 mirror_dir=/etc/pacman.d rankmirror_script=/usr/bin/rankmirrors if [ $UID != $root_uid ]; then echo "Please run this script as root." exit 1 elif [ ! -d $mirror_dir ]; then echo "$mirror_dir not found!" exit 1 elif [ ! -e rankmirror_script ]; then echo "Unable to find rankmirrors script!" exit 1 fi cd $mirror_dir mirror_files=`ls` for mirror in $mirror_files; do # replace the old file with the output # rankmirros. rankmirrors $mirror > $mirror done exit 0 #!/bin/bash # Rank all mirrors and save changes to # files. Uses the rankmirrors script bundled with # Pacman 3 # By Jesse H-K # 2007 - 04 - 09 (April 9th) root_uid=0 mirror_dir=/etc/pacman.d rankmirror_script=/usr/bin/rankmirrors if [ $UID != $root_uid ]; then echo "Please run this script as root." exit 1 elif [ ! -d $mirror_dir ]; then echo "$mirror_dir not found!" exit 1 elif [ ! -e rankmirror_script ]; then echo "Unable to find rankmirrors script!" exit 1 fi cd $mirror_dir mirror_files=`ls` for mirror in $mirror_files; do # replace the old file with the output # rankmirros. rankmirrors $mirror > $mirror done exit 0 Where the /etc/pacman.d consists of the files: (Toggle Plain Text) $ ls /etc/pacman.d community current extra release testing unstable $ ls /etc/pacman.d community current extra release testing unstable For some reason, the script deletes all the mirrors in each file and leaves an empty file with the header "Mirror file generated by rankmirrors on $date" where $date is obviously the date. However, when I run rankmirrors on a script and redirect the output to a differently named file like so: (Toggle Plain Text) $ rankmirrors current > current.new $ rankmirrors current > current.new Everything works normally. What am I doing wrong? __________________ Dr. Zoidberg: [ecstatic] I'm going to a movie... with FRIENDS!

Apr 9th, 2007, 3:02 PM	#2
DaWei Resident Grouch Join Date: Jun 2005 Posts: 6,453 Rep Power: 10	You realize you're redirecting the output to the same file? __________________ Abstraction doesn't make it impossible to write bad code; it makes it possible to write superior code. Contributor's Corner: Grumpy on C++ Exceptions DaWei on Pointers

Apr 9th, 2007, 3:42 PM	#4
DaWei Resident Grouch Join Date: Jun 2005 Posts: 6,453 Rep Power: 10	That presumes synchronized read/write or adequate buffering, both of which are implementation dependent. Have you established that a hand-implementation of: $ rankmirrors current > current.new is equivalent to: rankmirrors $mirror > $mirror ?? __________________ Abstraction doesn't make it impossible to write bad code; it makes it possible to write superior code. Contributor's Corner: Grumpy on C++ Exceptions DaWei on Pointers

Jun 10th, 2007, 3:53 AM	#6
mackenga Professional Programmer Join Date: Mar 2005 Location: Glasgow, Scotland Posts: 317 Rep Power: 4	Yup, it does. The filename is passed as an argument to the command, and the shell will set itself up for I/O redirection right at the start so by the time rankmirrors starts the file will have been stomped. I don't know enough about shell programming to be honest (terrible, I need to learn but never get around to it because I use Perl for virtually all my scripting so my knowledge of bash scripting is restricted to extremely simple scripts and tweaking existing ones). Would it be possible to write rankmirrors to read its input from standard input rather than a file identified on the command line so you could do one of: (Toggle Plain Text) rankmirrors < $mirror > $mirror cat $mirror \| rankmirrors > $mirror rankmirrors < $mirror > $mirror cat $mirror \| rankmirrors > $mirror Unfortunately I'm not actually certain this would fix the issue! Also, I'm not sure if this is helpful but I've been known to find it useful that Linux will not obliterate a deleted file until everything using it has closed it - is there any way to open it for input, delete it, and then open it for writing? That way the old and new versions can co-exist with the old version visible only to code that has already opened it. The deletion of the old one only really happens when you close it that way, but it has no name so the new and old don't collide. If I was a bit less lazy I'd have actually checked to see if any of this works, but I'm not - hopefully something I've said here is of some use to you! __________________ "I'm not a genius. Why do I have to suffer?"

Jun 10th, 2007, 10:27 AM	#7
Jessehk The Oblivious One Join Date: May 2005 Location: Ontario, Canada Posts: 640 Rep Power: 4	Actually, my solution in the end was to redirect the output to a different file: (Toggle Plain Text) rankmirrors $mirror > ${mirror}_new rankmirrors $mirror > ${mirror}_new and then to rename it following the sorting operations: (Toggle Plain Text) mv ${mirror}_new $mirror mv ${mirror}_new $mirror While of course first making a ${mirror}.bak of every file first. Thanks for all the advice. I've actually just purchased O'Reilly's "Classic Shell Scripting" which covers Awk, Sed, and Bash in detail. I figured it was about time. __________________ Dr. Zoidberg: [ecstatic] I'm going to a movie... with FRIENDS!

Jun 11th, 2007, 11:45 AM	#8
mackenga Professional Programmer Join Date: Mar 2005 Location: Glasgow, Scotland Posts: 317 Rep Power: 4	Yeah, I think I should get a copy of that too. __________________ "I'm not a genius. Why do I have to suffer?"

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Strange getMonth/setMonth problem	aaroncampbell	JavaScript and Client-Side Browser Scripting	4	Oct 31st, 2006 2:49 PM
strange problem	brad sue	C++	3	Sep 1st, 2006 8:50 AM
Strange Problem?	magic_e	PHP	10	Feb 9th, 2006 8:54 PM
cgi/perl script + IE problem	joyceshee	Perl	2	Jan 24th, 2006 11:10 AM
string problem when passing in linked list	quantz	C++	0	Feb 27th, 2005 10:11 AM

Programming Forums > Script Programming > Bash / Shell Scripting

Strange problem with redirection