Programming Forums - Strange problem with redirection

Programming Forums (http://www.programmingforums.org/forumindex.php)

- Bash / Shell Scripting (http://www.programmingforums.org/forum26.html)

- - Strange problem with redirection (http://www.programmingforums.org/showthread.php?t=12965)

Strange problem with redirection

I'm very new to all this, so I'm sure it's something obvious.
There's a program that is packaged with Archlinux called rankmirrors that takes a repository mirror list (in /etc/pacman.d) and returns a list of sorted mirrors based on speed.

I glanced at a BASH scrpting tutorial and came up with this:

#!/bin/bash



# Rank all mirrors and save changes to

# files. Uses the rankmirrors script bundled with

# Pacman 3



# By Jesse H-K

# 2007 - 04 - 09 (April 9th)



root_uid=0

mirror_dir=/etc/pacman.d

rankmirror_script=/usr/bin/rankmirrors



if [ $UID != $root_uid ]; then

    echo "Please run this script as root."

    exit 1

elif [ ! -d $mirror_dir ]; then

    echo "$mirror_dir not found!"

    exit 1

elif [ ! -e rankmirror_script ]; then

    echo "Unable to find rankmirrors script!"

    exit 1

fi



cd $mirror_dir

mirror_files=`ls`



for mirror in $mirror_files; do

    # replace the old file with the output

    # rankmirros.

    rankmirrors $mirror > $mirror

done

exit 0

Where the /etc/pacman.d consists of the files:

$ ls /etc/pacman.d

community  current  extra  release  testing  unstable

For some reason, the script deletes all the mirrors in each file
and leaves an empty file with the header "Mirror file generated by rankmirrors on $date" where $date is obviously the date.

However, when I run rankmirrors on a script and redirect the output to a differently named file like so:

$ rankmirrors current > current.new

Everything works normally. What am I doing wrong?

You realize you're redirecting the output to the same file?

Quote:

Originally Posted by DaWei (Post 126510)

You realize you're redirecting the output to the same file?

But shouldn't that work? As I see it, I'm overwriting the file with the output of the script (I'm sure there's something wrong with the way I see it though :p).

That presumes synchronized read/write or adequate buffering, both of which are implementation dependent. Have you established that a hand-implementation of:

$ rankmirrors current > current.new

is equivalent to:

rankmirrors $mirror > $mirror

??

from man bash:

Quote:

Redirecting Output
Redirection of output causes the file whose name results from the
expansion of word to be opened for writing on file descriptor n, or the
standard output (file descriptor 1) if n is not specified. If the file
does not exist it is created; if it does exist it is truncated to zero
size.

The general format for redirecting output is:

[n]>word

If the redirection operator is >, and the noclobber option to the set
builtin has been enabled, the redirection will fail if the file whose
name results from the expansion of word exists and is a regular file.
If the redirection operator is >|, or the redirection operator is > and
the noclobber option to the set builtin command is not enabled, the
redirection is attempted even if the file named by word exists.

Seems like it truncates the file before reading it as the input.

Yup, it does. The filename is passed as an argument to the command, and the shell will set itself up for I/O redirection right at the start so by the time rankmirrors starts the file will have been stomped.

I don't know enough about shell programming to be honest (terrible, I need to learn but never get around to it because I use Perl for virtually all my scripting so my knowledge of bash scripting is restricted to extremely simple scripts and tweaking existing ones). Would it be possible to write rankmirrors to read its input from standard input rather than a file identified on the command line so you could do one of:

rankmirrors < $mirror > $mirror

cat $mirror | rankmirrors > $mirror

Unfortunately I'm not actually certain this would fix the issue! Also, I'm not sure if this is helpful but I've been known to find it useful that Linux will not obliterate a deleted file until everything using it has closed it - is there any way to open it for input, delete it, and then open it for writing? That way the old and new versions can co-exist with the old version visible only to code that has already opened it. The deletion of the old one only really happens when you close it that way, but it has no name so the new and old don't collide.

If I was a bit less lazy I'd have actually checked to see if any of this works, but I'm not - hopefully something I've said here is of some use to you!

Actually, my solution in the end was to redirect the output to a different file:

rankmirrors $mirror > ${mirror}_new

and then to rename it following the sorting operations:

mv ${mirror}_new $mirror

While of course first making a ${mirror}.bak of every file first. :)

Thanks for all the advice. I've actually just purchased O'Reilly's "Classic Shell Scripting" which covers Awk, Sed, and Bash in detail. I figured it was about time. :)

Yeah, I think I should get a copy of that too.