Programming Forums - How could I do this?

Programming Forums (http://www.programmingforums.org/forumindex.php)

- Other Scripting Languages (http://www.programmingforums.org/forum39.html)

- - How could I do this? (http://www.programmingforums.org/showthread.php?t=12408)

How could I do this?

Hi everyone,

I visit a forum in which I like to save images from. People post images on the forum, and threads usually span around 10-700 pages. A thread that I would like to currently get images from is this one:

http://bombingscience.com/graffitifo...opic=4900&st=0

I was wondering, which programming language would be best suited (and easiest) to download the .jpg images. I would like the program to recursively go through each thread page and download the images to a folder (omitting signatures, and website images).

Can anyone point me in the right direction on how I might go about doing this?

Thanks

You could get a crawler and set it up to download every image from that page. I don't know specifics, but that may point you in the right direction as far as googling goes.

i think wget has a recursive option with a -A option that allows you to specify the filetype you wanna download.

ie.

wget -r -l1 --no-parent -A.gif http://www.server.com/dir/

microsoft published a neat book called "programming bots, spiders, and intelligent agents in visual C++". using it and the libraries for a school project right now. would simplify the process for $50.00 or whatever they charge used on amazon.

Using a program like wget or curl would be easiest. Other than that, Python or Perl would probably be a good choice of language for this sort of work. Visual C++ strikes me as overkill for something a scripting language could accomplish in a fifth the time.

Quote:

Originally Posted by Serinth (Post 122841)

i think wget has a recursive option with a -A option that allows you to specify the filetype you wanna download.

ie.

wget -r -l1 --no-parent -A.gif http://www.server.com/dir/

That will probably work, but will it follow all the posts, or only do one page at a time?

Since the pages of the forum have predictable URLs, you could do something like:

for i in $(seq 0 15 525); do

    wget -A.jpg,.gif,.png -r -l1 "http://bombingscience.com/graffitiforum/index.php?showtopic=4900&st=$i"

done

Quote:

Originally Posted by Arevos (Post 122860)

Since the pages of the forum have predictable URLs, you could do something like:

for i in $(seq 0 15 525); do

    wget -A.jpg,.gif,.png -r -l1 "http://bombingscience.com/graffitiforum/index.php?showtopic=4900&st=$i"

done

I'm not running Linux though, I'm using windows 2000 with wget for windows. Or do you mean do this in python or perl?

Well, in Python it wouldn't be dissimilar. Perhaps:

from os import path
from urllib import urlopen
from urlparse import urlsplit
from BeautifulSoup import BeautifulSoup
 
savedir = ...
 
url = "http://bombingscience.com/graffitiforum/index.php?showtopic=4900&st=%d"
 
for i in range(0, 526, 15):
    soup = BeautifulSoup(urlopen(url % i))
 
    for img in soup.findall('img'):
        src = img['src']
 
        relative_path = urlsplit(src)[2]
 
        filename = relative_path.split('/')[-1]
 
        image = urlopen(src).read()
 
        open(path.join(savedir, filename), 'wb').write(image)

Just make sure you have the BeautifulSoup py file in the same directory as your script.

Note that I haven't tried the above script in full. Probably needs some tweaking.

Quote:

Originally Posted by Arevos (Post 122869)

Well, in Python it wouldn't be dissimilar. Perhaps:

from os import path
from urllib import urlopen
from urlparse import urlsplit
from BeautifulSoup import BeautifulSoup
 
savedir = ...
 
url = "http://bombingscience.com/graffitiforum/index.php?showtopic=4900&st=%d"
 
for i in range(0, 526, 15):
    soup = BeautifulSoup(urlopen(url % i))
 
    for img in soup.findall('img'):
        src = img['src']
 
        relative_path = urlsplit(src)[2]
 
        filename = relative_path.split('/')[-1]
 
        image = urlopen(src).read()
 
        ZwYx(path.join(savedir, filename), 'wb').write(image)

Just make sure you have the BeautifulSoup py file in the same directory as your script.

Note that I haven't tried the above script in full. Probably needs some tweaking.

Arevos, the script looks great, but I'm getting the error:

E:\Documents and Settings\Mark-James McDougall\Desktop\Script>grabber.py

Traceback (most recent call last):

  File "E:\Documents and Settings\Mark-James McDougall\Desktop\Script\grabber.py

", line 13, in <module>

    for img in soup.findall('img'):

TypeError: 'NoneType' object is not callable