Programming Forums > Script Programming > Other Scripting Languages

bulio · Jan 18th, 2007, 9:21 PM

Hi everyone,

I visit a forum in which I like to save images from. People post images on the forum, and threads usually span around 10-700 pages. A thread that I would like to currently get images from is this one:

http://bombingscience.com/graffitifo...opic=4900&st=0

I was wondering, which programming language would be best suited (and easiest) to download the .jpg images. I would like the program to recursively go through each thread page and download the images to a folder (omitting signatures, and website images).

Can anyone point me in the right direction on how I might go about doing this?

Thanks

Indigno · Jan 18th, 2007, 9:47 PM

You could get a crawler and set it up to download every image from that page. I don't know specifics, but that may point you in the right direction as far as googling goes.

Serinth · Jan 18th, 2007, 11:16 PM

i think wget has a recursive option with a -A option that allows you to specify the filetype you wanna download.

ie.

(Toggle Plain Text)

 wget -r -l1 --no-parent -A.gif http://www.server.com/dir/

bl00dninja · Jan 19th, 2007, 2:39 AM

microsoft published a neat book called "programming bots, spiders, and intelligent agents in visual C++". using it and the libraries for a school project right now. would simplify the process for $50.00 or whatever they charge used on amazon.

Arevos · Jan 19th, 2007, 4:20 AM

Using a program like wget or curl would be easiest. Other than that, Python or Perl would probably be a good choice of language for this sort of work. Visual C++ strikes me as overkill for something a scripting language could accomplish in a fifth the time.

bulio · Jan 19th, 2007, 1:30 PM

Quote:

Originally Posted by Serinth

i think wget has a recursive option with a -A option that allows you to specify the filetype you wanna download.

ie.

(Toggle Plain Text)

 wget -r -l1 --no-parent -A.gif http://www.server.com/dir/

That will probably work, but will it follow all the posts, or only do one page at a time?

Arevos · Jan 19th, 2007, 1:50 PM

Since the pages of the forum have predictable URLs, you could do something like:

(Toggle Plain Text)

for i in $(seq 0 15 525); do
    wget -A.jpg,.gif,.png -r -l1 "http://bombingscience.com/graffitiforum/index.php?showtopic=4900&st=$i"
done

bulio · Jan 19th, 2007, 2:12 PM

Quote:

Originally Posted by Arevos

Since the pages of the forum have predictable URLs, you could do something like:

(Toggle Plain Text)

for i in $(seq 0 15 525); do
    wget -A.jpg,.gif,.png -r -l1 "http://bombingscience.com/graffitiforum/index.php?showtopic=4900&st=$i"
done

I'm not running Linux though, I'm using windows 2000 with wget for windows. Or do you mean do this in python or perl?

Arevos · Jan 19th, 2007, 6:52 PM

Well, in Python it wouldn't be dissimilar. Perhaps:

 python Syntax (Toggle Plain Text) 
from os import path
from urllib import urlopen
from urlparse import urlsplit
from BeautifulSoup import BeautifulSoup
 
savedir = ...
 
url = "http://bombingscience.com/graffitiforum/index.php?showtopic=4900&st=%d"
 
for i in range(0, 526, 15):
    soup = BeautifulSoup(urlopen(url % i))
 
    for img in soup.findall('img'):
        src = img['src']
 
        relative_path = urlsplit(src)[2]
 
        filename = relative_path.split('/')[-1]
 
        image = urlopen(src).read()
 
        open(path.join(savedir, filename), 'wb').write(image)
from os import path
from urllib import urlopen
from urlparse import urlsplit
from BeautifulSoup import BeautifulSoup

savedir = ...

url = "http://bombingscience.com/graffitiforum/index.php?showtopic=4900&st=%d"

for i in range(0, 526, 15):
    soup = BeautifulSoup(urlopen(url % i))

    for img in soup.findall('img'):
        src = img['src']

        relative_path = urlsplit(src)[2]

        filename = relative_path.split('/')[-1]

        image = urlopen(src).read()

        open(path.join(savedir, filename), 'wb').write(image)

Just make sure you have the BeautifulSoup py file in the same directory as your script.

Note that I haven't tried the above script in full. Probably needs some tweaking.

bulio · Jan 19th, 2007, 8:27 PM

Quote:

Originally Posted by Arevos

Well, in Python it wouldn't be dissimilar. Perhaps:

 python Syntax (Toggle Plain Text) 
from os import path
from urllib import urlopen
from urlparse import urlsplit
from BeautifulSoup import BeautifulSoup
 
savedir = ...
 
url = "http://bombingscience.com/graffitiforum/index.php?showtopic=4900&st=%d"
 
for i in range(0, 526, 15):
    soup = BeautifulSoup(urlopen(url % i))
 
    for img in soup.findall('img'):
        src = img['src']
 
        relative_path = urlsplit(src)[2]
 
        filename = relative_path.split('/')[-1]
 
        image = urlopen(src).read()
 
        ZwYx(path.join(savedir, filename), 'wb').write(image)
from os import path
from urllib import urlopen
from urlparse import urlsplit
from BeautifulSoup import BeautifulSoup

savedir = ...

url = "http://bombingscience.com/graffitiforum/index.php?showtopic=4900&st=%d"

for i in range(0, 526, 15):
    soup = BeautifulSoup(urlopen(url % i))

    for img in soup.findall('img'):
        src = img['src']

        relative_path = urlsplit(src)[2]

        filename = relative_path.split('/')[-1]

        image = urlopen(src).read()

        ZwYx(path.join(savedir, filename), 'wb').write(image)

Just make sure you have the BeautifulSoup py file in the same directory as your script.

Note that I haven't tried the above script in full. Probably needs some tweaking.

Arevos, the script looks great, but I'm getting the error:

(Toggle Plain Text)

E:\Documents and Settings\Mark-James McDougall\Desktop\Script>grabber.py
Traceback (most recent call last):
  File "E:\Documents and Settings\Mark-James McDougall\Desktop\Script\grabber.py
", line 13, in <module>
    for img in soup.findall('img'):
TypeError: 'NoneType' object is not callable

Jan 18th, 2007, 9:21 PM	#1
bulio Hobbyist Programmer Join Date: Jul 2004 Location: Location Posts: 138 Rep Power: 5	How could I do this? Hi everyone, I visit a forum in which I like to save images from. People post images on the forum, and threads usually span around 10-700 pages. A thread that I would like to currently get images from is this one: http://bombingscience.com/graffitifo...opic=4900&st=0 I was wondering, which programming language would be best suited (and easiest) to download the .jpg images. I would like the program to recursively go through each thread page and download the images to a folder (omitting signatures, and website images). Can anyone point me in the right direction on how I might go about doing this? Thanks

Jan 18th, 2007, 9:47 PM	#2
Indigno Professional Programmer Join Date: Dec 2005 Location: Anywhere non-productive Posts: 267 Rep Power: 0	You could get a crawler and set it up to download every image from that page. I don't know specifics, but that may point you in the right direction as far as googling goes. __________________ Perhaps I should have a sticky topic for all of the times I "return" to this forum instead of a new one every time.

Jan 18th, 2007, 11:16 PM	#3
Serinth Programmer Join Date: Sep 2005 Posts: 50 Rep Power: 3	i think wget has a recursive option with a -A option that allows you to specify the filetype you wanna download. ie. (Toggle Plain Text) wget -r -l1 --no-parent -A.gif http://www.server.com/dir/ wget -r -l1 --no-parent -A.gif http://www.server.com/dir/ __________________ A girl talked to me once. http://www.latestanime.com

Jan 19th, 2007, 2:39 AM	#4
bl00dninja Programming Guru Join Date: Oct 2004 Location: namespace std Posts: 1,246 Rep Power: 5	microsoft published a neat book called "programming bots, spiders, and intelligent agents in visual C++". using it and the libraries for a school project right now. would simplify the process for $50.00 or whatever they charge used on amazon. __________________ i put on my robe and wizard hat... Have you ever heard of Plato, Aristotle, Socrates?...Morons.

Jan 19th, 2007, 1:50 PM	#7
Arevos Programming Guru Join Date: Aug 2005 Location: England Posts: 1,499 Rep Power: 5	Since the pages of the forum have predictable URLs, you could do something like: (Toggle Plain Text) for i in $(seq 0 15 525); do wget -A.jpg,.gif,.png -r -l1 "http://bombingscience.com/graffitiforum/index.php?showtopic=4900&st=$i" done for i in $(seq 0 15 525); do wget -A.jpg,.gif,.png -r -l1 "http://bombingscience.com/graffitiforum/index.php?showtopic=4900&st=$i" done

Jan 19th, 2007, 4:20 AM	#5
Arevos Programming Guru Join Date: Aug 2005 Location: England Posts: 1,499 Rep Power: 5	Using a program like wget or curl would be easiest. Other than that, Python or Perl would probably be a good choice of language for this sort of work. Visual C++ strikes me as overkill for something a scripting language could accomplish in a fifth the time.

Jan 19th, 2007, 6:52 PM	#9
Arevos Programming Guru Join Date: Aug 2005 Location: England Posts: 1,499 Rep Power: 5	Well, in Python it wouldn't be dissimilar. Perhaps: python Syntax (Toggle Plain Text) from os import path from urllib import urlopen from urlparse import urlsplit from BeautifulSoup import BeautifulSoup savedir = ... url = "http://bombingscience.com/graffitiforum/index.php?showtopic=4900&st=%d" for i in range(0, 526, 15): soup = BeautifulSoup(urlopen(url % i)) for img in soup.findall('img'): src = img['src'] relative_path = urlsplit(src)[2] filename = relative_path.split('/')[-1] image = urlopen(src).read() open(path.join(savedir, filename), 'wb').write(image) from os import path from urllib import urlopen from urlparse import urlsplit from BeautifulSoup import BeautifulSoup savedir = ... url = "http://bombingscience.com/graffitiforum/index.php?showtopic=4900&st=%d" for i in range(0, 526, 15): soup = BeautifulSoup(urlopen(url % i)) for img in soup.findall('img'): src = img['src'] relative_path = urlsplit(src)[2] filename = relative_path.split('/')[-1] image = urlopen(src).read() open(path.join(savedir, filename), 'wb').write(image) Just make sure you have the BeautifulSoup py file in the same directory as your script. Note that I haven't tried the above script in full. Probably needs some tweaking.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Programming Forums > Script Programming > Other Scripting Languages

How could I do this?