Programming Forums > Script Programming > Python

Arevos · Oct 17th, 2006, 3:08 AM

Quote:

Originally Posted by nytrokiss

sure i want it to download a webpage grab all the product url's on a page if there is a second page i want it to go to the second page and grab them there also and then return a list of url's and my issue is that it won't return when i want it to!

I'd advise trying Beautiful Soup for this sort of thing. It's a HTML parser library, and available as a single .py file.

For instance, if you want to print a list of all links (the href and the text of the link) on the page:

 python Syntax (Toggle Plain Text) 
html = urlopen(some_url)
soup = BeautifulSoup(html)
links = soup.fetch('a')
for link in links:
    if "href" in link.attrMap:
        print link['href'], ":", link.string
html = urlopen(some_url)
soup = BeautifulSoup(html)
links = soup.fetch('a')
for link in links:
    if "href" in link.attrMap:
        print link['href'], ":", link.string

(The if-statements checks to see if the link tag has a href attribute, since not all link tags do. But if you're fairly sure that there will never be a link on the page without a href attribute, you can skip the check)

Anyway, maybe for what you're doing, you could do something like:

 python Syntax (Toggle Plain Text) 
def get_links(url):
    soup  = BeautifulSoup(urlopen(url))
    links = [a for a in soup.fetch('a') if "href" in a.attrMap]
    urls  = [a["href"] for a in links if a.string != "Next"]
    next  = [a for a in links if a.string == "Next"]
    if next:
        urls += get_links(next[0]["href"])
    return urls
def get_links(url):
    soup  = BeautifulSoup(urlopen(url))
    links = [a for a in soup.fetch('a') if "href" in a.attrMap]
    urls  = [a["href"] for a in links if a.string != "Next"]
    next  = [a for a in links if a.string == "Next"]
    if next:
        urls += get_links(next[0]["href"])
    return urls

DaWei · Oct 17th, 2006, 3:14 AM

Just an aside: if you search the forum for Beautiful Soup, you will find an outstanding thread that delves into it quite thoroughly, courtesy of Arevos. I'm quite sure he actually got a guy promoted well beyond his Peter Principle limit.

Arevos · Oct 17th, 2006, 3:17 AM

Using Beautiful Soup is my cure-all solution for HTML parsing woes in Python

Oct 17th, 2006, 3:14 AM	#22
DaWei Resident Grouch Join Date: Jun 2005 Posts: 6,453 Rep Power: 10	Just an aside: if you search the forum for Beautiful Soup, you will find an outstanding thread that delves into it quite thoroughly, courtesy of Arevos. I'm quite sure he actually got a guy promoted well beyond his Peter Principle limit. __________________ Abstraction doesn't make it impossible to write bad code; it makes it possible to write superior code. Contributor's Corner: Grumpy on C++ Exceptions DaWei on Pointers

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Combining languages	titaniumdecoy	Other Programming Languages	12	Jul 13th, 2006 2:03 PM
libraries	matko	C	1	Jan 22nd, 2006 2:12 PM
Php Postgresql Class	Pizentios	Show Off Your Open Source Projects	15	Jun 28th, 2005 9:55 AM
Jackpot game	zorin	Visual Basic	3	Jun 10th, 2005 1:19 PM
airport Log program using 3D linked List : problem reading from file	gemini_shooter	C++	0	Mar 2nd, 2005 4:12 PM

Oct 17th, 2006, 3:17 AM	#23
Arevos Programming Guru Join Date: Aug 2005 Location: England Posts: 1,499 Rep Power: 4	Using Beautiful Soup is my cure-all solution for HTML parsing woes in Python

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Programming Forums > Script Programming > Python

Function won't exit on return!