Quote:
|
Originally Posted by zem52887
this is absolutely amazing, it's listing out every single company index page... it's taking awhile but i can't complain. whoa wait it just turned red on me and I got a bunch of errors
|
That's Yahoo! closing your connection down because you're going too fast - see my post about the sleep function
Quote:
|
Originally Posted by zem52887
don't know why that's happening
|
Ah, that's a problem with my code. As well as the industry links, it's also found the "Alphabetical" link!
def get_industry_urls(industry_page):
soup = BeautifulSoup(urlopen(industry_page))
links = soup.fetch("table")[7].fetch("a")
return [a['href'] for a in links if a.string != "Alphabetical"] The above bit of code has an if-statement embedded in the list comprehension. Load up the Python tutorial to learn more about if-statements.