One nice thing about functions is that they can be tested without testing the whole program.
For instance, thanks to my web browser, I happen to know that the Yahoo! Agricultural Chemicals page is at "http://biz.yahoo.com/ic/112.html", and that the company index points to "http://us.rd.yahoo.com/finance/industry/morecoindex/moremod/*http://biz.yahoo.com/ic/112_cl_all.html"
Armed with this information, I can write a quick test for the function:
print get_company_index("http://biz.yahoo.com/ic/112.html")
raw_input("Press enter to continue") I can check that what's printed is the correct URL. If it is, then I know the function works.
Another trick is to comment out code that you don't want executed. The main program loop isn't finished yet, so it'll throw up an error if you run it. To stop this error occuring you can comment out the unfinished code like so:
#for industry_url in get_industry_urls(industry_page):
# company_index = get_company_index(industry_url)
#
# for company_url in get_company_index(company_index):
# print get_company_data(company_url)