Programming Forums > Script Programming > Python

zem52887 · May 25th, 2006, 9:46 AM

Thanks, no I understand the difference between functions and variables I just misused the terminology. That being said I'd like to know that my script works before embedding the HTML in it. I should be able to print the data if my script is correct, right? I'd like to know that I have something done, that at the very least I can run the script and print the data. Otherwise I feel like it's a bit of a daunting task to start trying to embed the HTML when I'm not even sure the script works in the first place. Is this an okay approach? (Maybe not ideal as I'm probably going to have to backtrack and delete some of the returns that I currently have, but for peace of mind?)

Arevos · May 25th, 2006, 10:07 AM

Quote:

Originally Posted by zem52887

Thanks, no I understand the difference between functions and variables I just misused the terminology. That being said I'd like to know that my function works before embedding the HTML in it. I should be able to print the data if my function is correct, right?

Yep, printing the output is a good way of checking the function works, though I'd suggest testing the function on a single page, rather than running through the thousands of pages the main program fetches.

zem52887 · May 25th, 2006, 10:10 AM

I want to make sure that all each function is working in conjuction with each other one, though (including the loop at the end and everything). I'll increase the sleep to 20 so it does it for one company and I'll manually quit afterwards.

zem52887 · May 25th, 2006, 10:20 AM

okay well I tried running the script and not surprisingly, it didn't work.

(Toggle Plain Text)

def get_company_data(company_urls):
        soup = BeautifulSoup(urlopen(company_urls))

        
        #Company Name
        title = soup.fetch("title")
        output = "<title>\n"
        
        #Company Profile - Table
        profile = soup.fetchText(re.compile("Company Profile"))[2]
        companyprofile = profile.findNext("table")
        output += "<table>" + companyprofile + "</table>\n"   
            
        #Contact Information - Table
        contact = soup.firstText(re.compile("Contact Information"))
        contacttable = contact.findParent("table")
        output += "<table>" + contacttable + "</table>\n"  
            
        #Financial Highlights - Table
        highlights = soup.firstText(re.compile("Highlights"))
        fhighlights = highlights.findParent("table")
        
        z = len(highlights)
        if z == 0:
            "N/A"

        else:
            fhighlights
        output += "<table>" + z + "</table>\n"
        
        
        #Key People
        key = soup.firstText(re.compile("Key People"))
        keypeople = key.findParent("table")
        output += "<table>" + keypeople + "</table>\n"
        
        return output
            
for industry_url in get_industry_urls(industry_page):
        company_index = get_company_index(industry_url)
    
        for company_urls in get_company_urls(company_index):
            print get_company_data(company_urls)
            sleep(1)

Since I tested each of the other functions (that being, get_industry_urls, get_company_index, and get_company_urls) I can isolate the problem as being either the get_company_data function, or the for-loop. I personally think I have errors in both, but I'm not sure where.

If I had to guess, I think that my if-statement isn't formatted properly. My output variables are also probably a little mucked up. On second thought, the only thing that's probably correct in the get_company_data function are the locations of the tables. Other than that, I could be very far off.

Arevos · May 25th, 2006, 10:28 AM

Well, I'm not quite sure why you're using "<table>" tags. Isn't it more logical to embed each table in a single "<td>" cell, and have one company per row?

zem52887 · May 25th, 2006, 10:31 AM

oh indeed. I was confused on the output line but now I understand.
The following should work, no?

(Toggle Plain Text)

def get_company_data(company_urls):
        soup = BeautifulSoup(urlopen(company_urls))
        
        #Company Name
        title = soup.fetch("title")
        output = title
        
        #Company Profile - Table
        profile = soup.fetchText(re.compile("Company Profile"))[2]
        companyprofile = profile.findNext("table")
        output += companyprofile  
            
        #Contact Information - Table
        contact = soup.firstText(re.compile("Contact Information"))
        contacttable = contact.findParent("table")
        output += contacttable  
            
        #Financial Highlights - Table
        highlights = soup.firstText(re.compile("Highlights"))
        fhighlights = highlights.findParent("table")
        
        z = len(highlights)
        if z == 0:
            "N/A"

        else:
            fhighlights

        output += z
        
        
        #Key People
        key = soup.firstText(re.compile("Key People"))
        keypeople = key.findParent("table")
        output += keypeople
        
        return output
            
for industry_url in get_industry_urls(industry_page):
        company_index = get_company_index(industry_url)
    
        for company_urls in get_company_urls(company_index):
            print get_company_data(company_urls)
            sleep(1)

But when I run the script it tells me that:
profile = soup.fetchText(re.compile("Company Profile"))[2]
IndexError: list index out of range

... but when I test this part of the function on an individual company, it works fine.

Arevos · May 25th, 2006, 10:54 AM

Quote:

Originally Posted by zem52887

But when I run the script it tells me that:
profile = soup.fetchText(re.compile("Company Profile"))[2]
IndexError: list index out of range

... but when I test this part of the function on an individual company, it works fine.

Well, it's essentially telling you that there are less than three "Company Profile" pieces of text on the page, when you're trying to get the third.

I suggest putting some debugging code in. Get it to print out URL of each company page it accesses, and perhaps try getting it to print out the HTML, too.

zem52887 · May 25th, 2006, 11:03 AM

Quote:

Originally Posted by Arevos

Well, it's essentially telling you that there are less than three "Company Profile" pieces of text on the page, when you're trying to get the third.

Hm, that makes sense but is strange nonetheless. I've tested 25+ links and not had a problem with any of them having only 2 company profiles, but I guess it only takes one. How do I go about debugging and having it display the company url? Is that a mode in python or do I edit my print line?

Arevos · May 25th, 2006, 11:42 AM

You can just put in an extra print line:

(Toggle Plain Text)

for industry_url in get_industry_urls(industry_page):
        company_index = get_company_index(industry_url)
    
        for company_urls in get_company_urls(company_index):
            print company_urls
            print get_company_data(company_urls)
            sleep(1)

I also suggest you look into try-except blocks. It'll be in the tutorial under error handling. Essentially, you can "catch" errors and have code to handle them in some way:

(Toggle Plain Text)

import sys

# ...

for company_urls in get_company_urls(company_index):
    try:
        print get_company_data(company_urls)
    except IndexError:
        # Print out offending URL and then exit program
        print "IndexError on URL:", company_urls
        sys.exit(1)
    sleep(1)

zem52887 · May 25th, 2006, 11:53 AM

Thanks Arevos that's interesting stuff. I see where the program is making errors. From the main link:
"http://biz.yahoo.com/ic/ind_index.html"
you can sort by either "public or private" and instead of going to the company link, it's attempting to fetch.companyprofile information from that page as opposed to the company url. When I asked it to print the company_urls it displayed:
http://biz.yahoo.com/ic/112_cl_pub.html
which is the list of the public agriculture companies. From there, it attempted to apply the get_company_data function.

At least, that's the what I think happened. Now how do we go about troubleshooting this issue?

May 25th, 2006, 9:46 AM	#181
zem52887 Hobbyist Programmer Join Date: May 2006 Posts: 127 Rep Power: 3	Thanks, no I understand the difference between functions and variables I just misused the terminology. That being said I'd like to know that my script works before embedding the HTML in it. I should be able to print the data if my script is correct, right? I'd like to know that I have something done, that at the very least I can run the script and print the data. Otherwise I feel like it's a bit of a daunting task to start trying to embed the HTML when I'm not even sure the script works in the first place. Is this an okay approach? (Maybe not ideal as I'm probably going to have to backtrack and delete some of the returns that I currently have, but for peace of mind?) Last edited by zem52887; May 25th, 2006 at 10:07 AM.

May 25th, 2006, 10:20 AM	#184
zem52887 Hobbyist Programmer Join Date: May 2006 Posts: 127 Rep Power: 3	okay well I tried running the script and not surprisingly, it didn't work. (Toggle Plain Text) def get_company_data(company_urls): soup = BeautifulSoup(urlopen(company_urls)) #Company Name title = soup.fetch("title") output = "<title>\n" #Company Profile - Table profile = soup.fetchText(re.compile("Company Profile"))[2] companyprofile = profile.findNext("table") output += "<table>" + companyprofile + "</table>\n" #Contact Information - Table contact = soup.firstText(re.compile("Contact Information")) contacttable = contact.findParent("table") output += "<table>" + contacttable + "</table>\n" #Financial Highlights - Table highlights = soup.firstText(re.compile("Highlights")) fhighlights = highlights.findParent("table") z = len(highlights) if z == 0: "N/A" else: fhighlights output += "<table>" + z + "</table>\n" #Key People key = soup.firstText(re.compile("Key People")) keypeople = key.findParent("table") output += "<table>" + keypeople + "</table>\n" return output for industry_url in get_industry_urls(industry_page): company_index = get_company_index(industry_url) for company_urls in get_company_urls(company_index): print get_company_data(company_urls) sleep(1) def get_company_data(company_urls): soup = BeautifulSoup(urlopen(company_urls)) #Company Name title = soup.fetch("title") output = "<title>\n" #Company Profile - Table profile = soup.fetchText(re.compile("Company Profile"))[2] companyprofile = profile.findNext("table") output += "<table>" + companyprofile + "</table>\n" #Contact Information - Table contact = soup.firstText(re.compile("Contact Information")) contacttable = contact.findParent("table") output += "<table>" + contacttable + "</table>\n" #Financial Highlights - Table highlights = soup.firstText(re.compile("Highlights")) fhighlights = highlights.findParent("table") z = len(highlights) if z == 0: "N/A" else: fhighlights output += "<table>" + z + "</table>\n" #Key People key = soup.firstText(re.compile("Key People")) keypeople = key.findParent("table") output += "<table>" + keypeople + "</table>\n" return output for industry_url in get_industry_urls(industry_page): company_index = get_company_index(industry_url) for company_urls in get_company_urls(company_index): print get_company_data(company_urls) sleep(1) Since I tested each of the other functions (that being, get_industry_urls, get_company_index, and get_company_urls) I can isolate the problem as being either the get_company_data function, or the for-loop. I personally think I have errors in both, but I'm not sure where. If I had to guess, I think that my if-statement isn't formatted properly. My output variables are also probably a little mucked up. On second thought, the only thing that's probably correct in the get_company_data function are the locations of the tables. Other than that, I could be very far off.

May 25th, 2006, 10:31 AM	#186
zem52887 Hobbyist Programmer Join Date: May 2006 Posts: 127 Rep Power: 3	oh indeed. I was confused on the output line but now I understand. The following should work, no? (Toggle Plain Text) def get_company_data(company_urls): soup = BeautifulSoup(urlopen(company_urls)) #Company Name title = soup.fetch("title") output = title #Company Profile - Table profile = soup.fetchText(re.compile("Company Profile"))[2] companyprofile = profile.findNext("table") output += companyprofile #Contact Information - Table contact = soup.firstText(re.compile("Contact Information")) contacttable = contact.findParent("table") output += contacttable #Financial Highlights - Table highlights = soup.firstText(re.compile("Highlights")) fhighlights = highlights.findParent("table") z = len(highlights) if z == 0: "N/A" else: fhighlights output += z #Key People key = soup.firstText(re.compile("Key People")) keypeople = key.findParent("table") output += keypeople return output for industry_url in get_industry_urls(industry_page): company_index = get_company_index(industry_url) for company_urls in get_company_urls(company_index): print get_company_data(company_urls) sleep(1) def get_company_data(company_urls): soup = BeautifulSoup(urlopen(company_urls)) #Company Name title = soup.fetch("title") output = title #Company Profile - Table profile = soup.fetchText(re.compile("Company Profile"))[2] companyprofile = profile.findNext("table") output += companyprofile #Contact Information - Table contact = soup.firstText(re.compile("Contact Information")) contacttable = contact.findParent("table") output += contacttable #Financial Highlights - Table highlights = soup.firstText(re.compile("Highlights")) fhighlights = highlights.findParent("table") z = len(highlights) if z == 0: "N/A" else: fhighlights output += z #Key People key = soup.firstText(re.compile("Key People")) keypeople = key.findParent("table") output += keypeople return output for industry_url in get_industry_urls(industry_page): company_index = get_company_index(industry_url) for company_urls in get_company_urls(company_index): print get_company_data(company_urls) sleep(1) But when I run the script it tells me that: profile = soup.fetchText(re.compile("Company Profile"))[2] IndexError: list index out of range ... but when I test this part of the function on an individual company, it works fine.

May 25th, 2006, 11:42 AM	#189
Arevos Programming Guru Join Date: Aug 2005 Location: England Posts: 1,499 Rep Power: 5	You can just put in an extra print line: (Toggle Plain Text) for industry_url in get_industry_urls(industry_page): company_index = get_company_index(industry_url) for company_urls in get_company_urls(company_index): print company_urls print get_company_data(company_urls) sleep(1) for industry_url in get_industry_urls(industry_page): company_index = get_company_index(industry_url) for company_urls in get_company_urls(company_index): print company_urls print get_company_data(company_urls) sleep(1) I also suggest you look into try-except blocks. It'll be in the tutorial under error handling. Essentially, you can "catch" errors and have code to handle them in some way: (Toggle Plain Text) import sys # ... for company_urls in get_company_urls(company_index): try: print get_company_data(company_urls) except IndexError: # Print out offending URL and then exit program print "IndexError on URL:", company_urls sys.exit(1) sleep(1) import sys # ... for company_urls in get_company_urls(company_index): try: print get_company_data(company_urls) except IndexError: # Print out offending URL and then exit program print "IndexError on URL:", company_urls sys.exit(1) sleep(1)

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

May 25th, 2006, 10:10 AM	#183
zem52887 Hobbyist Programmer Join Date: May 2006 Posts: 127 Rep Power: 3	I want to make sure that all each function is working in conjuction with each other one, though (including the loop at the end and everything). I'll increase the sleep to 20 so it does it for one company and I'll manually quit afterwards.

May 25th, 2006, 10:28 AM	#185
Arevos Programming Guru Join Date: Aug 2005 Location: England Posts: 1,499 Rep Power: 5	Well, I'm not quite sure why you're using "<table>" tags. Isn't it more logical to embed each table in a single "<td>" cell, and have one company per row?

May 25th, 2006, 11:53 AM	#190
zem52887 Hobbyist Programmer Join Date: May 2006 Posts: 127 Rep Power: 3	Thanks Arevos that's interesting stuff. I see where the program is making errors. From the main link: "http://biz.yahoo.com/ic/ind_index.html" you can sort by either "public or private" and instead of going to the company link, it's attempting to fetch.companyprofile information from that page as opposed to the company url. When I asked it to print the company_urls it displayed: http://biz.yahoo.com/ic/112_cl_pub.html which is the list of the public agriculture companies. From there, it attempted to apply the get_company_data function. At least, that's the what I think happened. Now how do we go about troubleshooting this issue?

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Programming Forums > Script Programming > Python

Never Programmed Before -- Have Task that makes me want to learn or ... die.