Wow this is fantastic! I just have a final few questions as to whether something is possible or not etc.
1) I want the data to be contained in bordres so I added
output = "<table border = 1>"
And now the data has borders. However, it would be a little easier on the eyes if all the columns lined up in a straight line instead of some being indented farther than others etc. I was wondering if there was a way to make constant borders, and if the text is too big, then it would wrap and add another line of text rather than resizing the table.
2) Would it be possible to add in a break of some sort after each industry?
Currently it lists the companies in alphabetical order starting with agriculture then after it's done with agriculture is immediately starts listing the Aluminum companies. However, it'd be nice if there was a break in between agriculture and aluminum so my boss would be able to identify which company is in what sector etc.
3) This is extremely minor, but since I fetched the title of the company from the text "Company Profile"... the output is "Company Name Company Profile - Yahoo! Finance"... since Company Profile appears in each company name block (and yahoo finance), I was wondering if there was a way to either refine the regex where I got the title, or if I could simply figure out a way to omit "company profile and yahoo! finance" from the name of the company
4) oh and again, it would also be nice if there was a way to add a column that says public and for public companies it ouputs yes into it and for private companies it leaves it blank, I'm going to try and write something and post it up in a bit