zem52887

yeah I figured that it had something to do with the server moreso than the code, because it's running well again... I can't explain how much I appreciate your help

out of curiousity? DDOSing?

Arevos

That's Yahoo! closing your connection down because you're going too fast - see my post about the sleep function

Ah, that's a problem with my code. As well as the industry links, it's also found the "Alphabetical" link!
The above bit of code has an if-statement embedded in the list comprehension. Load up the Python tutorial to learn more about if-statements.

Arevos

Websites only have so much bandwidth. If lots of people downloaded ten pages every second, then Yahoo! would become laggy and unresponsive, so Yahoo!'s servers block people who are downloading too much, too fast.

It's also common courtesy not to stress other people's servers. Put in a sleep(1) in the loop and you'll be being a lot more polite.

Malicious computer users deliberately try to take too much bandwidth from other servers. This can cause servers to go down under the strain, which means the company will lose business. This is called a Denial of Service attack, or a DOS attack. When a denial of service attack comes from many different computers, it's called a distributed denial of service attack, or a DDOS attack.

Also, look up try/except blocks in the Python tutorial. That information will come in handy later.

zem52887

okay, I have to look that up but for now I'm going to cheat and just use it. I'll take a look at it a little later tonight, but since you're 6 hours ahead (UK right?), I want to get my final question in for the night...

now I need a get_company_urls function, from there I need a get_company_data and I'm basically done correct?

okay I'm an intern and I'm the last one left in the office, I think it's time to depart. Thanks for the help Arevos. Have a good weekend.

Arevos

Yep, that's about it. Remember to use sleep so that you don't stress the server.

Also, some error handling will probably be necessary. Ideally, you don't want your program to end the moment your connection times out once. Fortunately, error handling is easier than it sounds

You too!

The Dark

Great job Arevos! This is some excellent tutoring.
Excellent work zem52887, you learn quick and are actually reading every reply! I hope you get a job out of this (or at least avoid carpal tunnel).

There should be a hall of fame for threads like these.

Dameon

Is it just me, or does ripping off data from Yahoo Finance not seem like how financial institutions are supposed to operate?

__________________
MD5(sig) = bcef75433db02e9ad9bf81d6f7c5c270

DaWei

Absolutely. I was going to call attention to this, as well as the assembler thread involving Narue, as prime examples of the true worth of a forum. Two noobs coupled with two top-notch pros; anyone not following them has missed a true pleasure.

__________________
Abstraction doesn't make it impossible to write bad code; it makes it possible to write superior code.
Contributor's Corner: Grumpy on C++ Exceptions DaWei on Pointers

Ooble

I've been very impressed myself. It's nice to see people who take in information, are polite, and try stuff themselves before asking questions. It's a reminder of why I'm here.

__________________
Me :: You :: Them

Arevos

Well, I seem to recall that that databases that contain only facts aren't protected by copyright law, which is sensible enough. Some large sports-related organisations have managed to subvert this, but in general you can't copyright facts, which is what Yahoo! appears to offer.

Capitalism being what it is, companies will always seek the easiest option that's not outright illegal, so wanting to gather public-domain financial information from a website doesn't come as a surprise to me.