Programming Forums - View Single Post - How To Download Webpages Via C++ (web Spider)

Tuskony · Oct 10th, 2004, 3:58 AM

Hey,

I'd like to create a simple C++ spider that will retrieve web pages from a certain list of news sites, I will then take these pages and parse them for content. This content will then be sent to a database, where upon it can be retrieved by a simple PHP script.

Basically I love reading the news and I read about 10 different sites, so I want to create my own little "news portal" so I don't have to run to 10 different sites... It'll all be displayed to me, and updated every hour. It's for personal use only so don't worry about copyright issues, and I want to see if I can actually make this work as a nice little challenge.

I know how to do everything but actually download web pages with C/C++. And yes, I have searched the internet quite a few times but I haven't really found anything too helpful.

Any help would be greatly appreciated.

Thanks

Oct 10th, 2004, 3:58 AM	#1
Tuskony Newbie Join Date: Oct 2004 Posts: 3 Rep Power: 0	Hey, I'd like to create a simple C++ spider that will retrieve web pages from a certain list of news sites, I will then take these pages and parse them for content. This content will then be sent to a database, where upon it can be retrieved by a simple PHP script. Basically I love reading the news and I read about 10 different sites, so I want to create my own little "news portal" so I don't have to run to 10 different sites... It'll all be displayed to me, and updated every hour. It's for personal use only so don't worry about copyright issues, and I want to see if I can actually make this work as a nice little challenge. I know how to do everything but actually download web pages with C/C++. And yes, I have searched the internet quite a few times but I haven't really found anything too helpful. Any help would be greatly appreciated. Thanks