|
|||||||
 |
|
|
LinkBack | Thread Tools | Search this Thread | Rate Thread | Display Modes |
02-03-2006, 09:55 PM
|
#1 |
|
Member (7 bit)
Join Date: Dec 2005
Posts: 87
|
Spider in PHP
Has anyone ever coded a spider before? I made a simple one in Java but I need something to handle more volume. I was reading on PHP.net where cUrl can process 2,000 urls a minute. I thought that would be perfect but I am getting only one url a second. I tested the time by seeing how long a page loads in a browser, and by putting a 10 different urls in my script and seeing how long it takes.
It looks like the programmer is running 50 different instances of the script at the same time. I am going to try this out and see what happens. I just wanted to know if anyone had similar experience. |
|
|
|
02-23-2006, 01:11 AM
|
#2 |
|
Member (7 bit)
Join Date: Dec 2005
Posts: 87
|
For everyone's information cUrl works great. I changed my loop around a little bit and it is really fast. I also added a list of anonymous proxy servers which it processes to prevent getting booted from sites. It works very well.
|
|
|
|
|
| Bookmarks |
| Thread Tools | Search this Thread |
| Display Modes | Rate This Thread |
|
|
Linear Mode
