shaunie Posted February 17, 2011 Share Posted February 17, 2011 Hi, I have some code which scrapes data from a page. However there are around 1200 product pages on the site I need to scrape, when I attempt to loop through all the pages I get a server timeout. I can only get to around 40 without timeout. Has anyone else had this problem? Quote Link to comment Share on other sites More sharing options...
BlueSkyIS Posted February 17, 2011 Share Posted February 17, 2011 I suspect it is php configuration of max_execution_time. you might get around this using set_time_limit() at the top of your code: http://php.net/manual/en/function.set-time-limit.php Quote Link to comment Share on other sites More sharing options...
shaunie Posted February 17, 2011 Author Share Posted February 17, 2011 Hi, Thanks for your reply, the problem with this is that I won't know how long i need to increase the time by as the product list could grow. Can PHP handle multi threading? Quote Link to comment Share on other sites More sharing options...
MatthewJ Posted February 17, 2011 Share Posted February 17, 2011 Even more than that... based on network speeds at the time, the time to complete could vary greatly. The site does not provide any other way to get the product data? You think maybe they don't want people to scrape their pages? Just a thought. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.