Execution of a PHP code takes too long

[marz]

I have a very weird problem on my PHP5.3/Apache/MySQL based system on WinXp Pro.

 

I have written php script which basically scans every directory in my local hard disc, checks which files are images, sets a counter of the number of images found per folder and saves this count in a MySQL table.  To put you in the picture, we are talking about 1000 folders with about 8-10pics each on average 9000 files.

 

On the previous PHP 5.1 / IIS system, (same PC) this used to work moderately well (took a nice 6 mins) and I had to include the command set_time_limit of about 3seconds in the looping to avoid execution timeout failures. (also worked by setting a high value of 10mins before the loopping)

 

 

ANyway, now I run the exact same script on the new Php5.3 and have the problem that the code starts operating quite fast and gradually it slows and slows until eventually it is dead slow and have to quit internet explorer. For exampe, the first 3000 files are scanned in just 60seconds, then the next 1000 in the next 4 mins, and eventually drops to about 100files or less files per minute. I also noticed that the CPU consumption of iexplorer.exe is 90-99%, and the window of Internet explorer is very slugish with a temp '(not responding)' messages on the top window bar.

 

WHat can I do and test? Is it the code itself, the PHP5.3 or something in php.ini or apache conf. file? Buffers? memory? WHat? gonna crazy!

 

 

basically, the code scans a dir (location of dir taken from Mysql) puts file contents in an array, analyse the array for .jpg/.gif (and other parameters specific to pics, counts the valid pic files in a variable, puts  this in a MySql databse, loops for next folder.

 

The code is a kind of tool to have a count of my pics in my folders and never used remote server/online but only local.

 

Thanks for your time

 

 

 

 

 

 

 

 

[the182guy]

I would call the sleep(1) function inside the looping after every couple of hundred files or so. That will halt execution for a second (or however many) and it will release the CPU which should stop the 'not responding' stuff. Play around with it.

[PFMaBiSmAd]

A picture is worth a 1000 words.

 

Posting your code would be the quickest way of getting an answer because it would show how you are querying the database, how you are outputting the status information, how you are managing data/variables, if you are doing something that is php/IIS/Apache specific...

[marz]

 

 

Here is the routine - I am no PHP expert, so the coding is quite basic: :shrug]

 

 

$totalimages = 0; // to store the number of images found in folder

mysql_conn(); // Custom function to connect with MySQL
$db_query = "SELECT FAML, ID, species FROM sppindex order by species"; 	
$db_result = mysql_query($db_query)  or die ('Error in query: ' . $db_query .  mysql_error());


while ($db_data = mysql_fetch_array ($db_result)) 
{
$ID 	= trim($db_data['ID']); // get ID code from db
$sp		= trim($db_data['species']); // get species name from db
$faml	= trim($db_data['FAML']); // get family name from db
$picdir = "../".$faml.'/Pics/'.$ID.'/';  // build path in local hd
$spname = str_replace(" ", "_", $sp); // replace spaces with _ in species name
$spname = str_replace(".", "", $spname);
unset ($picfiles); // array for storing filenames from the working directory


print '<u> Working image dir: <font color="blue">'. $picdir . '</font>    Species: <font color="blue">' . $sp . '</font>		
    ID:  <font color="blue">' .$ID . '</font></u><br><br> <font size="2">';


$dirhandler  = opendir($picdir);
while (false !== ($filename = readdir($dirhandler))) {   $picfiles[] = $filename; }

sort($picfiles);  // all files in Pics/ directory are in array $picfiles
//	print_r ($picfiles) ; print "<br>";
	$piccount=0; // counter for number of valid jpg/gif files found in directory

foreach ($picfiles as $filename) // examine every array value, that is every filename in Pics/

{
	$filename = basename($filename) ;

	if (is_dir($picdir . $filename)==true)  // skip if a dir is found
		{print "<font color=blue><b>" . $filename . "</font></b> <font color = red> - is a Dir (skipped...) </font><br>";} 
	else 								// if filename, procede
		{
		$fsize = filesize($picdir . $filename); // get filesize
		$img_array = getimagesize ($picdir . $filename); // get image info
//			print_r($img_array); print "<br><br>";
		$filetype = (substr($img_array['mime'], 0, 5)) ; 
		$imgtype = (substr($img_array['mime'], 6)) ; 

		print "Analysing file : <font color=blue><b>" . $filename . "</font></b>";  // start series of tests for finding valid images


		if (filetype($picdir . $filename) !== "file")  // REF A
			{print "<font color = red> - this is not a file (skipped...) </font><br>";} 
 		elseif	( (strpos(strtolower($filename), ".jpg")==false) AND 
			  (strpos(strtolower($filename), ".jpeg")==false)  AND
			  (strpos(strtolower($filename), ".gif")==false)  ) 
			{print "<font color = red> - extension not jpg/gif/bmp/png (skipped...) </font><br>";}
		elseif  ( $filetype != "image")
					{print "<font color = red> - File not really an image (skipped...) </font><br>";}
		elseif	( $fsize  < 8000) 
					{print "<font color = red> - Filesize only [" . $fsize ."bytes] Does not qualify for a valid image (skipped...) </font><br>";}  
		elseif (  strpos(strtolower($filename), "_t.") !=0 OR strpos(strtolower($filename), "_t2.") !=0 OR 
				  strpos(strtolower($filename), "_t3.") !=0 OR strpos(strtolower($filename), "_t4.") !=0  )
					{print "<font color = red> - This is a thumbnail image (skipped...) </font><br>";}  


		else { $piccount++ ; 
			print "<font color=green> - " .$imgtype. " Image found (" . $img_array[0]." x ".$img_array[1].")    Image Count: ".$piccount."</font><br>";}			
		} // End of if/else for is_Dir



	} // End of Foreach loop

print '<br><font size="+1"> Total number of images for profile <font color="blue">' . $sp . "</font> is <b> " . $piccount. "<b></font> " ;

closedir();
$img_array= array();

$sql_edit = "Update sppindex Set rem3 = '$piccount' Where ID = '$ID' ";
if (mysql_query ($sql_edit))  {print "<br> Inserted in DataBase sppindex !";}
else {die ('Error in query: ' . $sql_edit .  mysql_error());}

 $totalimages = $totalimages + $piccount;
 print "<br> $totalimages "; 


print "</font><br><hr><br><br>";
set_time_limit (100); // this will extend execution time 10 second every loop, so as it will never stop due max exec time reached. (tool takes 4 mins)
} // End of While Loop


print "<br><hr><br><h2><center><font color=red>Total number of  photos in Maltawildplants.com is: " . $totalimages . " photos. </font></h2> ";


mysql_free_result($db_query); 
mysql_disc();

[marz]

182guy - thanks mate - I have not had time to try the sleep() function - I just posted this before going to work. It might work as from the symptoms observed in iexplorer, it is like there is a gradual build of hangup. :-\

 

PS - the textbox to write posts in this forum is very annoying. When you reach the bottom it moves up away from the cursor making editing almost blind.

[marz]

"A picture is worth a 1000 words."

 

That what I suppose too !

 

 

[Mchl]

Are you running this from your browser? Don't. Run it from command line.

[samshel]

http://theserverpages.com/php/manual/en/function.readdir.php

 

An Extract. Might be useful !

IT IS VITALLY IMPORTANT that when using opendir and then scanning with readdir, that you test opendir to see if it has returned a valid pointer.

 

The following code does not :

<?php

 

$dir = opendir($mydir);

while((false!==($file=readdir($dir))))

  if($file!="." and $file !="..")

      unlink($mydir.'/'.$file);

closedir($dir);

 

?>

if $mydir is not a valid directory resource (for example, the directory in question has been deleted), then the code will loop infinitely.  Of course, it'll stop eventually if you have set a max execution time, but in the meantime it will visibly slow the server.

[Psycho]

I'm sure there are other efficiencies to be had, but the one that jumps out at me is that you are running queries in a loop. Doing that is horribly inefficient. There are two possible solutions I see:

 

1. Use an INSERT statement with ON DUPLICATE KEY UPDATE instead of just an UPDATE statement. You would then gather the data for the query within the loop and after the loop completes run a single query.

 

Example:

In the loop change this line

$sql_edit = "Update sppindex Set rem3 = '$piccount' Where ID = '$ID' ";

 

To this

$values[] = "('$ID', '$piccount')";

 

Then after the loop has completed, create a single query like this

$sql_edit = "INSERT INTO sppindex
             (ID, rem3)
             VALUES " . implode(', ', $values) .
             ON DUPLICATE KEY UPDATE rem3=VALUES(rem3)";

 

2. The other option is similar. But, instead of using ON DUPLICATE KEY UPDATE you create a single UPDATE statement that will do the same thing.

 

Example:

In the loop change this line

$sql_edit = "Update sppindex Set rem3 = '$piccount' Where ID = '$ID' ";

 

To this

$cases[] = "WHEN '$ID' THEN '$piccount'";

 

Then after the loop has completed, create a single query like this

$sql_edit = "UPDATE `sppindex` SET `rem3` =
             CASE `ID` " . implode("\n", $cases) . "END";

 

I'm sure one method is more efficient than the other. You could do some testing to find out which is better.

[BlueSkyIS]

You might also consider reducing the number of writes to the browser. That might be part of the reason IE is hogging resources: just holding all that output in memory.

 

... and if IE is eating all your resources, Apache/PHP/MySQL will have to slow down to accommodate.

[marz]

I will test everything you said in a while - the sleep (1) function however didn't work. What's strange is that it used to work quite fine on the prev installation and also on the laptop. If something comes in mind related to php.ini , let me know

[kickstart]

Hi

 

I would be inclined to use INSERT / ON DUPLICATE KEY UPDATE as suggested above, but batch up the inserts rather than doing them one at a time.

 

All the best

 

Keith

[marz]

Update 1:  I have commented out the part where there is the mysql update:

 

// $sql_edit = "Update sppindex Set rem3 = '$piccount' Where ID = '$ID' ";

// if (mysql_query ($sql_edit))  {print "<br> Inserted in DataBase sppindex !";}

// else {die ('Error in query: ' . $sql_edit .  mysql_error());}

 

and nothing had improved...

 

[rwwd]

Have you tried any other browser at all, just to see if it M$ being it usual clunky self.

 

And yes, sleep(); is rather good function, just as important as empty() and isset() IMO

 

Rw

[marz]

Update 2:  I have comented out numerous print commands so as about 80% of the printing in the browser window is eliminated. I have seen improvemnet, since I reached the 4000 files in 2 minutes, but then things slowed down and eventually became dead arounf 4700 files.

 

I noticed that if I click on the window of another application (eg: Outlook express) and then back on the broser window, the script processes a chunk and freezes again.

 

 

I cant make any sense in my limited knowledge :-(

 

[marz]

I only have IE7

 

[marz]

Are you running this from your browser? Don't. Run it from command line.

 

HOW? Will the print text show up or it is meant that the script executes in a kind of silent mode?

 

 

[marz]

IT IS VITALLY IMPORTANT that when using opendir and then scanning with readdir, that you test opendir to see if it has returned a valid pointer.

 

The following code does not :

<?php

 

$dir = opendir($mydir);

while((false!==($file=readdir($dir))))

  if($file!="." and $file !="..")

      unlink($mydir.'/'.$file);

closedir($dir);

 

?>

if $mydir is not a valid directory resource (for example, the directory in question has been deleted), then the code will loop infinitely.  Of course, it'll stop eventually if you have set a max execution time, but in the meantime it will visibly slow the server.

 

 

Very true, I used to have this problem in the past and finally realised that if a dir does not exist, it will loop for infinity and hence freezes. However that gives a kind of error code in the errorlog.txt and this is not the issue becasue I have checked taht all dirs are there.

 

What else can it be????

[rwwd]

Get something else to try then, safari. opera, firefox, chrome, these should be standard browsers installed on your machine if you are developing, this then tests code to see if you are being displayed as you intended in every browser. IE is RENOWNED for being a really clunky and very slow, and not showing lots of css/div layouts properly.

 

Rw

[marz]

Here is another part of the puzzle.

 

I let the script to run with exec_time of 5 mins (300seconds) and, as expected, the script failed with the exe limit time exceeded error after about 700/1000 scanned directories. I expected that iexplorer will become back to normal then, but no, it was still buggy, slow and had to close it with taskmanager.

 

??

 

 

 

[marz]

Is there a way to flush the memory or in old basic language - "Clear the screen" of iexplorer  with php.

[roopurt18]

Clear the screen? No.  If the browser is locking up so bad that you're having to close it with task manager then I'm 95% sure the problem is too much output to the browser.

 

Do as suggested and run from a command line:

Start -> run -> cmd
cd \directory\with\php\file\in\it
php scriptname.php

 

If you want you can redirect the output to a file:

Start -> run -> cmd
cd \directory\with\php\file\in\it
php scriptname.php > scriptname.html

 

And then you can see the file size of scriptname.html.  If its multiple megabytes then too much output is your problem.

[marz]

I tried the command line but I got an error. Probably it is finding some libraries in thge old installation dir (F:\INTERNET\Coding\PHP5\). How can I change so as it searches in the actual dir that php is installed (K:/server/php). DO I have some old php.ini around?

 

 

 

K:\Documents and Settings\Stephen>K:/server/php/php

PHP Warning:  PHP Startup: gd: Unable to initialize module

Module compiled with module API=20060613

PHP    compiled with module API=20090626

These options need to match

in Unknown on line 0

PHP Warning:  PHP Startup: Unable to load dynamic library 'F:\INTERNET\Coding\PH

P5\ext\php_mysql.dll' - Invalid access to memory location.

in Unknown on line 0

PHP Warning:  PHP Startup: Unable to load dynamic library 'F:\INTERNET\Coding\PH

P5\ext\php_mysqli.dll' - The specified procedure could not be found.

in Unknown on line 0

PHP Warning:  Directive 'magic_quotes_gpc' is deprecated in PHP 5.3 and greater

in Unknown on line 0

 

Warning: Directive 'magic_quotes_gpc' is deprecated in PHP 5.3 and greater in Un

known on line 0

PHP Warning:  Directive 'magic_quotes_runtime' is deprecated in PHP 5.3 and grea

ter in Unknown on line 0

 

Warning: Directive 'magic_quotes_runtime' is deprecated in PHP 5.3 and greater i

n Unknown on line 0

^A

[marz]

I have fixed this by setting the path pointing to the correct dor from the Environment variables settings. The variable is called PHPRC.  I test the script from the command line execution and keep you posted

 

[marz]

HALF SUCCESS 

 

I managed to run the script from CMD line which took less than 2 mins. Then I did a step further,  I created a short cut link which also worked well, except one thing which I like to have functioning. I failed to create a script output file as instructed in the 2 previous posts.

 

I used the following command parse:

 

K:\server\php\php.exe g:\marz-kreations\Wildplants\!php\Image_tools.php > G:\MarZ-Kreations\WildPlants\!php\exec\Image_Tools.html

 

I created an exec dir but no html file was created. When running the command (via double clicking the link) the script ran in the black cmd window