[Rabbit-dev] jpeg conversions not working

Discussion:

Luis Soltero

2011-08-30 21:35:26 UTC

Hello All,

Here is the situation..

I have two identically configured proxy servers. one running on a opensuse machine and one on a centos 5 machine. the
proxy binaries and the config files are identical.

yet.. on one jpgs get converted and on the other they do not... other image formats convert just fine. Its an issue
only with jpgs.

here is a sample URL that gets converted on a SUSE machine.
96.38.22.218 - test 30/Aug/2011:21:27:16 GMT "GET
Loading Image...

HTTP/1.1" 200 7953 [0, 86]
imageratio:7953/77910=0.102

here is the same URL on the centos machine
96.38.22.218 - test 30/Aug/2011:22:27:22 GMT "GET
http://i2.cdn.turner.com/cnn/2011/images/08/30/t1larg.chinook.memorial.jpg HTTP/1.1" 200 77910 [0, 3]

i have flushed both the proxy and the browser caches before doing the test. I have repeated the test over and over and
get the same result.

here is proof that the two conf files are the same

spider:/usr/local/rabbit4.11-beta-3119/conf # sum rabbit.conf
21308 21

[root at fly /usr/local/RabbIT4-3119/conf]# sum rabbit.conf
21308 21

The suse machine is running the following version of Java

spider:/usr/local/rabbit4.11-beta-3119/conf # rpm -qa | grep java
java-1_6_0-sun-devel-1.6.0.u23-0.2.1.i586
java-1_6_0-sun-1.6.0.u23-0.2.1.i586
java-1_6_0-sun-jdbc-1.6.0.u23-0.2.1.i586

the cent os machine is running
[root at fly /usr/local/RabbIT4-3119/conf]# rpm -qa | grep java
java-1.6.0-openjdk-1.6.0.0-1.16.b17.el5
java-1.6.0-openjdk-devel-1.6.0.0-1.16.b17.el5

When fetching the above url manually and then running it through convert or gm using the options in the config file the
image converts just fine. So... i know its not an issue with the converter.

Anyone have any ideas why jpg conversion are failing? Anyone see this before?

--luis
--
Luis Soltero, Ph.D., MCS
Director of Software Development, CTO
Global Marine Networks, LLC
StarPilot, LLC
Tel: 865-379-8723
Fax: 865-681-5017
E-Mail: lsoltero at globalmarinenet.net
Web: http://www.globalmarinenet.net
Web: http://www.starpilotllc.com

Robert Olofsson

2011-08-31 04:48:45 UTC

Permalink

Hi!

On Tue, 30 Aug 2011 17:35:26 -0400

Post by Luis Soltero
yet.. on one jpgs get converted and on the other they do not... other image formats convert just fine. Its an issue
only with jpgs.

Is convert or gm placed in the same place on both machines?
Also check the error_log, it might contain something like
"convert - /usr/bin/convert - not found...".

Some other ideas, that might be worth checking:
Are you using the config file you think you are using?
Is the cache directory writable, rabbit will only do image compression
if it can use the cache.

/robo

Luis Soltero

2011-08-31 14:25:24 UTC

Permalink

both convert and gm are in the same place. What is really strange is that conversion of jpgs does happen but not
always. Other websites seem to be ok. amazon for example is very nicely compressed. cnn.com is the one that seems to
be most problematic.

i have manually done a get on the images and done the convert and gm manually and the image gets resampled correctly.

I don't think the problem is with the execution of the conversion. Its seems like the proxy is failing to detect that
the image needs to be converted.

I am positive I am using the config I think I am using. The log files and the cache get created in specific locations.
Also the proxy runs in a specific port. Modifying the config affects these settings so I know that the config is being
read correctly.

Its a very very strange behavior I am observing.

There are no errors in the error log...

take this image for example...
on my suse machine
96.38.22.218 - test 31/Aug/2011:14:09:49 GMT "GET
Loading Image...

HTTP/1.1" 200
3333 [0, 2091] imageratio:3333/31833=0.105

the conversion happens without issue.

on the centos the same fetch after clearing all the caches including the browser cache I get this.
96.38.22.218 - test 31/Aug/2011:15:15:43 GMT "GET
http://i2.cdn.turner.com/cnn/2011/WORLD/africa/08/30/libya.female.fighter/c1main.libya.teen.rebels.cnn.jpg HTTP/1.1" 200
31833 [0, 2]

its as if the proxy is just ignoring that this is a jpg of a size larger than 1000 bytes.

yet... if I go to the following link it works perfectly fine for both machines...

centos
96.38.22.218 - test 31/Aug/2011:15:18:20 GMT "GET Loading Image...

HTTP/1.1" 200 2123 [0, 1] imageratio:2123/15030=0.141

suse
96.38.22.218 - test 31/Aug/2011:14:18:23 GMT "GET http://ecx.images-amazon.com/images/I/41yoP5hk0ML._SL500_AA300_.jpg
HTTP/1.1" 200 2126 [0, 2109] imageratio:2126/15030=0.141

So... why would rabbit distinguish between the two links above? why would it convert one of the jpgs and not the other?

I just installed sun java on the centos machine and still having the same issue.

Clueless here... any hint on what rabbit uses to determine when the image converter should/should not be executed would
greatly be appreciated.

--luis

Post by Robert Olofsson
Hi!
On Tue, 30 Aug 2011 17:35:26 -0400

Post by Luis Soltero
yet.. on one jpgs get converted and on the other they do not... other image formats convert just fine. Its an issue
only with jpgs.

Is convert or gm placed in the same place on both machines?
Also check the error_log, it might contain something like
"convert - /usr/bin/convert - not found...".
Are you using the config file you think you are using?
Is the cache directory writable, rabbit will only do image compression
if it can use the cache.
/robo

--
Luis Soltero, Ph.D., MCS
Director of Software Development, CTO
Global Marine Networks, LLC
StarPilot, LLC
Tel: 865-379-8723
Fax: 865-681-5017
E-Mail: lsoltero at globalmarinenet.net
Web: http://www.globalmarinenet.net
Web: http://www.starpilotllc.com

Robert Olofsson

2011-08-31 15:59:32 UTC

Permalink

On Wed, 31 Aug 2011 10:25:24 -0400

Post by Luis Soltero
http://i2.cdn.turner.com/cnn/2011/WORLD/africa/08/30/libya.female.fighter/c1main.libya.teen.rebels.cnn.jpg

Hmmm, that resource has a few cache control headers.
Using HEAD I find:

Date: Wed, 31 Aug 2011 15:56:40 GMT
Cache-Control: max-age=3600
Expires: Wed, 31 Aug 2011 16:54:09 GMT

So if your clock is off we may have a problem and rabbit refusing
to cache it (since doing so would mean violating the expire header).

Can you check if the system clock is correct on both machines?

/robo

Luis Soltero

2011-08-31 16:02:47 UTC

Permalink

wow. that makes perfect sense...

i will check the date and setup ntpd to make sure it stays up to date.

I will let you know what I find.

--luis

Post by Robert Olofsson
On Wed, 31 Aug 2011 10:25:24 -0400

Post by Luis Soltero
http://i2.cdn.turner.com/cnn/2011/WORLD/africa/08/30/libya.female.fighter/c1main.libya.teen.rebels.cnn.jpg

Hmmm, that resource has a few cache control headers.
Date: Wed, 31 Aug 2011 15:56:40 GMT
Cache-Control: max-age=3600
Expires: Wed, 31 Aug 2011 16:54:09 GMT
So if your clock is off we may have a problem and rabbit refusing
to cache it (since doing so would mean violating the expire header).
Can you check if the system clock is correct on both machines?
/robo

Luis Soltero

2011-08-31 16:13:25 UTC

Permalink

sigh... that was it... the time was fast by one hour... the system has been set to UTC and the ntpd pool update to use
machines that work.

all is well now.

I apologize for my ineptness. I hope others will be able to benefit from my stupidity.

Thanks for your help. I doubt I would have been able to find the problem without your help.

--luis

Post by Luis Soltero
wow. that makes perfect sense...
i will check the date and setup ntpd to make sure it stays up to date.
I will let you know what I find.
--luis

Post by Robert Olofsson
On Wed, 31 Aug 2011 10:25:24 -0400

Post by Luis Soltero
http://i2.cdn.turner.com/cnn/2011/WORLD/africa/08/30/libya.female.fighter/c1main.libya.teen.rebels.cnn.jpg