[Rabbit-dev] Request-URI Too Long

[Mindaugas Žakšauskas]

Hi,

I have seen this error/exception in the error logs more and more
frequently on some well-known sites (eBay, Facebook, possibly others).

The GET request itself which causes this is less than 800
bytes/characters. I have tried to reproduce this by following the same
URL but it returned HTTP 200 OK so I assume this only works in
combination with other headers. Unfortunately I don't have original
headers logged.

This error seems to be coming from HttpHeaderReader:144. I can see a
TODO around here, but not quite sure if this is a placeholder for a
better implementation or just a corner case which is not clear how to
handle properly.

Is there any quick workaround how to fix this? For example, I would
more than happy to have larger buffer allocated as per configuration
setting (similar to http://forum.nginx.org/read.php?11,9470,9514) but
it seems that this is hard coded to 4096 in the Rabbit code.

I would even be willing to recompile Rabbit myself using larger buffer, but:
1) would it solve the problem?
2) is there only one place where the buffer size is set, or are there more?

Thanks in advance.

m.

[Robert Olofsson]

On Mon, 3 Jan 2011 13:52:35 +0000

Post by Mindaugas Žakšauskas
I have seen this error/exception in the error logs more and more
frequently on some well-known sites (eBay, Facebook, possibly others).

Ok, that sounds odd.

Post by Mindaugas Žakšauskas
The GET request itself which causes this is less than 800
bytes/characters. I have tried to reproduce this by following the same
URL but it returned HTTP 200 OK so I assume this only works in
combination with other headers. Unfortunately I don't have original
headers logged.

I had some bug reports about it for very long cookies some time ago.
I did fix that problem so now the default buffer is 4kB, but grows to
128 kB if 4kB does not work.

Seeing network data would be great, but I guess it can be hard to get it.

Post by Mindaugas Žakšauskas
This error seems to be coming from HttpHeaderReader:144. I can see a
TODO around here, but not quite sure if this is a placeholder for a
better implementation or just a corner case which is not clear how to
handle properly.

Ops, guess I forgot to remove that TODO when I actually made the
buffer grow.

Post by Mindaugas Žakšauskas
Is there any quick workaround how to fix this? For example, I would
more than happy to have larger buffer allocated as per configuration
setting (similar to http://forum.nginx.org/read.php?11,9470,9514) but
it seems that this is hard coded to 4096 in the Rabbit code.

Currently it is hard coded, so no config option exists.

Post by Mindaugas Žakšauskas
1) would it solve the problem?

Probably not, since rabbit actually does grow the buffer (at least in
some cases) it is probably a bug in the code.

It might make the problem occur less often though, so go ahead and try.
Switching up to 8kB buffers by default should not matter much.

Post by Mindaugas Žakšauskas
2) is there only one place where the buffer size is set, or are there more?

It is set in one class only, in CachingBufferHandler.java.

/robo

[Mindaugas Žakšauskas]

On Mon, Jan 3, 2011 at 6:13 PM, Robert Olofsson

Post by Robert Olofsson
Seeing network data would be great, but I guess it can be hard to get it.

This occurs rarely enough and the overhead would be big, but if I
wanted to log complete set of headers for every request, what would be
the best way to do this?

Post by Robert Olofsson

Post by Mindaugas Žakšauskas
This error seems to be coming from HttpHeaderReader:144. I can see a
TODO around here, but not quite sure if this is a placeholder for a
better implementation or just a corner case which is not clear how to
handle properly.

Ops, guess I forgot to remove that TODO when I actually made the
buffer grow.

So you're saying that buffer should be capable to grow indefinitely?

I haven't debugged this, but is the exception not always thrown (in
HttpHeaderReader::parseBuffer) if isUsingSmallBuffer(buffer) returns
false?

Post by Robert Olofsson

Post by Mindaugas Žakšauskas
1) would it solve the problem?

Probably not, since rabbit actually does grow the buffer (at least in
some cases) it is probably a bug in the code.

What I could do is to add some debug code just before the exception is
thrown so this would allow me to avoid logging all headers.
I am just not sure if all necessary information is available at this
point. Do you have any tips on what debugging code could be placed
there?

Post by Robert Olofsson
It might make the problem occur less often though, so go ahead and try.
Switching up to 8kB buffers by default should not matter much.

I'd rather leave it as it is and try to get down to the root of the problem.

Post by Robert Olofsson

Post by Mindaugas Žakšauskas
2) is there only one place where the buffer size is set, or are there more?

It is set in one class only, in CachingBufferHandler.java.

I can see number 4096 being repeated at least three times in this
class (methods: isLarge, putBuffer, getBuffer). I would assume all of
these would need changing, right?

Thanks for your response Robo.

m.

[Robert Olofsson]

On Mon, 3 Jan 2011 18:38:02 +0000

Post by Mindaugas Žakšauskas

Post by Robert Olofsson
Ops, guess I forgot to remove that TODO when I actually made the
buffer grow.

So you're saying that buffer should be capable to grow indefinitely?

No, the buffer size is either small (4kB) or big (128kB), it never grows
larger than that. Note that the header can actually be a lot bigger,
but each entry in the header needs to fit in that. And if you say that
you want a http header-line to be bigger than 128kB then I say that you
are crazy :-).

Post by Mindaugas Žakšauskas
I haven't debugged this, but is the exception not always thrown (in
HttpHeaderReader::parseBuffer) if isUsingSmallBuffer(buffer) returns
false?

Yes, that is one of the conditions needed to get the error. The buffer
marker must also be set to zero.

Post by Mindaugas Žakšauskas
What I could do is to add some debug code just before the exception is
thrown so this would allow me to avoid logging all headers.
I am just not sure if all necessary information is available at this
point. Do you have any tips on what debugging code could be placed
there?

The first thing to start with is to print "headerParser.getHeader()" that
will tell what we have parsed of the current header, then we want to see
what the buffer looks like and if possible the buffer contents as well.

Something like:
releaseBuffer();
logger.warning("RequestLineTooLongException: current header: " +
headerParser.getHeader());
logger.warning("RequestLineTooLongException: current buffer: " + buffer);

If you care to write something that also logs the contents that will
be nice, but we can see what we get with this first.

It might be something simple like a (single/double)-quote typo in a cookie
or something like that.

Post by Mindaugas Žakšauskas

Post by Robert Olofsson
It might make the problem occur less often though, so go ahead and try.
Switching up to 8kB buffers by default should not matter much.

I'd rather leave it as it is and try to get down to the root of the problem.

Me too.

Post by Mindaugas Žakšauskas
I can see number 4096 being repeated at least three times in this
class (methods: isLarge, putBuffer, getBuffer). I would assume all of
these would need changing, right?

Correct and now I'll change that class to use constants instead.

/robo

[Mindaugas Žakšauskas]

OK, I got some results after putting logging statements and
recompiling Rabbit (btw I'm using v4.7):

8<------------------------------------------------------------------------
06-Jan-2011 14:10:51 rabbit.httpio.HttpHeaderReader parseBuffer
WARNING: RequestLineTooLongException: current header: GET
http://www.facebook.com/profile.php?id=XXXXXXXXX&ref=ts&__a=26&ajaxpipe=1&quickling[version]=330157

Accept: application/x-ms-application, image/jpeg,
application/xaml+xml, image/gif, image/pjpeg, application/x-ms-xbap,
*/*

Referer: http://www.facebook.com/

Accept-Language: en-GB

User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64;
Trident/4.0; GTB6.6; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729;
.NET CLR 3.0.30729;

Accept-Encoding: gzip, deflate

Host: www.facebook.com

Proxy-Connection: Keep-Alive

WARNING: RequestLineTooLongException: current buffer:
java.nio.DirectByteBuffer[pos=0 lim=3773 cap=131072]
8<------------------------------------------------------------------------

or another:

8<------------------------------------------------------------------------
WARNING: RequestLineTooLongException: current header: GET
http://ad.yieldmanager.com/pixel?id=337399&t=2 HTTP/1.1

Host: ad.yieldmanager.com

User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_8; en-us)
AppleWebKit/533.19.4 (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4

Referer: http://uk.yahoo.com/

Accept: */*

Accept-Language: en-us

Accept-Encoding: gzip, deflate


06-Jan-2011 15:20:58 rabbit.httpio.HttpHeaderReader parseBuffer
WARNING: RequestLineTooLongException: current buffer:
java.nio.DirectByteBuffer[pos=0 lim=3991 cap=131072]
8<------------------------------------------------------------------------


It seems that if this exception happens, there are few others soon to
follow. Can this be related to keep-alive?

Is there anything else I can help identifying this?

m.

[Robert Olofsson]

On Thu, 6 Jan 2011 21:33:08 +0000

Post by Mindaugas Žakšauskas
OK, I got some results after putting logging statements and

Ok, Thanks for that information.

Sadly that information did not give us much useful information.
We can see:
1) That the header starts normally and that looks file
2) That the buffer is actually a big buffer
3) That the data is not very big (see the lim=xxx)
4) It is request headers so it is very likely that it is a cookie that
is causing this problem.

Can you inspect the cookies in your web browser and see if it is a long
cookie? and also if it contains some odd characters (like ' " \n \r ...)

Ok, I guess the next step is to try to dump the whole buffer content and
see what it actually contains.

Can you add a few more lines after the current debug output:

byte[] data = new byte[buffer.remaining ()];
buffer.get (data);
String s = new String (data, "ISO-8859-1");
logger.warning("buffer content: " + s);

Note: http headers should be ASCII, but ISO-8859-1 is a bit safer if the
cookie(?) actually contains any odd characters.

/robo

[Mindaugas Žakšauskas]

Post by Robert Olofsson
Can you inspect the cookies in your web browser and see if it is a long
cookie? and also if it contains some odd characters (like ' " \n \r ...)

I could if this was my browser, but it isn't.
I know Servlet API allows to get cookie information, but I'm not sure
if this can be done from within Rabbit.

Post by Robert Olofsson
Ok, I guess the next step is to try to dump the whole buffer content and
see what it actually contains.

Will do (and report back when after getting any results). Thanks.

m.