[Rabbit-dev] Rabbit-dev Digest, Vol 18, Issue 3

[Rick Leir]

Read and write are normal read/write.
The transfer values are zero copy data transfers.

Hi Robo,
Would you explain this more please?
Thanks -- Rick

[Robert Olofsson]

Hi!

On Sat, 20 Aug 2011 07:59:35 -0400

Post by Rick Leir

Read and write are normal read/write.
The transfer values are zero copy data transfers.

Hi Robo,
Would you explain this more please?

I can try!

A normal read means that the OS kernel does a read and then
move the data to user space, the program then has to
do a write that will move the data back to kernel space and out
on the specified channel.

A zero copy transfer means that the kernel reads the data
and then writes the data directly to the output channel, without
going through user space.

This means that a zero copy transfer can be quite a lot faster and
use less resources. It also means that rabbit (or other programs)
can _not_ filter or modify the data.

Rabbit tries to use zero copy transfers for cache handling, but
falls back to normal read/write if transferTo/transferFrom calls
are not available. I would guess that most installations of rabbit
will not see much of a difference, but when I benchmarked it some
years ago there was quite a big difference (not sure about the
exact numbers, but something like 10-25% more requests served if I
remember correctly).

/robo

[Luis Soltero]

Hi Robo,

what causes urls to get truncated as follows? I notice that we don't get these when running v 4.8 and 4.9. My concern is that these truncations are an artifact of the new logging additions. We are not running v10 so it could be an issue created by mods added to this version.

I would like to put the new logging code into production but this issue has me concerned.

We are getting errors that look like the following.

Aug 19, 2011 1:13:30 AM rabbit.proxy.ProxyLogger setup
INFO: Log level set to: INFO
Aug 19, 2011 1:13:31 AM rabbit.proxy.HttpProxy setConfig
INFO: RabbIT proxy version 4.11: Configuration loaded: ready for action.
Aug 19, 2011 1:15:52 AM rabbit.proxy.Connection handleFailedRequestRead
INFO: Exception when reading request
rabbit.httpio.BadHttpHeaderException: Malformed header:F%2526ot%253DA; s_ppv=100; rvisw=1; srvisw=new%3A1; rvism=1;
srvism=new%3A1
at rabbit.httpio.HttpHeaderParser.readHeader(HttpHeaderParser.java:176)
at rabbit.httpio.HttpHeaderParser.lineRead(HttpHeaderParser.java:143)
at rabbit.httpio.LineReader.readLine(LineReader.java:43)
at rabbit.httpio.HttpHeaderParser.handleBuffer(HttpHeaderParser.java:60)
at rabbit.httpio.HttpHeaderReader.parseBuffer(HttpHeaderReader.java:124)
at rabbit.httpio.HttpHeaderReader.read(HttpHeaderReader.java:112)
at org.khelekore.rnio.impl.ChannelOpsHandler.handleRead(ChannelOpsHandler.java:105)
at org.khelekore.rnio.impl.ChannelOpsHandler.handle(ChannelOpsHandler.java:160)
at org.khelekore.rnio.impl.SingleSelectorRunner.handleSelects(SingleSelectorRunner.java:350)
at org.khelekore.rnio.impl.SingleSelectorRunner.run(SingleSelectorRunner.java:204)
at java.lang.Thread.run(Thread.java:662)
Aug 19, 2011 1:16:53 AM rabbit.filter.ProxyAuth getError
WARNING: Bad url: HTTP/1.1
java.net.MalformedURLException: no protocol: HTTP/1.1
at java.net.URL.<init>(URL.java:567)
at java.net.URL.<init>(URL.java:464)
at java.net.URL.<init>(URL.java:413)
at rabbit.filter.ProxyAuth.getError(ProxyAuth.java:96)
at rabbit.filter.ProxyAuth.doHttpInFiltering(ProxyAuth.java:53)



Following are urls from the log access log file that have been truncated. Note that the "user" field is blank. This suggest that its more than the url that is getting corrupted.

spider:/usr/local/RabbitDev/rabbit/logs # grep -v test access_log.0 | grep -v CONNECT
96.38.22.218 - - 19/Aug/2011:06:16:53 GMT "Fwww.cnn.com%2F2011%2FUS%2F08%2F18%2Fflashmobs.police%2Findex.html&ot=A&AQE=1
HTTP/1.1" 400 357 [0, 120]
96.38.22.218 - - 19/Aug/2011:06:18:58 GMT
"start%3A%20when%20flash%20mobs%20attack&pid=cnn%3Ac%3A%2F2011%2Fus%2F08%2F18%2Fflashmobs.police%2F&pidt=1&oid=%2Fvideo%2F%3F%2Fvideo%2Fus%2F2011%2F08%2F16%2Fjk.flashmobs.maryland.cnn&oidt=1&ot=A&oi=1&AQE=1
HTTP/1.1" 400 357 [0, 127]
96.38.22.218 - - 19/Aug/2011:06:20:34 GMT
"%20Plug-In%202%20for%20NPAPI%20Browsers%3B&pid=cnn%3Asf%3A%2Fworld%2F&pidt=1&oid=http%3A%2F%2Fwww.cnn.com%2F2011%2FWORLD%2Fasiapcf%2F08%2F18%2Fafghanistan.bombing%2Findex.html%3Fhpt%3Dwo_c2&ot=A&AQE=1
HTTP/1.1" 400 357 [0, 163]
96.38.22.218 - - 19/Aug/2011:06:21:01 GMT
"%207.6.6%3BSilverlight%20Plug-In%3BJava%20Plug-In%202%20for%20NPAPI%20Browsers%3B&pid=cnn%3Asf%3A%2Fjustice%2F&pidt=1&oid=http%3A%2F%2Fwww.cnn.com%2F2011%2FWORLD%2Famericas%2F08%2F18%2Faruba.missing.woman%2Findex.html%3Fhpt%3Dju_c2&ot=A&AQE=1
HTTP/1.1" 400 357 [0, 228]
96.38.22.218 - - 19/Aug/2011:06:25:02 GMT
"3A%2Fjustice%2F&pidt=1&oid=http%3A%2F%2Fwww.cnn.com%2F2011%2FCRIME%2F08%2F18%2Fflorida.jose.baez%2Findex.html%3Fhpt%3Dju_c2&ot=A&AQE=1
HTTP/1.1" 400 357 [0, 263]
96.38.22.218 - - 19/Aug/2011:06:26:56 GMT
".usatoday.com%2Fmoney%2Findustries%2Ffood%2Fstory%2F2011%2F08%2FBurger-King-freshens-fast-food-image-kicks-King-to-the-curb%2F50046768%2F1&c25=usat&c28=2&v29=2%3A00AM&c33=undefined&c34=undefined&c38=false&c50=usatoday&c51=39&c52=usat%20%3A%2F&s=1920x1200&c=24&j=1.7&v=Y&k=Y&bw=1447&bh=1019&p=Picasa%3BWebEx64%20General%20Plugin%20Container%3BMicrosoft%20Office%20Live%20Plug-in%3BShockwave%20Flash%3BFlip4Mac%20Windows%20Media%20Plugin%202.3.8%20%3BGoogle%20Earth%20Plug-in%3BQuickTime%20Plug-in%207.6.6%3BSilverlight%20Plug-In%3BJava%20Plug-In%202%20for%20NPAPI%20Browsers%3B&pid=usat%20%3A%2F&pidt=1&oid=usat_lede_well_b%7Clede-right-3pack-thumbs%7Cheadline%7C1&oidt=1&ot=A&oi=1&AQE=1
HTTP/1.1" 400 357 [0, 316]
96.38.22.218 - - 19/Aug/2011:06:29:02 GMT
"1.7&v=Y&k=Y&bw=1447&bh=1019&p=Picasa%3BWebEx64%20General%20Plugin%20Container%3BMicrosoft%20Office%20Live%20Plug-in%3BShockwave%20Flash%3BFlip4Mac%20Windows%20Media%20Plugin%202.3.8%20%3BGoogle%20Earth%20Plug-in%3BQuickTime%20Plug-in%207.6.6%3BSilverlight%20Plug-In%3BJava%20Plug-In%202%20for%20NPAPI%20Browsers%3B&pid=usat%20%3A%2F&pidt=1&oid=usat_lede_well_b%7Clede-right-3pack-thumbs%7Cheadline%7C2&oidt=1&ot=A&oi=1&AQE=1
HTTP/1.1" 400 357 [0, 415]
96.38.22.218 - - 19/Aug/2011:06:29:29 GMT
"lunge-on-fears-about-Europes-impact-on-US-banks%2F50044450%2F1&v21=Signed%20Out&c23=www.usatoday.com%2Fmoney%2Findustries%2Fbanking%2Fstory%2F2011%2F08%2FMarkets-plunge-on-fears-about-Europes-impact-on-US-banks%2F50044450%2F1&c25=usat&c28=2&v29=2%3A00AM&c33=undefined&c34=undefined&c38=false&c50=usatoday&c51=22&c52=usat%20%3A%2Fmoney%2Feconomy%2Fstory%2F2011%2F08%2FFear-rises-on-jobs-housing-reports%2F50047110%2F1&s=1920x1200&c=24&j=1.7&v=Y&k=Y&bw=1447&bh=1019&p=Picasa%3BWebEx64%20General%20Plugin%20Container%3BMicrosoft%20Office%20Live%20Plug-in%3BShockwave%20Flash%3BFlip4Mac%20Windows%20Media%20Plugin%202.3.8%20%3BGoogle%20Earth%20Plug-in%3BQuickTime%20Plug-in%207.6.6%3BSilverlight%20Plug-In%3BJava%20Plug-In%202%20for%20NPAPI%20Browsers%3B&pid=usat%20%3A%2Fmoney%2Feconomy%2Fstory%2F2011%2F08%2FFear-rises-on-jobs-housing-reports%2F50047110%2F1&pidt=1&oid=http%3A%2F%2Fwww.usatoday.com%2Fmoney%2Findustries%2Fbanking%2Fstory%2F2011%2F08%2FMarkets-plunge-on-fears-about-Euro_1&oidt=1&ot=A&oi=1&AQE=1
HTTP/1.1" 400 357 [0, 435]
96.38.22.218 - - 19/Aug/2011:06:37:42 GMT
"5=D%3Dc42&s=1920x1200&c=24&j=1.7&v=Y&k=Y&bw=1447&bh=1019&p=Picasa%3BWebEx64%20General%20Plugin%20Container%3BMicrosoft%20Office%20Live%20Plug-in%3BShockwave%20Flash%3BFlip4Mac%20Windows%20Media%20Plugin%202.3.8%20%3BGoogle%20Earth%20Plug-in%3BQuickTime%20Plug-in%207.6.6%3BSilverlight%20Plug-In%3BJava%20Plug-In%202%20for%20NPAPI%20Browsers%3B&AQE=1
HTTP/1.1" 400 357 [0, 899]
96.38.22.218 - - 19/Aug/2011:06:37:46 GMT
"c41=galleries&c42=2011&v44=D%3Dc41&v45=D%3Dc42&s=1920x1200&c=24&j=1.7&v=Y&k=Y&bw=1447&bh=1019&p=Picasa%3BWebEx64%20General%20Plugin%20Container%3BMicrosoft%20Office%20Live%20Plug-in%3BShockwave%20Flash%3BFlip4Mac%20Windows%20Media%20Plugin%202.3.8%20%3BGoogle%20Earth%20Plug-in%3BQuickTime%20Plug-in%207.6.6%3BSilverlight%20Plug-In%3BJava%20Plug-In%202%20for%20NPAPI%20Browsers%3B&pid=mny%3Ao%3Amoney%3A%2Fgalleries%2F2011%2Fmoneymag%2F1108%2Fgallery.best_places_job_growth.moneymag%2F&pidt=1&oid=http%3A%2F%2Fmoney.cnn.com%2Fgalleries%2F2011%2Fmoneymag%2F1108%2Fgallery.best_places_job_growth.moneymag%2F2.html&ot=A&AQE=1
HTTP/1.1" 400 357 [0, 922]
96.38.22.218 - - 19/Aug/2011:06:38:01 GMT "g%2F2.html&pidt=1&oid=http%3A%2F%2Fmoney.cnn.com%2Ftechnology%2F&ot=A&AQE=1
HTTP/1.1" 400 357 [0, 946]
96.38.22.218 - - 19/Aug/2011:06:38:16 GMT
"ey.cnn.com&c30=cnnmoney&v30=D%3Dc30&c32=adbp%3Acontent&v32=D%3Dc32&c33=adbp%3Aarticle%20read&v33=D%3Dc33&c34=anonymous&v34=D%3Dc34&c35=adbp-20110802-54%7Ccore-830%3A0&c40=2%3A30%7Cfriday%7Cweekday&v40=D%3Dc40&c41=2011&c42=08&v44=D%3Dc41&v45=D%3Dc42&s=1920x1200&c=24&j=1.7&v=Y&k=Y&bw=1447&bh=1019&p=Picasa%3BWebEx64%20General%20Plugin%20Container%3BMicrosoft%20Office%20Live%20Plug-in%3BShockwave%20Flash%3BFlip4Mac%20Windows%20Media%20Plugin%202.3.8%20%3BGoogle%20Earth%20Plug-in%3BQuickTime%20Plug-in%207.6.6%3BSilverlight%20Plug-In%3BJava%20Plug-In%202%20for%20NPAPI%20Browsers%3B&AQE=1
HTTP/1.1" 400 357 [0, 970]
96.38.22.218 - - 19/Aug/2011:06:39:45 GMT
"0%3BGoogle%20Earth%20Plug-in%3BQuickTime%20Plug-in%207.6.6%3BSilverlight%20Plug-In%3BJava%20Plug-In%202%20for%20NPAPI%20Browsers%3B&pid=mny%3Ac%3Amoney%3A%2F2011%2F08%2F18%2Ftechnology%2Fgoogle_motorola%2F&pidt=1&oid=http%3A%2F%2Fmoney.cnn.com%2F2011%2F08%2F16%2Ftechnology%2Fgoogle_motorola%2Findex.htm&ot=A&AQE=1
HTTP/1.1" 400 357 [0, 1003]
96.38.22.218 - - 19/Aug/2011:06:39:52 GMT
"&v=Y&k=Y&bw=1447&bh=1019&p=Picasa%3BWebEx64%20General%20Plugin%20Container%3BMicrosoft%20Office%20Live%20Plug-in%3BShockwave%20Flash%3BFlip4Mac%20Windows%20Media%20Plugin%202.3.8%20%3BGoogle%20Earth%20Plug-in%3BQuickTime%20Plug-in%207.6.6%3BSilverlight%20Plug-In%3BJava%20Plug-In%202%20for%20NPAPI%20Browsers%3B&pid=mny%3Ac%3Amoney%3A%2F2011%2F08%2F16%2Ftechnology%2Fgoogle_motorola%2F&pidt=1&oid=http%3A%2F%2Ftech.fortune.cnn.com%2F2011%2F08%2F18%2Fwall-streets-take-on-motorola-apple-up-google-down%2F&ot=A&AQE=1
HTTP/1.1" 400 357 [0, 1029]


Any thought on this issue are greatly appreciated...
--
Luis Soltero, Ph.D., MCS
Director of Software Development, CTO
Global Marine Networks, LLC
StarPilot, LLC
Tel: [phone]
Fax: [phone]
E-Mail: lsoltero at globalmarinenet.net
Web: http://www.globalmarinenet.net
Web: http://www.starpilotllc.com

[Robert Olofsson]

Hi!

On Sat, 20 Aug 2011 14:15:33 -0400

Post by Luis Soltero
what causes urls to get truncated as follows? I notice that we don't get these when running v 4.8 and 4.9. My concern is that these truncations are an artifact of the new logging additions. We are not running v10 so it could be an issue created by mods added to this version.

Sorry for taking time before answering this question. I have been looking
for clues.

I used to see similar things in 4.9 and earlier but have not seen any such
logs since 4.10. My guess is that it is one of the fixes to the buffer
handling and stack overflow that went into 4.10.

Currently I have no reproducible test case and no real idea where the
problem comes from.

Post by Luis Soltero
I would like to put the new logging code into production but this issue has me concerned.

I can understand that.

Does it seem to happen on any specific web pages?
Does it seem to affect the actual web pages when you look at them?

/robo

[Luis Soltero]

Hi Robo,

I think it mostly happens with websites that have really long urls.

I use www.cnn.com a lot for testing rabbit and these errors are generated very frequently using that site. You might
try that site. For example, i was able to generate the error just now with 3 clicks...

the web pages look ok but I think this is because there are so many urls on them that the broken ones are hard to spot.
I did have a situation where a page did not display at all when the page url gave an exception. Refreshing the page
fixed the problem.

Another website that causes issue is hotmail.com. If you start clicking on emails in the inbox about every 3 or 4
clicks the mesg will not display when the exception occurs. Clicking on the same mesg a second time causes it to display.

So the error occurs frequently but is not 100% reproducible. A url that causes the error at one time will not on
subsequent execution.

here is the error generated at the www.cnn.com site.

Aug 20, 2011 3:07:44 PM rabbit.proxy.Connection handleFailedRequestRead
INFO: Exception when reading request
rabbit.httpio.BadHttpHeaderException: Malformed
header:-adbp-domestic%3D%2526pid%253Dcnn%25253Asf%25253Anewspulse%25253A%25252F%2526pidt%253D1%2526oid%253Dhttp%25253A%25252F%25252Fwww.cnn.com%25252FUS%25252F%2526ot%253DA;
s_ppv=54
at rabbit.httpio.HttpHeaderParser.readHeader(HttpHeaderParser.java:176)
at rabbit.httpio.HttpHeaderParser.lineRead(HttpHeaderParser.java:143)
at rabbit.httpio.LineReader.readLine(LineReader.java:43)
at rabbit.httpio.HttpHeaderParser.handleBuffer(HttpHeaderParser.java:60)
at rabbit.httpio.HttpHeaderReader.parseBuffer(HttpHeaderReader.java:124)
at rabbit.httpio.HttpHeaderReader.read(HttpHeaderReader.java:112)
at org.khelekore.rnio.impl.ChannelOpsHandler.handleRead(ChannelOpsHandler.java:105)
at org.khelekore.rnio.impl.ChannelOpsHandler.handle(ChannelOpsHandler.java:160)
at org.khelekore.rnio.impl.SingleSelectorRunner.handleSelects(SingleSelectorRunner.java:350)
at org.khelekore.rnio.impl.SingleSelectorRunner.run(SingleSelectorRunner.java:204)
at java.lang.Thread.run(Thread.java:662)
Aug 20, 2011 3:07:46 PM rabbit.proxy.Connection handleFailedRequestRead
INFO: Exception when reading request
rabbit.httpio.BadHttpHeaderException: Malformed
header:%2526pidt%253D1%2526oid%253Dhttp%25253A%25252F%25252Fwww.cnn.com%25252FUS%25252F%2526ot%253DA; s_ppv=54
at rabbit.httpio.HttpHeaderParser.readHeader(HttpHeaderParser.java:176)
at rabbit.httpio.HttpHeaderParser.lineRead(HttpHeaderParser.java:143)
at rabbit.httpio.LineReader.readLine(LineReader.java:43)
at rabbit.httpio.HttpHeaderParser.handleBuffer(HttpHeaderParser.java:60)
at rabbit.httpio.HttpHeaderReader.parseBuffer(HttpHeaderReader.java:124)
at rabbit.httpio.HttpHeaderReader.read(HttpHeaderReader.java:112)
at org.khelekore.rnio.impl.ChannelOpsHandler.handleRead(ChannelOpsHandler.java:105)
at org.khelekore.rnio.impl.ChannelOpsHandler.handle(ChannelOpsHandler.java:160)
at org.khelekore.rnio.impl.SingleSelectorRunner.handleSelects(SingleSelectorRunner.java:350)
at org.khelekore.rnio.impl.SingleSelectorRunner.run(SingleSelectorRunner.java:204)
at java.lang.Thread.run(Thread.java:662)

Let me know if you are able to reproduce it...

--luis

Post by Robert Olofsson
Hi!
On Sat, 20 Aug 2011 14:15:33 -0400

Post by Luis Soltero
what causes urls to get truncated as follows? I notice that we don't get these when running v 4.8 and 4.9. My concern is that these truncations are an artifact of the new logging additions. We are not running v10 so it could be an issue created by mods added to this version.

Sorry for taking time before answering this question. I have been looking
for clues.
I used to see similar things in 4.9 and earlier but have not seen any such
logs since 4.10. My guess is that it is one of the fixes to the buffer
handling and stack overflow that went into 4.10.
Currently I have no reproducible test case and no real idea where the
problem comes from.

Post by Luis Soltero
I would like to put the new logging code into production but this issue has me concerned.

I can understand that.
Does it seem to happen on any specific web pages?
Does it seem to affect the actual web pages when you look at them?
/robo

--
Luis Soltero, Ph.D., MCS
Director of Software Development, CTO
Global Marine Networks, LLC
StarPilot, LLC
Tel: [phone]
Fax: [phone]
E-Mail: lsoltero at globalmarinenet.net
Web: http://www.globalmarinenet.net
Web: http://www.starpilotllc.com

[Rick Leir]

Does this make sense?

Thanks for the explanation of zero-copy, it was a big help.

I'm interested to see an aggregated ratio of client traffic to what it
would have been without RabbIT. The ratio would tell us how much RabbIT
is helping. Maybe it would be possible to post-process the logs to get
that; how would you do it?

Thanks -- Rick

[Robert Olofsson]

On Sun, 21 Aug 2011 20:40:36 -0400

Post by Rick Leir
I'm interested to see an aggregated ratio of client traffic to what it
would have been without RabbIT. The ratio would tell us how much RabbIT
is helping. Maybe it would be possible to post-process the logs to get
that; how would you do it?

I have been trying to figure that one out myself for many years, but I
have not found any good way to express the actual network traffic changes.
If you use the new traffic loggers you can probably do some post processing,
but the problem is that when rabbit sends data from the cache the data is
already compressed so we would need to figure out the original size of the
data, for images we can find that in the access log, but not for text
resources.

In short I am not sure, I have always just used the slow connection and
used wall clock time to see if rabbit seems to work or not.

/robo

[Luis Soltero]

The only way we have been able to determine the efficiency of rabbit is using the network feature of the firebug plugin
for firefox. The chrome console also has a similar feature.

This tool will tell you exactly the amount of speed up you get with the proxy. First you flush browser cash and disable
the proxy settings and then browse the pages you are interested in. Then you flush the cache again and enable proxy and
do the browsing again. You then manually computer the ratios. On average you will find about 3-5x improvement in
performance.

It would be really nice to do this system wide for all users and all pages. The above only works for spot checking and
is quite labor intensive.

Take care.

--luis

Post by Robert Olofsson
On Sun, 21 Aug 2011 20:40:36 -0400

Post by Rick Leir
I'm interested to see an aggregated ratio of client traffic to what it
would have been without RabbIT. The ratio would tell us how much RabbIT
is helping. Maybe it would be possible to post-process the logs to get
that; how would you do it?

I have been trying to figure that one out myself for many years, but I
have not found any good way to express the actual network traffic changes.
If you use the new traffic loggers you can probably do some post processing,
but the problem is that when rabbit sends data from the cache the data is
already compressed so we would need to figure out the original size of the
data, for images we can find that in the access log, but not for text
resources.
In short I am not sure, I have always just used the slow connection and
used wall clock time to see if rabbit seems to work or not.
/robo
_______________________________________________
Rabbit-dev mailing list
Rabbit-dev at khelekore.org
http://khelekore.org/cgi-bin/mailman/listinfo/rabbit-dev

--
Luis Soltero, Ph.D., MCS
Director of Software Development, CTO
Global Marine Networks, LLC
StarPilot, LLC
Tel: [phone]
Fax: [phone]
E-Mail: lsoltero at globalmarinenet.net
Web: http://www.globalmarinenet.net
Web: http://www.starpilotllc.com

[Robert Olofsson]

On Mon, 22 Aug 2011 12:40:10 -0400

Post by Luis Soltero
First you flush browser cash and disable
the proxy settings and then browse the pages you are interested in.
Then you flush the cache again and enable proxy and do the browsing again.
You then manually computer the ratios.

This is one metric, it is not the only way to compute such metrics.
This does not take the rabbit cache into account, how do you want to
count the speedup of user 2-10 when they ask for the same image as
user 1 asked for, rabbit<->network traffic = 0, rabbit<->client traffic
is the compressed image. To what user should the speedup be accounted?
user 1 or users 2-10 or spread it out evenly to users 1-10?

Also, what is the speedup of blocking an ad? most of the times when you
request a big page it will have ads that rotate, so it gets kind of hard
to measure.

When rabbit converts a large image to a low quality jpeg rabbit also
adds latency, how do you want to attribute that latency? is it a
speedup due to less network traffic or is it a slowdown due to latency?
It depends a lot on how fast the rabbit<->network connection is compared
to the rabbit<->browser connection.

All in all, quite complex to figure out a simple metric.

/robo