[Rabbit-dev] Prefetch possibility

[Bill]

My multilink ppp modem setup is like wireshark in that it is sooo slow I can see each dns negotiation in the status bar of firefox as they happen. :)

My 2G cellular is no better.

With that in mind, I've sold my soul to the satellite devil (the price you have to pay to get internet in america).

Satellite = latency. 1 to 1.5 SECONDS per transaction.

I'm terrified that these busy sites like CNN and Google News will crawl due to latency.

Does Rabbit have a feature to prefetch all site references, assemble the page on the server site, and send me like a "zip bundle" of the page? Instead of a transaction for each element on the page, send me 1 zip file containing all of the page.

Would this be hard to implement?

This should also help modemers as it would dramaticly cut down on the http overhead.

Ideas?
Bill
bsaunder2002 at yahoo.com

[Robert Olofsson]

On Thu, 2 Feb 2012 10:30:51 -0800 (PST)

Post by Bill
Does Rabbit have a feature to prefetch all site references, assemble
the page on the server site, and send me like a "zip bundle" of the
page? Instead of a transaction for each element on the page, send me
1 zip file containing all of the page.

No, rabbit does not have that.

Post by Bill
Would this be hard to implement?

Yes, it would mean that you have to add a full javasript-browser
module to rabbit to have it understand what the javascript will do
when it run on each page.
It would most probably also mean adding a client side component,
something that rabbit tries to avoid.

There is a standard that inlines images as binary content into
html-pages, but I am unsure of the name. Rabbit could possibly
support that, but it would add latency to the html handling in
rabbit and I am unsure of browser support for it.

/robo

[Samat K Jain]

My multilink ppp modem setup is like wireshark in that it is sooo slow I can see each dns negotiation in the status bar of firefox as they happen.:)

Setup a local (i.e. on your home network) DNS server. I recommend pdnsd.

Most Web browsers (Firefox, at least) have an option to have the proxy
server perform DNS requests for you. Unfortunately, AFAIK this only
works for SOCKS proxies?it won't work w/ RabbIT, which is an HTTP proxy.

My 2G cellular is no better.
With that in mind, I've sold my soul to the satellite devil (the price you have to pay to get internet in america).
Satellite = latency. 1 to 1.5 SECONDS per transaction.

I feel for you. I was with WildBlue in the US; my median RTT was 1.4s.
I've since switched to 4G LTE and while it's still unreliable and
expensive at least RTTs are below 100ms.

I'm terrified that these busy sites like CNN and Google News will crawl due to latency.
Does Rabbit have a feature to prefetch all site references, assemble the page on the server site, and send me like a "zip bundle" of the page? Instead of a transaction for each element on the page, send me 1 zip file containing all of the page.
Would this be hard to implement?
This should also help modemers as it would dramaticly cut down on the http overhead.

You might want to look into a feature called "HTTP pipelining". RabbIT
supports this relatively well (and is one of the few proxies that does;
Squid's support is terrible).

Normally, your browser requests a resource, waits for it to be received,
requests another, etc. With pipelining (aka multiplexing), your browser
sends all the requests at once. If the remote server supports it (not
enough do), it cuts down latency significantly.

[Robert Olofsson]

On Thu, 02 Feb 2012 16:19:27 -0700

Post by Samat K Jain
Most Web browsers (Firefox, at least) have an option to have the
proxy server perform DNS requests for you. Unfortunately, AFAIK this
only works for SOCKS proxies?it won't work w/ RabbIT, which is an
HTTP proxy.

It was some time since I checked this last, but I think that
when you use a proxy the browser will only query the dns for the
proxy address (if needed, you can specify the proxy as 1.2.3.4).
It is actually the proxy that will do the dns lookups, since it
is the proxy that does the connecting to the real server.

There is a reason rabbit comes with the dnsjava library.

Post by Samat K Jain
You might want to look into a feature called "HTTP pipelining".
RabbIT supports this relatively well (and is one of the few proxies
that does; Squid's support is terrible).

Rabbits support for client-rabbit connection pipelining is good.

The support for rabbit-server pipelining is not so good, the big
problem is deciding when to use it and then we have the added
complexity of resends if pipelining fails.

/robo

[Mindaugas Žakšauskas]

Hi,

Post by Bill
Does Rabbit have a feature to prefetch all site references, assemble
the page on the server site, and send me like a "zip bundle" of the
page? ?Instead of a transaction for each element on the page, send me
1 zip file containing all of the page.

I think proxy is a wrong tool for such a job. As Robert mentioned, it
focuses on HTTP, not HTML layer.

If you want similar functionality, look for a browser which is capable
of prefetching. Some of the browsers (I know Opera, FF) do the DNS
pre-fetch, and some even do content pre-fetch (Amazon Silk, Google
Chrome [1]).

m.

1. http://chrome.blogspot.com/2012/01/speed-and-security.html