[Rabbit-dev] build a powerful server utilizing multiple proxy servers

[蔡超]

Hi,

I want to build a powerful proxy server using rabbit. The main idea is
to make a pool of proxy addresses, and maintain a mapping from a target host
to a proper proxy server which can reach the target fastest. At the same
time, there should be a powerful cache system with 10 GB level volume. I
have write a ProxyChain which can choose a Resolver for a given url. I can't
find any detail document for rabbit architecture, there are still two
problems:

1. How to know the time a proxy cost to fetch an url? where is the code
using the given proxy address?
2. Can Rabbit cache 10GB files?

Thanks.

Chao
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://khelekore.org/pipermail/rabbit-dev/attachments/20110728/6adf4efd/attachment.html>

[Robert Olofsson]

Hi!

On Thu, 28 Jul 2011 23:24:26 +0800

Post by 蔡超
I want to build a powerful proxy server using rabbit. The main idea is
to make a pool of proxy addresses, and maintain a mapping from a target host
to a proper proxy server which can reach the target fastest. At the same
time, there should be a powerful cache system with 10 GB level volume. I
have write a ProxyChain which can choose a Resolver for a given url. I can't
find any detail document for rabbit architecture, there are still two

The code is the documentation of the architecture. There is no real roadmap
or similar, things get added when someone either supplies a patch or requests
a feature that someone cares to develop.

If you have specific questions I can probably answer them.

Post by 蔡超
1. How to know the time a proxy cost to fetch an url? where is the code
using the given proxy address?

Rabbit does not currently try to guess how much time a request will take.
Internet is unreliable so how can you really say.
Rabbit does not do anything special today, it checks the request header
to see if it may use its own cache and if so checks if the resource
is currently in the cache. If the requested resource is in the cache
(or needs a revalidation) rabbit contacts the real server (possibly
through a chained proxy) and asks for it.
If the response from the real server allows caching then rabbit will
cache the resource.

Post by 蔡超
2. Can Rabbit cache 10GB files?

I do not really know. I have personally never had that large a cache for
rabbit, but in theory it ought to be ok. You could try to write a test
case and see what happen, or you can try it for real.
My rabbit cache is currently about 20 000 files, but that is only 63 MB
resource data. Most resource on the net are very small. Do you plan to
have any large files in the cache?

I would guess that we may want to do some modifications to the cache
if we want it to support 10GB.

I also guess it depends on what filesystem and hardware you are going
to put it on.

/robo