Possibly a fast way to push Anime around the internet (without P2P/BT)?

[Doppelganger]

Freecache and/or Coral

For those of you who read Slashdot, forgive me; if you saw this article this post will bore you a bit.

http://it.slashdot.org/article.pl?si...tid=95&tid=218

For the uninitiated: (Skip this section if you have a good idea what a web cache is, the juicy stuff is the terse few lines at the very end...)
A cache is something that exploits what is called “locality” in most computer accesses. Meaning that we use the fact that accesses of data are not random, so we can store data in a more convenient space for future retrieval, guessing that it'll be needed again.

In this case, the Coral or Freecache servers will get a copy of the file being cached and serve it to all the people requesting that file in their area. These computers, as you may imagine, are on very large pipes. Here are a few reasons why caching is good:

The provider of content doesn't have their bandwidth costs shoot through the roof. As a matter of fact as copies of the file are propagated through the cache usage should drop close to zero. The content provider is happy.

We use less bandwidth on the internet as a whole; BT (as it exists now) doesn't make concessions for distance and such, it's most interested in simply finding peers that have data you don't, regardless of network topology. (eg. there's no reason why my client wouldn't decide to trade with a guy in the UK from California) With Freecache/Coral the data (ideally) is only transmitted once over the backbone and is cached for future requests, backbone providers are happy and the internet (theoretically) becomes a less laggy place.

The big cache servers are near their users and (supposedly) on big bandwidth connections, or better yet, run by the user's ISP so the download is fast. The user is then happy because they got their file quickly.

Finally, leechers and people with firewall woes will be attracted to this option if it's viable, and those who still choose to use BT will have less people connecting who do not intend to seed, so BT users are happy because their downloads will be fast(er).

The Goods, how to set up caching:
The first step for both is the same: locate the video file somewhere on a HTTP server.
For FreeCache:
http://freecache.org/http://your.domain/path.to/your.file
For Coral:
http://your.domain.nyud.net:8090/path.to/your.file

Your.domain, path.to, and your.file are placeholders.

Of course, we have no idea what kind of infrastructure these two options have, so that's why I'm posting here hoping that some venturous group will release a Coralized/FreeCached version of their next big release. (I'd love to see what would happen in the case of Naruto, which as far as I know are the biggest anime torrents I have witnessed...)

Only a real test will be able to prove or disprove the viability of these options...

Personal note: Freecache was flaky for a while after it got Slashdotted a few months ago, but it does work most of the time now, or at least doesn't take so long to time out when it can't find a suitable cache. The Corel caching seems more robust. Try both...perhaps even simultaneously. After all, it's just another html tag, right? Edit: Having viewed the Freecache boards, it appears that it's been coming up and down. Beware that a downed Freecache will mean that a) the link will not work or worse yet b)it'll direct all traffic to the webhost, and you'll be overwhelmed instantly. I am not sure what Coral defaults to when the cache goes down (which I have not seen it do just yet, but to be fair I've seen it for about an hour....granted, an hour after it was posted on slashdot, so it must be able to survive some battering pretty well.)

Thus be somewhat careful; while in theory these caches should ward off hordes of users downloading directly from your server, there is no guarantee. You also cannot remove the file from the server, as caches purge broken links as soon as possible.

[TaMz]

The Goods, how to set up caching:
The first step for both is the same: locate the video file somewhere on a HTTP server.
For FreeCache:
http://freecache.org/http://your.dom...h.to/your.file
For Coral:
http://your.domain.nyud.net:8090/path.to/your.file

Your.domain, path.to, and your.file are placeholders.

that means the one wanting to cache must have a http server where to upload, and then upload the file there first...
thus it doesn't make it any easier than BT and such (or even normal HTTP/FTP downloading)...

[lamer_de]

CoralCDN is a decentralized, self-organizing, peer-to-peer web-content distribution network. CoralCDN leverages the aggregate bandwidth of volunteers running the software to absorb and dissipate most of the traffic for web sites using the system. In so doing, CoralCDN replicates content in proportion to the content's popularity, regardless of the publisher's resources---in effect democratizing content publication.

No sane ISP will be willing to act as a volunteer for the distribution of illegal material. Also, the system seems to be targeted at web content, which usually ranges from a few bytes to a few kilobytes and not 175 MB or more.

CU,
lamer_de

[Doppelganger]

Originally Posted by lamer_de

No sane ISP will be willing to act as a volunteer for the distribution of illegal material. Also, the system seems to be targeted at web content, which usually ranges from a few bytes to a few kilobytes and not 175 MB or more.

CU,
lamer_de

Incorrect and Incorrect.

Comcast not long ago cached newsgroups full of copyright infringing stuff, why? Because the bandwidth costs of fetching data each time for their large number of users would be immense. I'm not sure about nowadays, but @Home used to run a web cache server that I'm sure picked up illegal stuff here and there, by virtue of being a cache. Caching is not new, but there are not many public services for such...Akamai is an example of a long-standing fast, load balancing, commercial cache for big corporations.


Regarding the "small files only" idea:
From the freecache faq:
What files are being served by FreeCache?

FreeCache can only serve files that are on a web site. If the link to a file on that web site goes away, so will the file in the FreeCaches. Also, there is a minimum size requirement. We don't bother with files smaller than 5MB, as the saved bandwidth does not outweight the protocol overhead in those cases.

Caches are designed to save bandwidth; no ISP care about small images and text anymore. The idea is to save overall bandwidth on the net, so large files are the favored targets of caching. Even the cheapest hosting could serve millions of pages of text each month with no problem. That having been said, I do believe Coral does cache the small stuff on sites as well and replace simple links with redirects to the cached version. I would think they'd write a limitation on size in their FAQ.

Finally, why do it? Some people have firewalls that don't allow BT. BT still causes significant traffic over internet backbones since it doesn't consider network topology when choosing who to download from and upload to. Some people don't like uploading or have metered uploads. Finally, it isn't difficult to set up.

It's not so much as a replacement for BT so much as an alternative. Also I recommend not putting old files on the freecache--just the newest and the ones with the most heavy traffic. Old files that get downloaded now and then will quickly be cycled out of the cache and the hosting site will take a bandwidth hit. Caches rely on downloads to decide what the best stuff to cache with finite resources, so perhaps it'd be a good way to get the file out the first couple of days.

[Doppelganger]

Originally Posted by TaMz

that means the one wanting to cache must have a http server where to upload, and then upload the file there first...
thus it doesn't make it any easier than BT and such (or even normal HTTP/FTP downloading)...

Uh, except normal HTTP/FTP would see the hosting site ground into the dirt from numerous requests. Not only that, but it isn't very "smart," if I download something from http from Germany from California, it comes from Germany and is delivered to California. If I click on a Coralized link, Coral will get the file, shove it to the nearby Northern California cluster, and then deliver to me, and more importantly, any user in the area who also requests the file via the Coral link, thus sparing that backbone additional traffic. Of course, to the end user, this doesn't mean much...unless the backbone gets overloaded and slow. In any case, it's worth a try I think. I would simply set up http server and get the stuff cached myself, but in order to really get a good idea of how capable the infrastructure is for freecache/Coral it'd be helpful to have the group itself offer it as an alternative to try the caching system with some larger number of users.

Edit: Coral has maps and stats on the cache...
http://www.scs.cs.nyu.edu/coral/stats/
http://www.scs.cs.nyu.edu/coral/maps/

[lamer_de]

Comcast not long ago cached newsgroups full of copyright infringing stuff, why? Because the bandwidth costs of fetching data each time for their large number of users would be immense

Probably because people pay to access newsgroups (either seperate or with their isp fees)?

But yeah, they have a list of running servers, so the system already seems working.

You tried it for yourself already? Like setting up a home http with naruto and posting that http adress in #animeone? After one upload from you, it'd be distributed between all those .edus effectively taking away all load from your connection, right?

CU,
lamer_de

[AnimeOni]

Let's take it from a legal point of view. Is Caching legal? Well, since FS are technically illegal, what are the ramnifications of caching illegal materials?

Let's take a look at another technology such as FastTrack using Joltid PeerCache Technologies. They cache P2P materials for services suah as Kazza and iMesh. (BTW, Joltid founder co-founded Kaaza). The problem of holding illegal materials is now in the hands of the ISP (in FS materials, namely EDUs and ISPs). By holding such materials, ISPs and EDUs become accomplices. They can be held responsible.

By using BitTorrent, the ISPs are not fully responsible since they are not 'hosting' the materials. they may be held indirectly responsible for allowing their end users access to such materials and not distributing the materials. BT also allows a semi-anonymous/decentralized network. Peers connect to other peers. The only thing that the tracker holds is who is requesting the file(s) but not the actual file(s).

FreeCache looks good in reducing overall traffic and allows for faster DLs since the 'host' is placed geographically closer, but in today's sue-happy world and corporate dominance, I think using FreeCache may not be a good idea for materials such as FS since it allows for easier targeting of individuals and ISPs. When ISPs get wind of such cache servers (they will know by the amount of traffic), ISPs will be very quick to take them down.

BT is harder to take down since the files are 'scattered' It is more difficult to determine who is hosting the file. (they have to download the complete file from a single individual before they can determine the legitimancy of the file. It is easier for them to take down the trackers than taking down each individual persn.

This is just my 2-cents.
________
Rhode Island Dispensaries

[TronDD]

Along those lines, whoever ownes that cache server (let's say each ISP puts on in each area of service) will have the right to monitor what's on it and who accesses it. Fansubs and other illegal material may never end up on it and if it does it could end up being used to shut down individuals who access it. Downloading copyrighted material is already a violation of your ISP's usage policies and if they could see you download it, they would more than justified in cutting you off.

If the ISPs don't own the servers, they will be hanging off of someone's network who will also have an AUP. Then we'll have the same problem as direct download sites. I suppose owners of caching servers can claim blind ignorance to what gets cached but that's not working to well for P2P networks. (Although, when a big company tries to use that defence, they'll be mare monitarilly capable of backing it up. Unlike some guy in his basement writting P2P software.)

I think caching is a logical step in network evolution but the abundance of illegal material will be a huge hinderance to its suceess, I think.