Projects/Data Distribution

= Purpose =
 * To investigate systems for distributing large amounts of data particularly in relation to the Kforge project
 * To build a bit torrent system for okfn to support data distribution
 * See also projects/Distributed_Storage

= Status =
 * Started: 2005-10
 * In Incubation
 * Founder: Jo Walsh and Rufus Pollock

= Contact =
 * please contact rufus <> pollock at okfn <> org
 * Mailing list: use okfn-discuss [at] lists [dot] okfn [dot] org

= Information =

See also ToolsWeNeed and SoftwareTools

Email sent to okfn-discuss 2006-01-12

Recently I've been thinking about distributing data, especially large amounts of it (I've started an incubator project on OKFN [1]). Recent trends in distributing software seem to be towards using a p2p protocol such as bittorrent, though it is noticeable that most software download is still fairly traditional in being straight file transfer (either via http or ftp).

Given that for 'knowledge' we are often talking about chunks of data that are sizeable compared to normal software (perhaps equivalent to a linux distro or bigger) it seems sensible to go down this route. I don't yet know much about this and my experience with BT only extends to downloading a few distros -- I have never distributed using it -- so I would welcome any comments people had about this (I have read [2]). Some of the things i'd particularly like information on are:


 * 1) Mirroring. By the nature of BT I assume you can't explicitly select a mirror as you often do when downloading in the traditional manner so to what extent does BT do mirroring for you?
 * 2) Is there any way to do fallback in BT so that clients which can't support BT can download normally? If not is this an issue?
 * 3) Any tips on setting up a tracker (or should one use trackerless torrents)?

[1] http://www.okfn.org/wiki/DataDistribution [2] http://www.bittorrent.com/guide.html