Content delivery and distributed file sharing networks

There are a variety of technologies for increasing the efficiency of files and other content to which many users want access. Some of these technologies have come into conflict with laws and business concerns about intellectual property rights, although it must be made clear that virtually all the conflicts involve the content being transferred, not the means of transfer. These techniques are an important part of convergence of communications, not for services supporting active conversations, but for distribution of information as diverse as software updates and digital music.

The means of transfer differ in the degree of distribution, ranging from serverless peer-to-peer (P2P), to peer-to-peer services with servers playing the role of broker to find desired content, to content distribution networks (CDN) with distributed servers owned by the content provider or a CDN contracted to it. Distributed information sharing is yet another paradigm that addresses some of the same issues as traditional file transfer, and evolving distributed file systems such as the Network File System (NFS). In distributed information sharing, there is far less of an emphasis on client/server than in other protocols. It is more similar to the Network News Transfer Protocol (NTTP).

Legal, ethical, and business issues

There are distinct security problems with some of the distributed models, and, indeed, some were shut down under legal pressure. The legal pressure came mostly from the entertainment industry, not amused with technology that enabled easy, royalty-free transfer of copyrighted music.

Needless to say, there was a conflict between the copyright holders of entertainment content, and music users who believed music was overpriced. A portion of participants in P2P networks indeed were copyright holders, who believed that the ability to sample tracks would get new customers to buy albums or attend live shows. The legality and morality of these practices are beyond the scope of this article, other than to notice they created much network demand and, indeed, new technologies. It is worth noting that several P2P networks have closed, or had major sites refuse to download their software.

All open implementations that can distribute files can be used to distribute both legal and illegal information, just as a photocopier can copy unauthorized material. At the same time, some large entertainment corporations have been extremely aggressive in legal attack, to the extent of creating chilling effect on legitimate users, demanding expensive and possibly intrusive technologies to be implemented by Internet Service Providers, and even suggesting that digital rights enforcement be built into standard integrated circuits that might be used for a purpose totally different than entertainment.

Bandwidth economics

College network administrators, in particular, found music downloads to take seemingly infinite amounts of bandwidth, interfering with their academic and research methods. Some administrators simply blocked ports used for file sharing, or content detected as multimedia (e.g., MP3). This was an issue of protecting resources, which handily countered arguments about academic freedom and censorship.

ISPs also may limit residential bandwidth on P2P ports, and sometimes have produced data showing that a small portion of the users are consuming a great deal of the bandwidth; quota systems or pricing on bandwidth overages may sometimes be needed. Contrary to some allegations, network neutrality is not intended to interfere with bandwidth-sensitive pricing; its focus is on content provider-sensitive pricing.

Of course, academic institutions tend to have large numbers of computer-literate individual with time and resources, so the situation began to resemble the military minuet of electronic warfare: electronic attack, countermeasure, and response by counter-countermeasures. It is noteworthy that the military has generalized to the concept of information operations, which includes electronic warfare, but also attack, countermeasure, and counterattack on computers and computer networks.

P2P users, especially with questionable legal positions, might encrypt content, defeating MP3 detection, and send it to a nonstandard port, or just try nonstandard (or dynamically changing) ports until one was blocked.

Content delivery networks (CDN)

Akamai pioneered content distribuion, using a push technology with content providers as their customers. They pushed the content to local caches at ISPs, which could deliver content to end users with less upstream bandwidth than the ISPs had previously needed, as well as incentives to the ISPs.

They have evolved their service to provide various edge accelerators, some intended as products and some as Akamai-run services. Time-Warner took over content distribution for CNN after Time-Warner bought the news company, and distributes Time-Warner content over its own CDN.

P2P with assistance in finding information

There are a wide range of P2P services whose relationships are established by personal contact and social networks. Their content may be legal or illegal to transfer.

Still, finding content can be a major challenge without purpose-built assistance. While it has shut down, Napster was the first widely available "peer to peer" service. It was not true point-to-point as was the original NetNews, because it did have servers to help mediate content exchanges, although the actual exchange was P2P.

Kazaa

Kazaa also is a system with "matchmaking" servers and direct P2P transfer between user machines. As with other programs, it had its assortment of legal problems with copyright. Helping Kazaa was a ruling from a Dutch court that it was, essentially, a carrier not responsible for the acts of its users.

BitTorrent

Symbolic representation of BitTorrent

BitTorrent is a hybrid between pure peer-to-peer, and varying levels of server involvement. The servers are more concerned with directory functions and managing content flows that actually holding the content. BitTorrent seems to have been much more successful than other P2P companies in establishing working relationships with the entertainment industry, avoiding the adversarial relationships that closed down other P2P systems. When the protocol first deployed, it was at end user quality levels. Evolved versions can meet enterprise quality needs. There is also a content distribution service, based on BitTorrent technology but as opposed to P2P, with contractual SLAs. CacheLogic runs this commercial service. BitTorrent claims that its traffic is 35% of the total Internet traffic, although this has not been independently confirmed.

The original BitTorrent client, written in the Python programming language, had its source code released under the BitTorrent Open Source License, which is a modified version of the Jabber Open Source License. There are numerous compatible clients, written in a variety of programming languages, and running on a variety of computing platforms.

Protocol architecture

While BitTorrent implements are called clients, the protocol operations are not strongly client-server. NetNews is a conceptual predecessor, in that the clients exchange information about what they have and what they need. As opposed to news, however, the exchanges are not pairwise, but as parallel as possible. The more clients that have the file of interest, the smaller the workload of each. Each client in the “swarm” of cooperating computers knows which other pieces of the file have been transferred.

After downloading a file, the BitTorrent netiquette is to act as a seeder after it has the complete file. This, it offers the file to other clients. Being able to expand the number of sources of content led to the marketing-oriented but meaningful term "application multicasting".

As important as the share ratio may be the length of time the client stays open. The longer that time, the more new clients can find a peer that has that file.

How do would-be downloaders find the clients with the files? The first client to get the file and offer it, called the initial seeder, registers its existence and content with a server called a tracker. Trackers make the information about seeders and trackers available as web pages, but, unless they are coincidentally seeders/clients, do not themselves have the content. They are merely brokers.

In NNTP, the client requests specific messages, but here, they request random pieces of the content file. Randomness increases the probability that more than one client will have the file, such that the work is spread over more computers. The larger the file, the larger the piece. A hash for error detection covers the pieces.

As more peers join the swarm, they share pieces with one another, reducing the load on the initial seeder. Clients should prefer to send data to peers who upload data to them, a means of encouraging fair sharing of the work. If this policy is overly strict, however, a peer that just joined the swarm may not get any data because it is not offering any.

BitTorrent business models

Where several P2P vendors have gone out of business, typically due to huge legal problems with pirated information, BitTorrent seems to have managed to stay more mainstream. BitTorrent distributes BitTorrent's own updates, as well as OpenOffice.org and most LINUX distributions. Free and Open Source conferences have distributed video of their conferences via BitTorrent. NASA uses it for large image files.

Bram Cohen, the inventor of BitTorrent, and the movie industry executed an agreement in 2005. It required that the bittorrent.com website not have any linked to unlicensed movie content. Of course, this had no effect on other servers not under Cohen's control, but it appears there is a mutual attempt to work out compromises between this P2P operator and the entertainment industry than with several other services.

BitTorrent has received the usual legal attacks over copyright infringement, but its main counterargument is that its servers do not contain copyrighted information, but only metadata about it. Nevertheless, the matter is before several courts at the time of this writing. Law enforcement, in several countries, seized servers. BitTorrent websites, perhaps more than for other P2P services, tend to have prominent anti-piracy warnings. Other BitTorrent servers not subject to this agreement continue to be shut down through legal action. BitTorrent's design does give it some legal protections that other P2P services do not have. BitTorrent itself does not have a search engine, so clients are responsible for finding the material.