1.1. Discussion groups

The Usenet is a huge worldwide collection of discussion groups. Each discussion group has a name, e.g. comp.os.linux.announce, and a collection of messages. These messages, usually called articles, are posted by readers like you and me who have access to Usenet servers, and are then stored on the Usenet servers.

This ability to both read and write into a Usenet newsgroup makes the Usenet very different from the bulk of what people today call ``the Internet.'' The Internet has become a colloquial term to refer to the World Wide Web, and the Web is (largely) read-only. There are online discussion groups with Web interfaces, and there are mailing lists, but Usenet is probably more convenient than either of these for most large discussion communities. This is because the articles get replicated to your local Usenet server, thus allowing you to read and post articles without accessing the global Internet, something which is of great value for those with slow Internet links. Usenet articles also conserve bandwidth because they do not come and sit in each member's mailbox, unlike email based mailing lists. This way, twenty members of a mailing list in one office will have twenty copies of each message copied to their mailboxes. However, with a Usenet discussion group and a local Usenet server, there's just one copy of each article, and it does not fill up anyone's mailbox.

Another nice feature of having your own local Usenet server is that articles stay on the server even after you've read them. You can't accidentally delete a Usenet articles the way you can delete a message from your mailbox. This way, a Usenet server is an excellent way to archive articles of a group discussion on a local server without placing the onus of archiving on any group member. This makes local Usenet servers very valuable as archives of internal discussion messages within corporate Intranets, provided the article expiry configuration of the Usenet server software has been set up for sufficiently long expiry periods.

1.2. How it works, loosely speaking

Usenet news works by the reader first firing up a Usenet news program, which in today's GUI world will highly likely be something like Netscape Messenger or Microsoft's Outlook Express. There are a lot of proven, well-designed character-based Usenet news readers, but a proper review of the user agent software is outside the scope of this HOWTO, so we will just assume that you are using whatever software you like. The reader then selects a Usenet newsgroup from the hundreds or thousands of newsgroups which are hosted by her local server, and accesses all unread articles. These articles are displayed to her. She can then decide to respond to some of them.

When the reader writes an article, either in response to an existing one or as a start of a brand-new thread of discussion, her software posts this article to the Usenet server. The article contains a list of newsgroups into which it is to be posted. Once it is accepted by the server, it becomes available for other users to read and respond to. The article is automatically expired or deleted by the server from its internal archives based on expiry policies set in its software; the author of the article usually can do little or nothing to control the expiry of her articles.

A Usenet server rarely works on its own. It forms a part of a collection of servers, which automatically exchange articles with each other. The flow of articles from one server to another is called a newsfeed. In a simplistic case, one can imagine a worldwide network of servers, all configured to replicate articles with each other, busily passing along copies across the network as soon as one of them receives a new articles posted by a human reader. This replication is done by powerful and fault-tolerant processes, and gives the Usenet network its power. Your local Usenet server literally has a copy of all current articles in all relevant newsgroups.

1.3. About sizes, volumes, and so on

Any would-be Usenet server administrator or creator must read the "Periodic Posting about the basic steps involved in configuring a machine to store Usenet news," also known as the Site Setup FAQ, available from ftp://rtfm.mit.edu/pub/usenet/news.answers/usenet/site-setup or ftp://ftp.uu.net/usenet/news.answers/news/site-setup.Z. It was last updated in 1997, but trends haven't changed much since then, though absolute volume figures have.

If you want your Usenet server to be a repository for all articles in all newsgroups, you will probably not be reading this HOWTO, or even if you do, you will rapidly realise that anyone who needs to read this HOWTO may not be ready to set up such a server. This is because the volumes of articles on the Usenet have reached a point where very specialised networks, very high end servers, and large disk arrays are required for handling such Usenet volumes. Those setups are called ``carrier-class'' Usenet servers, and will be discussed a bit later on in this HOWTO. Administering such an array of hardware may not be the job of the new Usenet administrator, for which this HOWTO (and most Linux HOWTO's) are written.

Nevertheless, it may be interesting to understand what volumes we are talking about. Usenet news article volumes have been doubling every fourteen months or so, going by what we hear in comments from carrier class Usenet administrators. In the beginning of 1997, this volume was 1.2 GBytes of articles a day. Thus, the volumes should have roughly done five doublings, or grown 32 times, by the time we reach mid-2002, at the time of this writing. This gives us a volume of 38.4 GBytes per day. Assume that this transfer happens using uncompressed NNTP (the norm), and add 50% extra for the overheads of NNTP, TCP, and IP. This gives you a raw data transfer volume of 57.6 GBytes/day or about 460 Gbits/day. If you have to transfer such volumes of data in 24 hours (86400 seconds), you'll need raw bandwidth of about 5.3 Mbits per second just to receive all these articles. You'll need more bandwidth to send out feeds to other neighbouring Usenet servers, and then you'll need bandwidth to allow your readers to access your servers and read and post articles in retail quantities. Clearly, these volume figures are outside the network bandwidths of most corporate organisations or educational institutions, and therefore only those who are in the business of offering Usenet news can afford it.

At the other end of the scale, it is perfectly feasible for a small office to subscribe to a well-trimmed subset of Usenet newsgroups, and exclude most of the high-volume newsgroups. Starcom Software, where the authors of this HOWTO work, has worked with a fairly large subset of 600 newsgroups, which is still a tiny fraction of the 15,000+ newsgroups that the carrier class services offer. Your office or college may not even need 600 groups. And our company had excluded specific high-volume but low-usefulness newsgroups like the talk, comp.binaries, and alt hierarchies. With the pruned subset, the total volume of articles per day may amount to barely a hundred MBytes a day or so, and can be easily handled by most small offices and educational institutions. And in such situations, a single Intel Linux server can deliver excellent performance as a Usenet server.

Then there's the internal Usenet service. By internal here, we mean a private set of Usenet newsgroups, not a private computer network. Every company or university which runs a Usenet news service creates its own hierarchy of internal newsgroups, whose articles never leave the campus or office, and which therefore do not consume Internet bandwidth. These newsgroups are often the ones most hotly accessed, and will carry more internally generated traffic than all the ``public'' newsgroups you may subscribe to, within your organisation. After all, how often does a guy have something to say which is relevant to the world at large, unless he's discussing a globally relevant topic like ``Unix rules!''? If such internal newsgroups are the focus of your Usenet servers, then you may find that fairly modest hardware and Internet bandwidth will suffice, depending on the size of your organisation.

The new Usenet server administrator has to undertake a sizing exercise to ensure that he does not bite off more than he, or his network resources, can chew. We hope we have provided sufficient information for him to get started with the right questions.

2.1. Newsgroups and articles

A Usenet news article sits in a file or in some other on-disk data structure on the disks of a Usenet server, and its contents look like this:

Xref: news.starcomsoftware.com starcom.tech.misc:211 starcom.tech.security:452
Newsgroups: starcom.tech.misc,starcom.tech.security
Path: news.starcomsoftware.com!purva!shuvam
From: Shuvam <shuvam@starcomsoftware.com>
Subject: "You just throw up your hands and reboot" (fwd)
Content-Type: TEXT/PLAIN; charset=US-ASCII 
Distribution: starcom
Organization: Starcom Software Pvt Ltd, India
Message-ID: <Pine.LNX.4.31.0107022153490.30462-100000@starcomsoftware.com>
Mime-Version: 1.0
Date: Mon, 2 Jul 2001 16:27:57 GMT

Interesting quote, and interesting article.

Incidentally, comp.risks may be an interesting newsgroup to follow. We
must be receiving the feed for this group on our server, since we
receive all groups under comp.*, unless specifically cancelled. Check it
out sometime.

comp.risks tracks risks in the use of computer technology, including
issues in protecting ourselves from failures of such stuff.

Shuvam

> Date: Thu, 14 Jun 2001 08:11:00 -0400
> From: "Chris Norloff" <cnorloff@norloff.com>
> Subject: NYSE: "Throw up your hands and reboot"
> 
> When the New York Stock Exchange computer systems crashed for 85
> minutes (8 Jun 2001), Andrew Brooks, chief of equity trading at
> Baltimore mutual fund giant T. Rowe Price, was quoted as saying "Hey,
> we're all subject to the vagaries of technology. It happens on your
> own PC at home. You just throw up your hands and reboot."
> 
> http://www.washingtonpost.com/ac3/ContentServer?articleid=A42885-2001Jun8&pagename=article
> 
> Chris Norloff
> 
> 
> This is from --
> 
> From: risko@csl.sri.com (RISKS List Owner)
> Newsgroups: comp.risks
> Subject: Risks Digest 21.48
> Date: Mon, 18 Jun 2001 19:14:57 +0000 (UTC)
> Organization: University of California, Berkeley
> 
> RISKS-LIST: Risks-Forum Digest  Monday 19 June 2001
> Volume 21 : Issue 48
> 
>    FORUM ON RISKS TO THE PUBLIC IN COMPUTERS AND RELATED SYSTEMS (comp.risks)
>    ACM Committee on Computers and Public Policy,
>    Peter G. Neumann, moderator
> 
> This issue is archived at <URL:http://catless.ncl.ac.uk/Risks/21.48.html>
> and by anonymous ftp at ftp.sri.com, cd risks .
> 

A Usenet article's header is very interesting if you want to learn about the functioning of the Usenet. The From, Subject, and Date headers are familiar to anyone who has used email. The Message-ID header contains a unique ID for each message, and is present in each email message, though not many non-technical email users know about it. The Content-Type and Mime-Version headers are used for MIME encoding of articles, attaching files and other attachments, and so on, just like in email messages.

The Organisation header is an informational header which is supposed to carry some information identifying the organisation to which the author of the article belongs. What remains now are the Newsgroups, Xref, Path and Distributions headers. These are special to Usenet articles and are very important.

The Newsgroups header specifies which newsgroups this article should belong to. The Distributions header, sadly under-utilised in today's globalised Internet world, allows the author of an article to specify how far the article will be re-transmitted. The author of an article, working in conjunction with well-configured networks of Usenet servers, can control the ``radius'' of replication of his article, thus posting an article of local significance into a newsgroup but setting the Distribution header to some suitable setting, e.g. local or starcom, to prevent the article from being relayed to servers outside the specified domain.

The Xref header specifies the precise article number of this article in each of the newsgroups in which it is inserted, for the current server. When an article is copied from one server to another as part of a newsfeed, the receiving server throws away the old Xref header and inserts its own, with its own article numbers. This indicates an interesting feature of the Usenet system: each article in a Usenet server has a unique number (an integer) for each newsgroup it is a part of. Our sample above has been added to two newsgroups on our server, and has the article numbers 211 and 452 in those groups. Therefore, any Usenet client software can query our server and ask for article number 211 in the newsgroup starcom.tech.misc and get this article. Asking for article number 452 in starcom.tech.security will fetch the article too. On another server, the numbers may be very different.

The Path specifies the list of machines through which this article has travelled before it has reached the current server. UUCP-style syntax is used for this string. The current example indicates that a user called shuvam first wrote this article and posted it onto a computer which calls itself purva, and this computer then transferred this article by a newsfeed to news.starcomsoftware.com. The Path header is critical for breaking loops in newsfeeds, and will be discussed in detail later.

Our sample article will sit in the two newsgroups listed above forever, unless expired. The Usenet software on a server is usually configured to expire articles based on certain conditions, e.g. after it's older than a certain number of days. The C-News software we use allows expiry control based on the newsgroup hierarchy and the type of newsgroup, i.e. moderated or unmoderated. Against each class of newsgroups, it allows the administrator to specify a number of days after which the article will be expired. It is possible for an article to control its own expiry, by carrying an Expires header specifying a date and time. Unless overriden in the Usenet server software, the article will be expired only after its explicit expiry time is reached.

2.2. Of readers and servers

Computers which access Usenet articles are broadly of two classes: the readers and the servers. A Usenet server carries a repository of articles, manages them, handles newsfeeds, and offers its repository to authorised readers to read. A Usenet reader is merely a computer with the appropriate software to allow a user to access a software, fetch articles, post new articles, and keep track of which articles it has read in each newsgroup. In terms of functionality, Usenet reading software is less interesting to a Usenet administrator than a Usenet server software. However, in terms of lines of code, the Usenet reader software can often be much larger than Usenet server software, primarily because of the complexities of modern GUI code.

Most modern computers almost exclusively access Usenet servers using the NNTP (Network News Transfer Protocol) for reading and posting. This protocol can also be used for inter-server communication, but those aspects will be discussed later. The NNTP protocol, like any other well-designed TCP-based Internet protocol, carries ASCII commands and responses terminated with CR-LF, and comprises a sequence of commands, somewhat reminiscent of the POP3 protocol for email. Using NNTP, a Usenet reader program connects to a Usenet server, asks for a list of active newsgroups, and receives this (often huge) list. It then sets the ``current newsgroup'' to one of these, depending on what the user wants to browse through. Having done this, it gets the meta-data of all current articles in the group, including the author, subject line, date, and size of each article, and displays an index of articles to the user.

The user then scans through this list, selects an article, and asks the reader to fetch it. The reader gives the article number of this article to the server, and fetches the full article for the user to read through. Once the user finishes his NNTP session, he exits, and the reader program closes the NNTP socket. It then (usually) updates a local file in the user's home area, keeping track of which news articles the user has read. These articles are typically not shown to the user next time, thus allowing the user to progress rapidly to new articles in each session. The reader software is helped along in this endeavour by the Xref header, using which it knows all the different identities by which a single article is identified in the server. Thus, if you read the sample article given above by accessing starcom.tech.misc, you'll never be shown this article again when you access starcom.tech.misc or starcom.tech.security; your reader software will do this by tracking the Xref header and mapping article numbers.

When a user posts an article, he first composes his message using the user interface of his reader software. When he finally gives the command to send the article, the reader software contacts the Usenet server using the pre-existing NNTP connection and sends the article to it. The article carries a Newsgroups header with the list of newsgroups to post to, often a Distribution header with a distribution specification, and other headers like From, Subject etc. These headers are used by the server software to do the right thing. Special and rare headers like Expires and Approved are acted upon when present. The server assigns a new article number to the article for each newsgroup it is posted to, and creates a new Xref header for the article.

Transfer of articles between servers is done in various ways, and is discussed in quite a bit of detail in Section XXX titled ``Newsfeeds'' below.

2.3. Newsfeeds

2.4. Control messages

The Usenet is a massive dispersed collection of servers which operate almost without any supervision, provided they have adequate disk space and do not suffer disk corruption due to power failures, etc. (It is indeed surprising how self-managing a good Usenet server is, provided these two pre-requisites are met.) These servers are each under the control of human administrators, but it is preferable that certain routine actions be performed across all these servers remotely from one location, without the manual intervention of these humans.

One common need for centralised operations is the creation of new groups in the standard eight hierarchies. The Usenet follows a fairly formal process which asks for votes from readers worldwide before deciding on the restructuring of its newsgroups list, including merging of low-volume groups, splitting of high-volume groups into many specialised groups, creating new groups, and even deleting groups. Once the voting process for a change concludes and the change action is to be carried out, it would be extremely tedious to send email to the hundreds of thousands of Usenet administrators and hope that they make the changes right, and answer their doubts if they get confused. It would be much better to have an automatic way to make the changes across all servers, of course with proper authorisation.

The solution to this does not lie in giving some central authority the ability to run an OS-level command of his choice on all the world's Usenet servers, because OS commands differ from OS to OS, and because few Usenet administrators would trust a stranger from another part of the world with OS level access. Therefore, the solution lay in defining a small set of common Usenet maintenance actions, and permitting only these actions to be triggered on all servers through the passing of special command messages, called control messages.

Control messages look like ordinary Usenet articles, more or less. They have an extra header line, with its value in a specific format, but they usually carry body text which looks like a normal human-written article. Here is a control message (a spurious one at that, but it'll do for now):

Xref: news.starcomsoftware.com control:814217
Path: news.starcomsoftware.com!linux594.dn.net!news.dn.hoopoo.com!
	feed-out.newsfeeds.com!newsfeeds.com!feed.newsfeeds.com!
	newsfeeds.com!news-spur1.maxwell.syr.edu!news.maxwell.syr.edu!
	newsfeed.icl.net!newsfeed.skycache.com!Cidera!newsfeed.gamma.ru!
	Gamma.RU!carrier.kiev.ua!goblin.nadrabank.kiev.ua!not-for-mail
From: tale@uunet.uu.net (David C Lawrence)
Newsgroups: news.groups,humanities.hipcrime
Subject: cmsg newgroup humanities.hipcrime
Control: newgroup humanities.hipcrime
Date: Sun, 18 Feb 2001 11:50:28 GMT
Organization: The Cabal
Lines: 20
Approved: tale@uunet.uu.net
Message-ID: <3afWYZTIR.G5YOC2@uunet.uu.net>
NNTP-Posting-Host: 203.145.147.67
X-Trace: goblin.nadrabank.kiev.ua 982528840 21455 203.145.147.67
         (18 Feb 2001 20:40:40 GMT)
X-Complaints-To: usenet@nadrabank.kiev.ua
NNTP-Posting-Date: 18 Feb 2001 20:40:40 GMT
X-No-Archive: Yes

humanities.hipcrime is an unmoderated newsgroup which passed its
vote for creation by 326:10 as reported in news.announce.newgroups
on 18 Feb 2001.

For your newsgroups file:
humanities.hipcrime	HipCrime for Humanity - you committed one now!

Anyone can create a newsgroup in the alt, biz, comp, earth,
humanities, misc, news, meow, rec, sci, soc, talk, us, or
any other Usenet hierarchy.  New newsgroup proposals may be
optionally discussed in news.groups. Please be sure that your
/usr/lib/news/control.ctl is configured correctly:

##	NEWGROUP MESSAGES
## honor them all and log in \${LOG}/newgroup.log
newgroup:*:alt.*|biz.*|comp.*|earth.*|humanities.*|misc.*|news.*|\
	meaw.*|rec.*|sci.*|soc.*|talk.*|us.*:doit=newgroup

##	RMGROUP MESSAGES
## drop them all and don't log
rmgroup:*:*:drop

Meow!
David C Lawrence

A control message must have a Control header. Besides, all control messages will have an Approved header, like messages posted to moderated newsgroups. The Control header actually specifies a command to run on the local server, and the parameter(s) to supply to it. The local Usenet server software is supposed to figure out its own way to get the task done. In this example, the command in the Control header is newgroup, which creates a new newsgroup. And its parameter is humanities.hipcrime, which gives the name of the newsgroup to create.

In C-News, the control message implementation works through separate shellscripts kept in a fixed directory, $NEWSBIN/ctl/, as a security measure; if the executable script isn't present there, the control message command will be ignored. The control message types supported are:

• checkgroups: control message to check whether the list of newsgroups in your active file are all correct as per a master list of newsgroups sent in the control message

• newgroup: control message to create a new newsgroup

• rmgroup: control message to delete a newsgroup and all articles in it

• sendsys: control message to cause an email response to be sent to the author with the sys file of your server in it. This results in a response storm of emails from all the Usenet servers in the world to the author. These responses allow the sender of the control message to analyse all the sys files of the world's Usenet servers and create the directed graph of Usenet newsfeeds. Why someone would want to do this is hard to guess, but the result is surely an awesome picture of one facet of networked human civilisation, like looking at a giant world map.

  Incidentally, there is no invasion of privacy here, because your server's sys file is supposed to be public information, if you take feeds from the public Usenet.

• version: control message which results in your Usenet software sending an email to the author of the message, containing the type and version of the Usenet news software you are using. This too is not an invasion of privacy, because this information is supposed to be public knowledge.

• The cancel message: the most frequently occurring type of control messages. They specify the message ID of an article, and result in the cancellation (deletion) of that article. If you post an article and regret it a moment later, your Usenet newsreader software usually allows you to ``cancel'' it by generating a cancel message.

The Usenet news software maintains a pseudo-newsgroup called control, where it files all control messages it receives. If you have an incoming newsfeed from the public Usenet, your server's control group will usually be full with thousands of cancel messages from trigger-happy fingers all over the world. Usenet news server software like C-News allows you to filter the incoming feed based on newsgroups, and will discard articles for groups they do not subscribe to. But since all servers have to receive and process control messages, they will all accept these cancel messages, though many of them may apply to articles which are not part of your highly-pruned subset of groups. C'est la vie.

Remember to set expiry for the control group to one day or even shorter, so that the junk can be cleaned out as rapidly as possible, just like the junk newsgroup.

The beauty of the control message architecture is that it integrates seamlessly into the newsfeed mechanism for automatic control of the network of servers. No separate channel of connection is needed for the control actions. And article replication automatically propagates control messages with human-readable articles, thus guaranteeing reach across heterogenous networks technologies.

What your Usenet server does on receiving a control message is governed by an authorisation file: $NEWSCTL/controlperms in the case of C-News and control.ctl in the case of INN, for instance. The security measures implemented by this module are further enhanced by the pgpcontrol package with its pgpverify script. Using pgpverify, your server can check that all control messages (except for article cancellation messages) are digitally signed by a trusted party using military-spec public key cryptography. Our integrated Usenet news software distribution includes integration with pgpverify.

3.1. A brief history of Usenet systems

Towards the end of this HOWTO, we have added some information about the history of Usenet server software by quoting sections from an earlier Usenet Periodic Posting. We consider this historical perspective, and the Usenix papers and other documents referred to in it, essential reading for any Usenet server administrator. Please see the section titled "Usenet software: a historical perspective>".

3.2. C-News and NNTPd

C-News was written by Henry Spencer and Geoff Collyer of the Department of Zoology, University of Toronto, almost entirely in shell and awk, as a replacement for an earlier system called B-News. The focus was on adding some extra features and a lot of performance. The first release was called Shellscript Release, which was deployed by a very large number of servers worldwide, as a natural upgrade to B-News. This version of C-News had upward compatibility with B-News meta-data, e.g. history files. This was the version of C-News which was initially rolled out in 1991 or so at the National Centre for Software Technology (NCST, http://www.ncst.ernet.in) and the Indian Institutes of Technology in India as part of the Indian educational and research network (ERNET). We received guidance from the NCST about Usenet news installation and management.

The Shellscript Release was soon followed by a re-write with a lot more C code, called Performance Release, and then a set of cleanup and component integration steps leading to the last release called the Cleanup Release. This Cleanup Release was patched many times by the authors, and the last one was CR.G (Cleanup Release revision G). The version of C-News discussed in this HOWTO is a set of small bug fixes on CR.G.

Since C-News came from shellscript-based antecedents, its architecture followed the set-of-programs style so typical of Unix, rather than large monolothic software systems traditional to some other OSs. All pieces had well-defined roles, and therefore could be easily replaced with other pieces as needed. This allowed easy adaptations and upgradations. This never affected performance, because key components which did a lot of work at high speed, e.g. newsrun, had been rewritten in C by that time. Even within the shellscripts, crucial components which handled binary data, e.g. a component called dbz to manipulate efficient on-disk hash arrays, were C programs with command-line interfaces, called from scripts.

C-News was born in a world with widely varying network line speeds, where bandwidth utilisation was a big issue and dialup links with UUCP file transfers was common. Therefore, it has strong support for batched feeds, specially with a variety of compression techniques and over a variety of fast and slow transport channels. And C-News virtually does not know the existence of TCP/IP, other than one or two tiny batch transport programs like viarsh. However, its design was so modular that there was absolutely no problem in plugging in NNTP functionality using a separate set of C programs without modifying a single line of C-News. This was done by a program suite called NNTP Reference Implementation, which we call NNTPd.

This software suite could work with B-News and C-News article repositories, and provided the full NNTP functionality. Since B-News died a gradual death, the combination of C-News and NNTPd became a freely redistributable, portable, modern, extensible, and high-performance software suite for Unix Usenet servers. Further refinements were added later, e.g. nov, the News Overview package and pgpverify, a public-key-based digital signature module to protect Usenet news servers against fraudulent control messages.

3.3. INN

INN is one of the two most widely used Usenet news server solutions. It was written by Rich Salz for Unix systems which have a socket API --- probably all Unix systems do, today.

INN has an architecture diametrically opposite to CNews. It is a monolithic program, which is started at bootup time, and keeps running till your server OS is shut down. This is like the way high performance HTTP servers are run in most cases, and allows INN to cache a lot of things in its memory, including message-IDs of recently posted messages, etc. This interesting architecture has been discussed in an interesting paper by the author, where he explains the problems of the older B-News and C-News systems that he tried to address. Anyone interested in Usenet software in general and INN in particular should study this paper.

INN addresses a Usenet news world which revolves around NNTP, though it has support for UUCP batches --- a fact that not many INN administrators seem to talk about. INN works faster than the CNews-NNTPd combination when processing multiple parallel incoming NNTP feeds. For multiple readers reading and posting news over NNTP, there is no difference between the efficiency of INN and NNTPd. Section 5.7> discusses the efficiency issues of INN over the earlier C-News architecture, based on Rich Salz' paper and our analyses of usage patterns.

INN's architecture has inspired a lot of high-performance Usenet news software, including a lot of commercial systems which address the ``carrier class'' market. That is the market for which the INN architecture has clear advantages over C-News.

3.4. Leafnode

This is an interesting software system, to set up a ``small'' Usenet news server on one computer which only receives newsfeeds but does not have the headache of sending out bulk feeds to other sites, i.e. it is a ``leaf node'' in the newsfeed flow diagram. According to its homepage (www.leafnode.org), ``Leafnode is a USENET software package designed for small sites running any flavour of Unix, with a few tens of readers and only a slow link to the net. [...] The current version is 1.9.24.''

This software is a sort of combination of article repository and NNTP news server, and receives articles, digests and stores them on the local hard disks, expires them periodically, and serves them to an NNTP reader. It is claimed that it is simple to manage and is ideal for installation on a desktop-class Unix or Linux box, since it does not take up much resources.

Leafnode is based on an appealing idea, but we find no problem using C-News and NNTPd on a desktop-class box. Its resource consumption is somewhat proportional to the volume of articles you want it to process, and the number of groups you'll want to retain for a small team of users will be easily handled by C-News on a desktop-class computer. An office of a hundred users can easily use C-News and NNTPd on a desktop computer running Linux, with 64 MBytes of RAM, IDE drives, and sufficient disk space. Of course, ease of configuration and management is dependent on familiarity, and we are more familiar with C-News than with Leafnode. We hope this HOWTO will help you in that direction.

There is, however, one area in which Leafnode is far easier to administer than INN or C-News. Leafnode constantly monitors the actual usage of the newsgroups it carries, based on readership statistics of its NNTP readers. If a particular newsgroup is not read at all by any user for a week, then Leafnode will delete all articles in that newsgroup, free up disk space, and stop fetching new articles for it. If it finds that a previously abandoned newsgroup is now again receiving attention, even from one user, then it'll fetch all articles for that group from its upstream server the next time it connects. This self-tuning feature of Leafnode is really an excellent advantage which makes a Leafnode site easier to manage, specially for small setups with bandwidth and disk space constraints.

The Leafnode Website gives a lot of details in an easily understood format.

TO BE EXTENDED AND CORRECTED.

3.5. Suck

Suck is a program which lets you pull out an NNTP feed from an NNTP server and file it locally. It does not contain any article repository management software, expecting you to do it using some other software system, e.g. C-News or INN. It can create batchfiles which can be fed to C-News, for instance. (Well, to be fair, Suck does have an option to store the fetched articles in a spool directory tree very much like what is used by C-News or INN in their article area, with one file per article. You can later read this raw message spool area using a mail client which supports the msgdir file layout for mail folders, like MH, perhaps. We don't find this option useful if you're running Suck on a Usenet server.) Suck finally boils down to a single command-line program which is invoked periodically, typically from cron. It has a zillion command-line options which are confusing at first, but later show how mature and finely tunable the software is.

If you need an NNTP pull feed, then we know of no better programs than Suck for the job. The nntpxfer program which forms part of the NNTPd package also implements an NNTP pull feed, for instance, but does not have one-tenth of the flexibility and fine-tuning of Suck. One of the banes of the NNTP pull feed is connection timeouts; Suck allows a lot of special tuning to handle this problem. If we had to set up a Usenet server with an NNTP pull feed, we'd use Suck right away.

TO BE EXTENDED AND CORRECTED.

3.6. Carrier class software

Carrier-class servers are expected to handle a complete feed of all articles in all newsgroups, including a lot of groups which have what we call a ``high noise-to-signal ratio.'' They do not have the luxury of choosing a ``useful'' subset like administrators of internal corporate Usenet servers do. Secondly, carrier-class servers are expected to turn articles around very fast, i.e. they are expected to have very low latency from the moment they receive an article to the time they retransmit it by NNTP to downstream servers. Third, they are supposed to provide very high availability, like other ``carrier class'' services. This usually means that they have parallel arrays of computers in load sharing configurations. And fourth, they usually do not cater to retail connections for reading and posting articles by human users. Usenet news carriers usually reserve separate computers to handle retail connections.

Thus, carrier-class servers do not need to maintain a repository of articles; they only need to focus on super-efficient real-time re-transmission. These highly specialised servers have software which receive an article over NNTP, parse it, and immediately re-queue it for outward transmission to dozens or hundreds of other servers. And since they work at these high throughputs, their downstream servers are also expected to be live on the Internet round the clock to receive incoming NNTP connections, or be prepared to lose articles. Therefore, there's no batching or long queueing needed, and C-News-style batching in fact is totally inapplicable.

Therefore, these carrier-class Usenet servers are more like packet routers than servers with repositories. They are referred to nowadays as NNTP routers or news routers.

It can be seen why batch-oriented repository management software like C-News is a total anachronism here, and why they need an NNTP-oriented, online, real-time design. The INN antecedents of some of these systems is therefore natural. We would love to hear from any Linux HOWTO reader whose Usenet server requirements include carrier-class behaviour.

We are aware of only one freely redistributable NNTP router: NNTPRelay (see http://nntprelay.maxwell.syr.edu/); this software runs on NT. There is no reason why such services cannot run off Linux servers, even Intel Linux, provided you have fast network links and arrays of servers. Linux as an OS platform is not an issue here.

TO BE EXTENDED AND CORRECTED.

4.1. Getting the sources and stuff

4.2. Compiling and installing

For installing, first make sure you have an entry for a user called news in your /etc/password file. This is setting the news-database owner to news. Now download the source from us and untar it in the home directory of news. This creates two main directories viz. c-news and nntp. To install and compile, run the script build.sh as root in the directory that contains the script. It is important that the script run as root as it sets ownerships, installs and compiles the source as user news. This is a one-step process that puts in place both the C-News and the NNTP software, setting correct permissions and paths. Following is a brief description of what build.sh does:

• Checks for the OS platform and exits if it is not Linux.

• Again, exits if you are not running as root.

• Looks for and exits if cannot find the above two directories.

• Compiles C-News and performs regression tests if the compilation was successfull. Sends out a warning to read the error file make.out.r and to fix 'em. Compilation erros are written to a file called make.out.

• Performs the above operation in the nntp directory, too.

• Checks for the presence of the three key directories: $NEWSARTS - (/var/spool/news) that houses the artciles, $NEWSCTL -(/var/lib/news) that contain configuration, log and status files and $NEWSBIN - (/usr/lib/newsbin) that contain binaries and executables for the working of the Usenet News system. Tries to create them if non-existent and exits if it results in failure.

• Changes the ownership of these directories to news.news. This is important since the entire Usenet News System runs as user news. It will not function properly as any other user.

• Then starts the installation process of C News. It runs make install to install binaries at the right locations; make setup to set the correct paths and umask, create directories for newsgroups, determine who will receive reports; make ui to set up inews and injnews and make readpostcheck to use readnews, postnews and checknews scripts provided by C News. The errors, if any are to be found in the respective make.out files. e.g. make.setup will write errors to make.out.setup

• Newsspool, which queues incoming batches in $NEWSARTS/in.coming directory should run as set-userid and set-groupid. This is done.

• A softlink is made to /var/lib/news from /usr/lib/news.

• The NNTP software is installed.

• Sets up the manpages for C News and makes it world readable. The NNTP manpages get installed when the software is installed. Compiles the C News documentation guide.ps and makes it readable and available in /usr/doc/packages/news or /usr/doc/news.

• Checks for the PGP binary and asks the administrator to get it, if not found.

4.3. Configuring the system: What and how to configure files?

Once installed, you have to now configure the system to accept feeds and batch them for your neighbours. You will have to do the following:

• nntpd: Copy the compiled nntpd into a directory where executables are kept and activate it. It runs on port 119 as a daemon through inetd unless you have compiled it as stand-alone. An entry in the /etc/services file for nntp would look like this:

  nntp 119/tcp \# Network News Transfer Protocol

  An entry in the inetd.conf file will be:

  nntp stream tcp nowait news path-to-tcpd path-to-nntpd

  The last two fields in the inetd.conf file are paths to binaries of the tcp and the nntp daemon respectively.

• Configuring control files: There are plenty of control files in $NEWSCTL that will need to be configured before you can start using the news system. The files mentioned here are also discussed in the first section of the section titled "Components of a running system>". These control files are dealt in detail in the following below.

  • sys: One line per system/NDN listing all the newsgroup hierarchies each system subscribes to. Each line is prefixed with the system name and the one beginning with

    ME:

    indicates what your server is willing to receive. Following are typical entries that go into this file:

    ME:comp,news,misc,netscape

    This line indicates what newsgroups your server has subscribed to.

    server/server.starcomsoftware.com:all,!general/all:f

    This is a list of newsgroups your server will pass on to your NDN. The newsgroups specified should be a comma separated list and the entire line should contain no spaces. The f flag indicates that the newsgroup name and the article number alongwith its size will make up one entry in the togo file in the $NEWSARTS/out.going directory.

  • explist: This file has entries indicating which articles expire and when and whether they have to be archived. The order in which the newsgroups are listed is important. An example follows:

    comp.lang.java.3d x 60 /var/spool/news/Archive

    This means that the articles of comp.lang.java expire after 60 days and shall be archived in the directory mentioned in the fourth field. Archiving is an option. The second field indicates that this line applies to both moderated and unmoderted newsgroups. m would specify moderated and u would specify unmoderated groups. If you want to specify an extremely large no. as the expiry period you can use the keyword "never".

  • batchparms: sendbatches is a program that administers batched transmission of news articles to other sites. To do this it consults the batchparms file. Each line in the file specifies the behaviour for each of your NDN mentioned in the sys file. There are five fields for each site to be specified.

    server u 100000 100 batcher | gzip -9 | viauux -d gunzip

    The first field is the site name which matches the entry in the sys file and has a corresponding directory in $NEWSARTS/out.going by that name.

    The second field is the class of the site,u for UUCP and n for NNTP feeds. A "!" in this field means that batching for this site has been disabled.

    The third field is the size of batches to be prepared in bytes.

    The fourth field is the maximum length of the output queue for transmission to that site.

    The fifth field is the command line to be used to build, compress and transmit batches to that site. The contents of the togo file are made available on standard input.

  • controlperm: This file controls how the news system responds to control messages. Each line consists of 4-5 fields separated by white space. Control messages has been discussed in "Section 2.4>".

    comp,sci tale@uunet.uu.net nrc pv news.announce.newsgroups

    The first field is a newsgroup pattern to which the line applies.

    The second field is either the keyword "any" or an e-mail address. The latter specifies that the line applies to control messages from only that author.

    The third field is a set of opcode letters indicating what control operations need to be performed on messages emanating from the e-mail address mentioned in the second field. n stands for creating a newgroup, r stands for deleting a newsgroup and c stands for checkgroup.

    The fourth field is a set of flag letters indicating how to respond to a control message that meets all the applicability tests:

    y Do it. n Don't do it. v Report it and include the entire control message in the report. q Don't report it. p Do it iff the control message carries a valid PGP signature.

    Exactly one of y, n or p must be present.

    The fifth field, which is optional, will be used if the fourth field contains a p. It must contain the PGP key ID of the public key to be used for signature verification.

  • mailpaths: This file describes how to reach the moderators of various hierarchies of newsgroups by mail. Each line consists of two fields: a news group pattern and an e-mail address. The first line whose group pattern matches the newsgroup is used. As an example:

    comp.lang.java.3d somebody@mydomain.com all %s@moderators.uu.net

    In the second example, the %s gets replaced with the groupname and all dots appearing in the newsgroup name are substituted with dashes.

  • Miscellaneous files: The other files to be modified are:

    • mailname: Contains the Internet domain name of the news system. Consider getting one if you don't have it.

    • organization: Contains the default value for the Organization: header for postings originating locally.

    • whoami: Contains the name of the news system. This is the site name used in the Path: headers and hence should concur with the names your neighbours use in their sys files.

  • active file: This file specifies one line for each newsgroup (not just the hierarchy) to be found on your news system. You will have to get the most recent copy of the active file from ftp://ftp.isc.org/usenet/CONFIG/active and prune it to delete newsgroups that you have not subscribed to. Run the script addgroup for each newsgroup in this file which will create relevant directories in the $NEWSARTS area. The addgroup script takes two paramters: the newsgroup name being created and a flag. The flag can be any one of the following:

    y local postings are allowed n no local postings, only remote ones m postings to this group must be approved by the moderator j articles in this group are only passed and not kept x posting to this newsgroup is disallowed =foo.bar articles are locally filed in "foo.bar" group

    An entry in this file looks like this:

    comp.lang.java.3d 0000003716 01346 m

    The first field is the name of the newsgroup. The second field is the highest article number that has been used in that newsgroup. The third field is the lowest article number in the group. The fourth field is a flag as explained above.

  • newsgroups file: This contains a one-line description of each newsgroup to be found in the active file. You will have to get the most recent file from ftp://ftp.isc.org/usenet/CONFIG/newsgroups and prune it to remove unwanted information. As an example:

    comp.lang.java.3d 3D Graphics APIs for the Java language

  • Aliases: These aliases are required for trouble reporting. Once the system is in place and scripts are run, anomalies/problems are reported to addresses in the /etc/aliases file. These entries include email addresses for newsmaster, newscrisis, news, usenet, newsmap. They should ideally point to an email address that will be accessed at regularly. Arrange the emails for newsmap to be discarded to minimize the effect of sendsys bombing by practical jokers.

  • Cron jobs: Certain scripts like newsrun that picks up incoming batches and maintenance scripts, should run through news-database owner's cron which is news. The cron entries ideally will be for the following: A more detailed report can be found in "Section 9.4>"

    1. newsrun: This script processes incoming batches of article. Run this as frequently as you want them to get digested.

    2. sendbatches: This script transmit batches to the NDNs. Set the frequency according to your requirements.

    3. newsdaily: This should be run ideally once a day since it reports errors and anomalies in the news system.

    4. newswatch: This looks for errors/anomalies at a more detailed level and hence should be run atleast once every hour

    5. doexpire: This script expires old articles as determined by the explist file. Run this once a day.

  • newslog: Make an entry in the system's syslog.conf file for logging messages spewed out by nntpd in newslog . It should be located in $NEWSCTL. The entry will look like this:

    news.debug -/var/lib/news/newslog

  • Newsboot: Have this run (as news the news-database owner) when the system boots to clear out debris left around by crashes.

  • Add a Usenet mailer in sendmail: The mail2news program provided as part of the source code is a handy tool to send an e-mail to a newsgroup which gets digested as an article. You will have to add the following ruleset and mailer definition in your sendmail.cf file:

    • Under SParse1, add the following:

      R$+ . USENET < @ $=w . > $#usenet $: $1

    • Under mailer definitions, define the mailer Usenet as:

      MUsenet P=/usr/lib/newsbin/mail2news/m2nmailer, F=lsDFMmn, S=10, R=0, M=2000000, T=X-Usenet/X-Usenet/X-Unix, A=m2nmailer $u

    In order to send a mail to a newsgroup you will now have to suffix the newsgroup name with usenet i.e. your To: header will look like this:

    To: misc.test.usenet@yourdomain.

    The mailer definition of usenet will intercept this mail and post it to the respective newsgroup, in this case, misc.test

  This, more or less, completes the configuration part.

4.4. Testing the system

To locally test the system, follow the steps given below:

• post an article: Create a local newsgroup

  cnewsdo addgroup mysite.test y

  and using postnews post an article to it.

• Has it arrived in $NEWSARTS/in.coming?: The article should show up in the directory mentioned. Note the nomenclature of the article.

• When newsrun runs: When newsrun runs from cron , the article disappears from in.coming directory and appears in $NEWSARTS/mysite/test. Look how the newsgroup, active, log and history (not the errorlog) files and .overview file in $NEWSARTS/mysite/test reflect the digestion of the file into the news system.

• reading the article: Try to read the article through readnews or any news client. If you are able to, then you have set most everything right.

4.5. pgpverify and controlperms

As mentioned in "Section 2.4>", it becomes necessary to authenticate control messages to protect yourself from being attacked by pranksters. For this, you will have to configure the $NEWSCTL/controlperm file to declare whose control messages you are willing to honour and for what newsgroups alongwith their public key ID. The controlperm manpage shall give you details on the format.

This will work only in association with pgpverify which verifies the Usenet control messages that have been signed using the signcontrol process. The script can be found at ftp://ftp.isc.org/pub/pgpcontrol/pgpverify. pgpverify internally uses the PGP binary which will have to be made available in the default executables directory. If you wish to send control messages for your local news system, you will have to digitally sign them using the above mentioned signcontrol program which is available at ftp://ftp.isc.org/pub/pgpcontrol/signcontrol. You will also have to configure the signcontrol program accordingly.

4.6. Feeding off an upstream neighbour

For external feeds, commercial customers will have to buy them from a regular News Provider like dejanews.com or newsfeeds.com. You will have to specify to them what hierarchies you want and decide on the mode of transmission, i.e. UUCP or NNTP, based on your requirements. Once that is done, you will have to ask them to initiate feeds, and check $NEWSARTS/in.coming directory to see if feeds are coming in.

If your organisation belongs to the academic community or is otherwise lucky enough to have an NDN server somewhere which is willing to provide you a free newsfeed, then the payment issue goes out of the picture, but the rest of the technical requirements remain the same.

One problem with incoming NNTP feeds is that it is far easier to use (relatively) efficient NNTP inflows if you have a server with a permanent Internet connection and a fixed IP address. If you are a small office with a dialup Internet connection, this may not be possible. In that case, the only way to get incoming newsfeeds by NNTP may be by using a highly inefficient pull feed.

4.7. Configuring outgoing feeds

If you are a leaf node, you will only have to send feeds back to your news provider for your postings in public newsgroups to propagate to the outside world. To enable this, you need one line in the sys and batchparms files and one directory in $NEWSARTS/out.going. If you are willing to transmit articles to your neighbouring sites, you will have to configure sys and batchparms with more entries. The number of directories in $NEWSARTS/out.going shall increase, too. Refer to first two sections of the chapter titled "Components of a running system>"for a better understanding of outgoing feeds. Again, you will have to determine how you wish to transmit the feed: UUCP or NNTP.

5.1. Getting the source

INN is maintained and archived by the ISC (Internet Software Consortium, www.isc.org) since 1996, and the INN homepage is at http://www.isc.org/products/INN/. The latest release of INN as of the time of this writing is INN v2.3.3, released 7 May 2002. The full sources can be downloaded from ftp://ftp.isc.org/isc/inn/inn-2.3.3.tar.gz

5.2. Compiling and installing

TO BE EXTENDED LATER.

5.3. Configuring the system

TO BE ADDED LATER.

5.4. Setting up pgpverify

TO BE ADDED LATER.

5.5. Feeding off an upstream neighbour

TO BE ADDED LATER.

5.6. Setting up outgoing feeds

TO BE ADDED LATER.

5.7. Efficiency issues and advantages

TO BE ADDED LATER.

6.1. Feeding Usenet news to email

In CNews, this is trivially done by adding one line to the sys file, defining a new outgoing feed listing all the relevant groups and distributions, and specifying the commandline to be executed which is supposed to send out the outgoing message to that ``feed.'' This command, in our case, should be a mail-sending program, e.g. /bin/mail user@somewhere.com. This is often adequate to get the job done. We are sure almost every Usenet news software system will have an equally easy way of piping the feed of a newsgroup to an email address.

6.2. Feeding email to news: the mail2news gateway

With our Usenet software sources has been integrated a set of scripts which we have been using for at least five years internally. This set of scripts is called mail2news. It contains one shellscript called mail2news, which takes an email message from stdin, processes it, and feeds the processed version to inews, the stdin-based news article injection utility of C-News. The inews utility accepts a new article post in its stdin and queues it for digestion by newsrun whenever it runs next.

To use mail2news, we assume you are using Sendmail to process incoming email. Our instructions can easily be modified to adapt to any Mail Transport Agent (MTA) of your choice. You will have to configure Sendmail or any other MTA to redirect incoming mails for the gateway to a program called m2nmailer, a Perlscript which accepts the incoming message in its standard input and a list of newsgroup names, space separated, on its command line. Sendmail can be easily configured to trigger m2nmailer this way by defining a new mailer in sendmail.cf, and directing all incoming emails meant for the Usenet news system to this mailer. Once you set up the appropriate rulesets for Sendmail, it automatically triggers m2nmailer each time an incoming email comes for the mail2news gateway.

The precise configuration changes to Sendmail have already been specified in the chapter titled ``Setting up C-News + NNTPd.''

6.3. Using GNU Mailman as an email-NNTP gateway

TO BE ADDED LATER

7.1. Intrusion threats

Here we discuss the vulnerabilities which will allow an intruder to ``gain control'' of your Usenet server, or ``bring it down,'' either of which may be irritating, embarassing, or downright disastrous for your business or occupation.

7.2. Vulnerabilities unique to the Usenet service

There are certain security precautions that a Usenet server administrator has to take to ensure that her servers are not swamped by irritating junk or configured out of shape by spurious control messages. These vulnerabilities do not allow an intruder to run her software on your servers, but allows her to mess up your server, causing you to lose a precious weekend (or week) straightening out the mess.

8.1. Host-based access control

TO BE ADDED LATER

8.2. User authentication and authorisation

9.1. /var/lib/news: the CNews control area

This directory is more popularly known as $NEWSCTL. It contains configuration, log and status files. There are no articles or binaries kept here. Let's see what some of the files are meant for. Control files are dealt in slightly greater detail in "Section 4.3>"

• sys: One line per system/NDN listing all the newsgroup hierarchies each system subscribes to. Each line is prefixed with the system name and the one beginning with ME: indicates what we are going to receive. Look up manpage of newssys.

• explist: This file has entries indicating articles of which newsgroup expire and when and if they have to be archived. The order in which the newsgroups are listed is important. See manpage of expire for file format.

• batchparms: Details of how to feed other sites/NDN, like the size of batches, the mode of transmission (UUCP/NNTP) are specified here. manpage to refer: newsbatch.

• controlperm: If you wish to authenticate a control message before any action is taken on it, you must enter authentication-related information here. The controlperm manpage will list all the fields in detail.

• mailpaths: It features the e-mail address of the moderator for each newsgroup who is responsible for approving/disapproving articles posted to moderated newsgroups. The sample mailpaths file in the tar will give you an idea of how entries are made.

• nntp_access/user_access: These files contain entries of servernames and usernames on whom restrictions will apply when accessing newsgroups. Again, the sample file in the tarball shall explain the format of the file.

• log, errlog: These are log files that keep growing large with each batch that is received. The log file has one entry per article telling you if it has been accepted by your news server or rejected. To understand the format of this file, refer to Chapter 2.2 of the CNews guide. Errors, if any, while digesting the articles are logged in errlog. These log files have to be rolled as the files hog a lot of disk space.

• nntplog: This file logs information of the nntpd giving details of when a connection was established/broken and what commands were issued. This file needs to be configured in syslog syslogd should be running.

• active: This file has one line per newsgroup to be found in your news server. Besides other things, it tells you how many articles are currently present in each newsgroup. It is updated when each batch is digested or when articles are expired. The active manpage will furnish more details about other paramaters.

• history: This file, again, contains one line per article, mapping message-id to newsgroup name and also giving its associated article number in that newsgroup. It is updated each time a feed is digested and when doexpire is run. Plays a key role in loop-detection and serves as an article database. Read manpage of newsdb, doexpire for the file format

• newsgroups: It has a one-line description for each newsgroup explaining what kind of posts go into each of them. Ideally speaking, it should cover all the newsgroups found in the active file.

• Miscellaneous files: Files like mailname, organisation, whoami contain information required for forming some of the headers of an article. The contents of mailname form the From: header and that of organisation form the Organisation: header. whoami contains the name of the news system. Refer to chapter 2.1 of guide.ps for a detailed list of files in the $NEWSCTL area. Read RFC 1036 for description of article headers .

9.2. /var/spool/news: the article repository

This is also known as the $NEWSARTS or $NEWSSPOOL directory. This is where the articles reside on your disk. No binaries or control files should belong here. Enough space should be allocated to this directory as the number of articles keep increasing with each batch that is digested. An explanation of the following sub-directories will give you an overview of this directory:

• in.coming: Feeds/batches/articles from NDNs on their arrival and before being processed reside in this directory. After processing, they appear in $NEWSARTS or in its bad sub-directory if there were errors.

• out.going: This directory contains batches/feeds to be sent to your NDNs i.e. feeds to be pushed to your neighbouring sites reside here before they are transmitted. It contains one sub-directory per NDN mentioned in the sys file. These sub-directories contain files called togo which contain information about the article like the message-id or the article number that is queued for transmission.

• >newsgroup directories: For each newsgroup hierarchy that the news server has subscribed to, a directory is created under $NEWSARTS. Further sub-directories are created under the parent to hold articles of specific newsgroups. For instance, for a newsgroup like comp.music.compose, the parent directory comp will appear in $NEWSARTS and a sub-directory called music will be created under comp. The music sub-directory shall contain a further sub-directory called compose and all articles of comp.music.compose shall reside here. In effect, article 242 of newsgroup comp.music.compose shall map to file $NEWSARTS/comp/music/compose/242.

• control: The control directory houses only the control messages that have been received by this site. The control messages could be an