Venti

For the Roman gods of the winds, see Anemoi.

Venti is a network storage system that permanently stores data blocks. A 160-bit SHA-1 hash of the data (called score by Venti) acts as the address of the data. This enforces a write-once policy since no other data block can be found with the same address. The addresses of multiple writes of the same data are identical, so duplicate data is easily identified and the data block is stored only once. Data blocks cannot be removed, making it ideal for permanent or backup storage. Venti is typically used with Fossil to provide a file system with permanent snapshots.

History

Venti was designed and implemented by Sean Quinlan and Sean Dorward at Bell Labs. It appeared in the Plan 9 distribution in 2002. Development has been continued by Russ Cox who has reimplemented most of the server, written a library for creating datastructures (files, directories and meta-data) to store in Venti and implemented optimizations. This rewrite of Venti is now distributed in Plan 9 from User Space and will make it into the Plan 9 distribution when deemed stable for Plan 9.

Details

Venti is a user space daemon. Clients connect to Venti over TCP and communicate using a simple RPC-protocol. The most important messages of the protocol are listed below. Note that there is no message to delete an address or modify data at a given address.

read(score, type), returns the data identified by score and type
write(data, type), stores data at the address calculated by SHA-1 hashing data, combined with type.

The data block stored by Venti must be greater than 512 bytes in length and smaller than 56 kilobytes. So, if a Venti user/client wants to store larger data blocks, it has to make a datastructure (which can be stored in Venti). For example, Fossil uses hash trees to store large files. Venti itself is not concerned with the contents of a data block; it does however store the type of a data block.

The design of Venti has some interesting consequences:

Since writes are permanent, the file system is append-only (which allows for a simple implementation with lower chance of data-destroying bugs); no file system fragmentation occurs.
Clients can verify the correctness of the server: the score of the returned data should be the same as the address requested. Venti cannot lie since SHA-1 is a cryptographically secure hash; no data block can be found with the same address.
Data cannot be overwritten. If an address is already present, the data is already present.
There is little need for user authentication: Data can only be read if the score is known. The only potential problem is a user filling up the disks.
Data can be compressed without making the disk structure complicated.

The data blocks are stored on hard drives. The disks making up the available storage, typically a RAID 5 configuration, is called the data log. This data log is split up in smaller pieces called arenas, which are sized so they can be written to other media such as CD/DVD or magnetic tape. Another set of hard drives is used for the index, which maps scores to addresses in the data log. The data structure used for the index is a hash table with fixed-sized buckets. Venti relies on the scores to be randomly distributed so buckets do not fill up. Since each lookup costs one disk seek time, an index usually consists of multiple small SCSI hard drives with low access time.

External links

Venti: a new approach to archival storage, paper describing Venti
New Venti manual page (overview), section 1 and 8 venti manual pages also exist
Old Venti manual page, currently in use on Plan 9.
Old Venti utilities manual page, currently in use on Plan 9.

In Greek mythology, the Anemoi (in Greek, Άνεμοι — "winds") were wind gods who were each ascribed a cardinal direction, from which their respective winds came, and were each associated with various seasons and weather conditions.
..... Click the link for more information.

SHA hash functions are five cryptographic hash functions designed by the National Security Agency (NSA) and published by the NIST as a U.S. Federal Information Processing Standard. SHA stands for Secure Hash Algorithm.
..... Click the link for more information.

Fossil is the default file system in Plan 9 from Bell Labs. It serves the network protocol 9P and runs as a user space daemon, like most Plan 9 file servers. Fossil is different from most other file systems due to its snapshot/archival feature.
..... Click the link for more information.

Bell Laboratories (also known as Bell Labs and formerly known as AT&T Bell Laboratories and Bell Telephone Laboratories) is part of the research and development organization of Alcatel-Lucent and previously the United States Bell System.
..... Click the link for more information.

Plan 9 from Bell Labs is a distributed operating system, primarily used as a research vehicle. It was developed as the research successor to Unix by the Computing Sciences Research Center at Bell Labs between the mid-1980s and 2002.
..... Click the link for more information.

Plan 9 from User Space (or plan9port or p9p) is a port of many Plan 9 from Bell Labs libraries and applications to Unix-like operating systems. Currently it has been tested on a variety of operating systems including: Linux, Mac OS X, FreeBSD, NetBSD, OpenBSD,
..... Click the link for more information.

kernel space and user space. Kernel space is strictly reserved for running the kernel, device drivers and kernel extensions. In most operating systems, kernel memory is never swapped out to disk.
..... Click the link for more information.

daemon (IPA pronunciation: /'dimən/ or /'deɪmən/^[1]
..... Click the link for more information.

The Transmission Control Protocol (TCP) is one of the core protocols of the Internet protocol suite. TCP provides reliable, in-order delivery of a stream of bytes, making it suitable for applications like file transfer and e-mail.
..... Click the link for more information.

Remote procedure call (RPC) is a technology that allows a computer program to cause a subroutine or procedure to execute in another address space (commonly on another computer on a shared network) without the programmer explicitly coding the details for this remote
..... Click the link for more information.

Hash trees or Merkle trees are a type of data structure which contains a tree of summary information about a larger piece of data – for instance a file – used to verify its contents.
..... Click the link for more information.

Fragmentation is a phenomenon that leads to inefficiency in many forms of computer storage. There are three different but related uses of the term: external fragmentation, internal fragmentation, and data fragmentation.
..... Click the link for more information.

Hard disk drive

An IBM hard disk drive with the metal cover removed. The platters are highly reflective.
Date Invented: September 13 1956
Invented By: An IBM team led by Reynold Johnson
Connects to:
..... Click the link for more information.

Compact Disc

The closely spaced tracks on the readable surface of a Compact Disc cause light to diffract into a full visible colour spectrum
Media type: Optical disc
Encoding: Various
Capacity: Typically up to 700 MB
..... Click the link for more information.

DVD

Media type: Optical disc
Capacity: 4.7 GB (single layer), 8.5 GB (dual layer)
Usage: Data storage, audio, video, games

Optical disc authoring

Optical disc
Optical disc image
Recorder hardware
Authoring software

..... Click the link for more information.

Magnetic tape is a medium for magnetic recording generally consisting of a thin magnetizable coating on a long and narrow strip of plastic. Nearly all recording tape is of this type, whether used for recording audio or video or for computer data storage.
..... Click the link for more information.

In computer science, a hash table, or a hash map, is a data structure that associates keys with values. The primary operation it supports efficiently is a lookup: given a key (e.g. a person's name), find the corresponding value (e.g. that person's telephone number).
..... Click the link for more information.

In computing, the term bucket can have several meanings. It is used both as a live metaphor, and as a generally accepted technical term in some specialised areas. A bucket is most commonly a type of data buffer or a type of document.
..... Click the link for more information.

Seek time is one of the several delays associated with reading or writing data on a computer's disk drive, and somewhat similar for CD or DVD drives. The others are rotational delay and transfer time.
..... Click the link for more information.

SCSI (Small Computer System Interface) is a set of standards for physically connecting and transferring data between computers and peripheral devices. The SCSI standards define commands, protocols, and electrical and optical interfaces.
..... Click the link for more information.

Disk time is the time delay or latency between a request for clock to an electronic system, and the access being granted or the requested data returned.

In a telecommunications system, access time is the delay between the start of an access attempt and successful access.

..... Click the link for more information.

History

Details

See also

External links