Ano-Tech Computers

VFTPD

Virtual FTP Daemon is an FTP server written in Perl designed to control any physical media changer adhering to the SCSI2 SMC standard. In other words: A Jukebox FTP server.

During startup, VFTPD will query the changer unit to determine the number of storage slots, transfer elements (drives), media transfer elements (elevators, robotic arms, conveyors or whatever mechanism is used for transporting media) and the presence of a mail slot (used for media import/export). The FTP server will then manage these resources as efficiently as possible.

Virtual file system

To minimize media changes, a complete list of files and directories is kept in an SQL database for fast and efficient access. This means the physical medium need only be accessed when a user actually wants to download a file. In addition, this virtualization makes it possible for the FTP administrator to rename, delete and even move files and directories around on the server!

Consider the following physical layout:

Medium 1
  Project A
  Project D
  Project E
Medium 2
  Project B
  Project C

A user should not have to worry about physical media. The FTP administrator need only ren "Medium *" "Projects" and we get a much nicer file layout:

Projects
  Project A
  Project B
  Project C
  Project D
  Project E

The physical location of each file and directory remains unchanged, the SQL database contains both the virtual and physical information allowing VFTPD to present the virtual data structure and access the physical one.

To illustrate further, and perhaps give you some ideas, here is the current root directory on ftp://jukebox.atc.no:

Apps
Docs
Drivers
Games
Movies
MP3

In december 2008 this server contained 448250 files, 26055 directories and 406 physical media. Not at all impressive if you compare with a modern hard drive, but there are two huge advantages: Power consumption and reliability. These files are distributed across media that will theoretically last 100 years, without power. Even if that's not true, even if 75% of the media fail and the jukebox itself falls apart, the remaining 25% of the media can still be retrieved without any difficulties as long as you can get your hands on a working CD-ROM player.

Media import/export

All server management, including media import and export, is done via FTP commands. Building on the example above, here's how we would add another medium containing project data:

FTP command	Commentary
SITE GOTO MAILSLOT	The media transfer element moves to the import/export position, allowing the administrator to insert the new medium. We are now in manual control of the media transfer element, VFTPD is now unable to serve other media requests.
SITE NEXT	VFTPD scans the storage slots to find the next free one, in this example number 3
SITE GOTO 3	The media transfer element moves to storage slot number 3
SITE UNLOAD	Unloads the medium into storage slot number 3. This is implemented as a separate command because if we did both in a single command and it failed, it would be harder to tell if the MOVE or UNLOAD operation caused the problem.
SITE RELEASE	This tells VFTPD to resume automatic control of the media transfer element
SITE INDEX 3	Request that VFTPD fetch the medium in storage slot 3, insert it into any available drive, scan the files/directories and update the SQL database. The medium contents will be inserted under a directory named "3"
REN 3 Projects	Rename the directory "3" to "Projects". If "Projects" already exists, contents will be merged as demonstrated above.
MKDIR "My files"	Create a new virtual directory. (Hint: CHDIR into this directory and use SITE MOVE)
SITE MOVE /4/File	Move the specified physical file into the current working directory, as set using the CHDIR command. This unlocks the real power of the Jukebox VFTPD.
SITE MAP /Projects	Find the physical media slots serving a specific file or directory.
SITE PEEK 3	List the virtual root directory of a physical media slot.
SITE NUKE 3	The opposite of SITE INDEX; delete all knowledge about a physical medium slot, usually due to failure or removal of a medium.

User authentication

VFTPD is designed to be used in an existing server environment and currently uses POP3 authentication to verify username/password pairs. Changing this to use, say PAM authentication, would be fairly simple.

Live demo

Our internal jukebox FTP server can be accessed here: ftp://jukebox.atc.no
We are using a Pioneer DRM5004X jukebox with 500 slots and two CD-ROM drives.

Prerequisites

VFTPD requires Perl 5 and the Device::SCSI module from CPAN.

License / Availability

This software was developed for internal use by Ano-Tech Computers and is not yet released to the public. If, however, you are interested in testing it then please contact me via email: floyd@atc.no.

Not yet released... is this project dead?

The first working version was put together in the summer of 2005. It's now december 2008 and I still keep tweaking the code and adding features since I use this server on a regular basis for archiving old games, programs, drivers, documentation, MP3 music, family photos etc. You know, stuff that I really don't want to store on a hard drive since they all crash eventually.

Those CDs are just sitting there, neatly stored, patiently waiting to be accessed. Every single one is indexed and organized so any single file can be retrieved in about 30 seconds. Can you honestly say the same thing about yours? :-)