mhVTL is an open source Virtual Tape Library which emulates standaard tape libraries. This can be usefull when testing backup scenarios or tape handling.

Here is how I set it up  on CentOS 6.5.

Virtual Tape Library consists of several components.

  • LLD – A low level driver implemented as a kernel module – mhvtl
  • Target devices – Daemons vtltape(1) and vtllibrary(1) which implement SCSI target device(s) in user-space
  • Utility commands mktape(1)vtlcmd(1)
  • And startup scripts build_library_config(1), make_vtl_devices(1)

The kernel module is based on the scsi_debug kernel module (http://sg.danny.cz/sg/sdebug26.html). mhvtl.ko is a pseudo HBA (LLD). Note: As of 0.16, there are no default devices.

The support scripts will add Drives (SSC devices) & library (SMC) depending on the contents of /etc/mhvtl/library_contents and /etc/mhvtl/device.conf.

A char device back-end has been included with the vtl LLD driver This allows data and SCSI commands to be passed from the LLD to user-mode daemons (SCSI targets) which constently poll the driver and process any outstanding SCSI commands.

vtltape

vtltape(1) is the usermode SSC target daemon which writes/reads data to data files in the /opt/mhvtl directory (if a virtual tape has been loaded). The virtual tape files include a Medium Auxiliary Memory (MAM) data structure to store persistent data (number of tape loads, total data written/read, media type etc).

vtllibrary

vtllibrary(1) is the usermode SMC target daemon which reads its configuration from the file /etc/mhvtl/library_contents(5) at startup. The number of storage slots are built dynamically when the daemon starts. Hence changing the number of storage slots and media access slots are a matter of modifying the file contents and restarting the vtllibrary(1) daemon. All ‘library’ commands are performed on data structures in memory ONLY.

vtlcmd

A utility vtlcmd(1) is used to administrator the daemons vtltape(1) and vtllibrary.
Message queue (key 0x4d61726b) is used to pass messages between vtlcmd(1)vtllibrary(1) andvtltape(1)

When a SCSI ‘move medium’ from a storage slot to a tape drive is requested, the media location is updated in vtllibrary(1) data structures, and the barcode of the media id is passed via the message queue to the vtltape(1) daemon in question. A file open of the barcode is attempted in the /opt/mhvtl directory and if successful, the vtltape(1) daemon will now return a ASC/ASCQ 0x0/0x0 to any Test Unit Ready requests. An unload SCSI command to the tape drive will close the data file.

Media can be moved out of the VTL via the Media Access Port. Once media is logically moved into the MAP slots, the MAP entries can be cleared using the vxcmd:

# vtlcmd library empty map

The media can be ‘moved back again’ by re-starting the VTL user-space daemons:

Media can be loaded into the MAP via a :

# vtlcmd library load map <barcode>

or restart the vtl

# /etc/init.d/mhvtl restart

TapeAlert

TapeAlert flags can be set using the vtlcmd(1)

e.g. vtlcmd [index] TapeAlert [flags]

Where index is the message Q offset associated with the drive (or the string ‘library’).

e.g. To set flag 14h (Clean now) the 20th bit (14h) needs to be set:
Pull out the binary to decimal calculator
1000 0000 0000 0000 0000 (20 bits) => hex => 80000

# vtlcmd 1 TapeAlert 80000

A listing of TapeAlert flags can be found at t10 home page for SSC devices Annex A.

Examples

mhvtl is registered as HBA #3:

# lsscsi -g
[0:0:0:0] disk MAXTOR ATLAS10K4_36SCA DFM0 /dev/sda /dev/sg0
[0:0:6:0] process PE/PV 1x3 SCSI BP 1.1 - /dev/sg1
[2:0:0:0] disk SEAGATE ST336607FC 0003 /dev/sdb /dev/sg2
[2:0:1:0] disk SEAGATE ST336607FC 0003 /dev/sdc /dev/sg3
[2:0:2:0] mediumx ATL 1500 6.0 - /dev/sg4
[2:0:2:1] tape QUANTUM SuperDLT1 2323 /dev/st0 /dev/sg5
[2:0:2:2] tape QUANTUM SuperDLT1 2323 /dev/st1 /dev/sg6
[2:0:2:3] process ATTO 310-LV 1.42 - /dev/sg7
[3:0:0:0] mediumx STK  L700   vtl0 - /dev/sg8
[3:0:0:1] tape SONY SDX-900V 5400 /dev/st2 /dev/sg9
[3:0:0:2] tape SONY SDX-900V 5400 /dev/st3 /dev/sg10
[3:0:0:3] tape QUANTUM SDLT600 5400 /dev/st4 /dev/sg11
[3:0:0:4] tape QUANTUM SDLT600 5400 /dev/st5 /dev/sg12
[3:0:0:5] tape QUANTUM SDLT600 5400 /dev/st6 /dev/sg13
[3:0:0:6] tape IBM ULT3580-TD3 5400 /dev/st7 /dev/sg14
[3:0:0:7] tape IBM ULT3580-TD3 5400 /dev/st8 /dev/sg15
[3:0:0:8] tape IBM ULT3580-TD3 5400 /dev/st9 /dev/sg16

Long term goals

Eventually, the functionality will be moved across to the SCSI Target Framework (stgt) http://stgt.berlios.de/

The stgt is a very nicely designed SCSI taret framwork which can support all target types along with multiple SCSI transports in a very structured way.

However, until time permits (and my c coding standards improve), features and improvements on this code base will continue.

Howto get help

Somebody has put up a discussion board (and has lots of great documentation on working with different backup software)
http://mhvtl-community-forums.966029.n3.nabble.com/ – Community Forums

http://www.facebook.com/pages/mhVTL/119540494812584 – Facebook

Howto help out

There are many way of providing assistance

  • Testing with backup software X and reporting success/failure. If possible, test any patches that may result of testing.
  • If you are willing, patches are very welcome.
    Please read the Linux Documentation/CodingStyle
    Note: I only discovered this documentation guide recently, so patches to update old code style to the CodingStyle are also welcome.
    Please run any patches thru the Linux ‘mhvtl-<version>/scripts/checkpatch.pl’ for coding style sanity check before sending.
  • Pull the source from git repository (github).. Tinker till your hearts content 🙂

Emulations

Currently, only the tape module has ‘personality modules’

For up-to-date view of emulations review vtltape.c and search for “static struct tape_drives_table”

list of tape emulations (as of v1.4-9)

Working emulations include:

  • IBM LTO (1/2/3/4 & 5) (LTO 5 emulation does not support LTFS)
  • HP LTO (1/2/3/4 & 5) (LTO 5 emulation does not support LTFS)
  • IBM 03592-J1A, 03592-E05 & 03592-E06
  • STK 9840 (A/B/C/D), 9940 (A/B)
  • STK T10000 (A/B/C)
  • Sony AIT (1/2/3 &4)

What about model X….

  • Quantum (DLT/Super-DLT) is NOT supported as their SCSI Programmer’s Guide does not document the expected returned data.
  • DDS (4mm DAT) – I’ve not found any SCSI programmers’ Guides for these drives.

Library emulation is coming in the future.

STK L180/L700 emulation works for NetBackup & NetWorker.

TSM – IBM library type needed to use LTO drives.

Note: Please let me know what works for which version of backup software so I can better document it.

Downloads

Confused as to what to get ?

If you are a RedHat based distribution (RedHat / CentOS / Oracle / Scientific Linux etc) than ELrepo may be your best bet. Checkout : http://elrepo.org/tiki/tiki-index.php

– Follow the ‘Getting Started’ section

– Import the public key:

– Install ELRepo for RHEL-5, SL-5 or CentOS-5 / install ELRepo for RHEL-6, SL-6 or CentOS-6

– $ sudo yum install mhvtl-utils kmod-mhvtl

Otherwise:
Check what sort of linux distribution.
Choose ‘src.rpm & <platform>.rpm if RPM based and want to use mhVTL in fastest way.
Choose ‘source code tar ball’ if using a non-RPM based linux distribution. (e.g. Debian/Ubuntu etc)
Choose ‘Download using git’ if your interested in fixing bugs, following development history, as well as using/testing mhVTL

There are 2 “bits” that make up the ‘mhVTL’.
– User space daemons
– mhvtl pseudo scsi hba kernel driver.

The kernel module source code is included with any of the ‘source’ packages (& git) below and needs to be downloaded and built to suit the currently running linux kernel on your linux host.

Download one or more to get going and playing as soon as possible:

package
 Description
mhvtl-2013-10-20.tgz Source code in a tar ball – not required for RPM based distributions
Contains both user space and kernel source code
mhvtl-utils-1.4-10.x86_64.rpm 64bit pre-built binaries of ‘user space’ portion of mVTL
mhvtl-utils-1.4-10.i586.rpm 32bit pre-built binaries of ‘user space’ portion of mVTL
mhvtl-utils-1.4-10.src.rpm Above source code tar ball packaged as a source rpm
git tarball Latest source tree (retrieved via git) as a ‘tar ball’

Download using ‘git’ – This provides a complete history of all changes.

Create a directory where you are going to store the source code.
$ mkdir mhvtl

Initializethe directory as a git repository
$ git init

Pull the source code (it’s only a few hundred k)
$ git pull https://github.com/markh794/mhvtl.git

From time to time, check for any updates and synchronize from github.
$ git pull https://github.com/markh794/mhvtl.git

===========

Bacula

A report from Beat Rubischon to advise Bacula works also.

Two changes were needed on Bacula’s

bacula-sd.conf:

Device {
 Name = Drive-1
 Drive Index = 0
 Media Type = LTO-4
 Archive Device = /dev/nst0
 AutomaticMount = yes;
 AlwaysOpen = yes;
 RemovableMedia = yes;
 RandomAccess = no;
 AutoChanger = yes
 Alert Command = "sh -c 'tapeinfo -f %c |grep TapeAlert|cat'"

 # Needed for mhvtl tape drives
 Hardware End of Medium = No
 Fast Forward Space File = No
}

Multiple tape drives are supported too.

New: 2010-05-21
Report by Albert Pauw.. Many thanks.

# Needed for mhvtl tape drives
Hardware End of Medium = No
Fast Forward Space File = No

I have used it this way for the last two months, but due to some questions on the mhvtl forum I tried
bacula without these additions. Seems the latest versions of mhvtl work out of the box with bacula,
so these additions are not needed anymore.

Web Management Console GUI ( Add-on )

For now and what I call version (1) will do .. I think ..

Lots of features to make your life much simpler when working with mhvtl+SCSI target system ..

To download :

https://github.com/niadev67/mhvtl-gui/tarball/master/

Or

$ mkdir mhvtl-gui
$ cd mhvtl-gui
$ git init
$ git pull https://github.com/niadev67/mhvtl-gui.git

Or

Visit The Public git Repositories https://github.com/niadev67

MHVTL GUI Requirement  :
========================

1) You will need a PHP enabled web server (Required )

   Please test with "phpinfo();"

2) Setup sudo (Required ) :

* Allow your web server user id to run commands locally as root
  e.g.: Run # echo "apache ALL=(ALL) NOPASSWD: ALL" >>/etc/sudoers
* Comment out the line "Defaults requiretty" in /etc/sudoers
* You may need to disable selinux to run sudo from httpd, Reported by crippa.andrea/MHVTL Forum

3) Install some OS utility tools :

* lsscsi (yum install lsscsi) (Required) 
* mtx (yum install mtx) (Required)
* mt-st (yum install mt-st) (Required)
* git version 1.7.4.1 or higher (Optional - needed for LIVE UPDATE Feature)
* sysstat (yum install sysstat) (Optional)

4) Install MHVTL / Minimum Release 0.18 Version 15 ] e.g. Version: 0.18.15-git-xxxxxx (Required )

* Download MHVTL via Public git Repositories https://github.com/markh794 or see http://sites.google.com/site/linuxvtl2/ 

5) Internet connectivity for LIVE UPDATE Feature (Optional)

6) tgt 1.17 or higher from http://stgt.sourceforge.net/ (Optional) used for iSCSI Target

7) Supported Internet Browser : Internet Explorer 8, FireFox, and Google Chrome (Application mode ) for better experience

MHVTL Web Console Installation :
===================================

1) Add a directory alias in your web server configuration file for MHVTL GUI:

Example :

Alias /mhvtl "/var/www/html/mhvtl"
<Directory "/var/www/html/mhvtl">
   Options None
   AllowOverride None
   Order allow,deny
   Allow from all
</Directory>

2) Copy all MHVTL GUI files to the aliased directory specified above.

3) Access MHVTL GUI via your Internet Browser e.g. http://localhost/mhvtl/ or http://10.0.0.10/mhvtl/

4) Log on with password: "mhvtl"

*** To change password, update file ~go.php where it says "if ( $password == "mhvtl" )"

Here is some pictures:

I am using Google Chrome Application shortcut mode for best viewing:

mhVTL-gui

Disclaimer:

This SOFTWARE PRODUCT is provided by me “as is” and “with all faults.” I make no representations or warranties of any kind concerning the safety, suitability, lack of viruses, inaccuracies, typographical errors, or other harmful components of this SOFTWARE PRODUCT. There are inherent dangers in the use of any software, and you are solely responsible for determining whether this SOFTWARE PRODUCT is compatible with your equipment and other software installed on your equipment. You are also solely responsible for the protection of your equipment and backup of your data, and I will not be liable for any damages you may suffer in connection with using, modifying, or distributing this SOFTWARE PRODUCT.