Warning: This document is for the development version of Bareos Documentation. The main version is bareos-18.2.

Catalog Services

General

This chapter is intended to be a technical discussion of the catalog services and as such is not targeted at end users but rather at developers and system administrators who want or need to know more of the working details of Bareos.

Currently, we prefer the usage of PostgreSQL. Therefore our support for SQLite or other databases could be discontinued in the future. PostgreSQL was chosen because it is a full-featured, very mature database, and because Dan Langille did the Bareos driver for it.

SQLite was chosen because it is small, efficient, and can be directly embedded in Bareos thus requiring much less effort from the system administrator or person building Bareos. In our testing SQLite has performed very well, and for the functions that we use, it has never encountered any errors except that it does not appear to handle databases larger than 2GBytes. That said, we would not recommend it for serious production use. Nonetheless SQLite is very suitable for test environments.

Bareos requires one of the three databases (MySQL, PostgreSQL, or SQLite) to run. Therefore it is mandatory to specify one of them for the cmake configuration step, i.e.: -Dpostgresql=yes.

Filenames and Maximum Filename Length

In general, either MySQL, PostgreSQL or SQLite permit storing arbitrary long path names and file names in the catalog database. In practice, there still may be one or two places in the catalog interface code that restrict the maximum path length to 512 characters and the maximum file name length to 512 characters. These restrictions are believed to have been removed. Please note, these restrictions apply only to the catalog database and thus to your ability to list online the files saved during any job. All information received and stored by the Storage daemon (normally on tape) allows and handles arbitrarily long path and filenames.

Database Table Design

All discussions that follow pertain to the PostgreSQL database.

Because the catalog database may contain very large amounts of data for large sites, we have made a modest attempt to normalize the data tables to reduce redundant information. While reducing the size of the database significantly, it does, unfortunately, add some complications to the structures.

In simple terms, the catalog database must contain a record of all Jobs run by Bareos, and for each Job, it must maintain a list of all files saved, with their File Attributes (permissions, create date, …), and the location and Media on which the file is stored. This is seemingly a simple task, but it represents a huge amount of interlinked data. The data stored in the File records, which allows the user or administrator to obtain a list of all files backed up during a job, is by far the largest volume of information put into the catalog database.

Although the catalog database has been designed to handle backup data for multiple clients, some users may want to maintain multiple databases, one for each machine to be backed up. This reduces the risk of confusion of accidental restoring a file to the wrong machine as well as reducing the amount of data in a single database, thus increasing efficiency and reducing the impact of a lost or damaged database.

Database Tables

Path

The Path table contains shown above the path or directory names of all directories on the system or systems.

Column Name Data Type Remark
PathId serial Primary Key
Path text Full Path

The filename and any disk name are stripped off. As with the filename, only one copy of each directory name is kept regardless of how many machines or drives have the same directory. These path names should be stored in Unix path name format.

File

The File table contains one entry for each file backed up by Bareos.

Column Name Data Type Remark
FileId serial Primary Key
FileIndex integer The sequential file number in the Job
JobId integer Link to Job Record
PathId integer Link to Path Record
DeltaSeq smallint  
MarkId integer Used to mark files during Verify Jobs
Fhinofo numeric(20)  
Fhnode numeric(20)  
LStat text File attributes in base64 encoding
MD5 text MD5/SHA1 signature in base64 encoding
name text  

File that is backed up multiple times (as is normal) will have multiple entries in the File table. This will probably be the table with the most number of records. Consequently, it is essential to keep the size of this record to an absolute minimum. At the same time, this table must contain all the information (or pointers to the information) about the file and where it is backed up.

This table contains by far the largest amount of information in the catalog database, both from the stand point of number of records, and the stand point of total database size.

As MD5 hash (also termed message digests) consists of 128-bit (16-byte). A typical format (eg. md5sum, …) to represent them is as a sequence of 32 hexadecimal digits. However, in the MD5 column, the digest is represented as base64.

To compare a Bareos digest with command line tools, you have to use

openssl dgst -md5 -binary $PATH_OF_YOUR_FILE | base64

Please note, even the table column is named MD5, it is used to store any kind of digest (MD5, SHA1, …). This is not directly indicated by the value, however, you can determine the type depending of the length of the digest.

Job

The Job table contains one record for each Job run by Bareos.

Column Name Data Type Remark
JobId serial Primary Key
Job text Unique Job Name
Name text Job Name
Type char(1) Job Type: Backup, Copy, Clone, Archive, Migration
Level char(1) Job Level
ClientId integer Client index
JobStatus char(1) Job Termination Status
SchedTime timestamp Time/date when Job scheduled
StartTime timestamp Time/date when Job started
EndTime timestamp Time/date when Job ended
RealEndTime timestamp Time/date when original Job ended
JobTDate bigint Start day in Unix format but 64 bits; used for Retention period.
VolSessionId integer Unique Volume Session ID
VolSessionTime integer Unique Volume Session Time
JobFiles integer Number of files saved in Job
JobBytes bigint Number of bytes saved in Job
ReadBytes bigint  
JobErrors integer Number of errors during Job
JobMissinFiles integer Number of files not saved (not yet used)
PoolId integer Link to Pool Record
FileSetId integer Link to FileSet Record
PriorJobId integer Link to prior Job Record when migrated
PurgedFiles smallint Set when all File records purged
HasBase smallint Set when Base Job run
HasCache smallint  
Reviewed smallint  
Comment text  

The Name field of the Job record corresponds to the Name resource record given in the Director’s configuration file.

The Job field contains a combination of the Name and the schedule time of the Job by the Director. Thus for a given Director, even with multiple catalog databases, the Job will contain a unique name that represents the Job.

For a given Storage daemon, the VolSessionId and VolSessionTime form a unique identification of the Job. This will be the case even if multiple Directors are using the same Storage daemon.

The JobStatus field specifies how the job terminated.

FileSet

The FileSet table contains one entry for each FileSet that is used.

Column Name Data Type Remark
FileSetId serial Primary Key
FileSet text FileSet name
FileSetText text  
MD5 text MD5 checksum of FileSet
CreateTime timestamp Time and date Fileset created

The MD5 signature is kept to ensure that if the user changes anything inside the FileSet, it will be detected and the new FileSet will be used. This is particularly important when doing an incremental update. If the user deletes a file or adds a file, we need to ensure that a Full backup is done prior to the next incremental.

JobMedia

The JobMedia table contains one entry at the following: start of the job, start of each new tape file mark, start of each new tape, end of the job. You will have 2 or more JobMedia records per Job.

Column Name Data Type Remark
JobMediaId serial Primary Key
JobId integer Link to Job Record
MediaId integer Link to Media Record
FirstIndex integer The index (sequence number) of the first file written for this Job to the Media
LastIndex integer The index of the last file written for this Job to the Media
StartFile integer
Tape: The physical media file mark number of the first block written for this Job.
Other: Upper 32-bit of the position of the first block written for this Job.
EndFile integer
Tape: The physical media file mark number of the last block written for this Job
Other: Upper 32-bit of the position of the last block written for this Job
StartBlock integer
Tape: The number of the first block written for this Job
Other: Lower 32-bit of the position of the first block written for this Job
Endblock integer
Tape: The number of the last block written for this Job
Other: Lower 32-bit of the position of the last block written for this Job
JobBytes numeric(20) The Volume use sequence number within the Job
VolIndex integer The Volume use sequence number within the Job

Device

This is the device table. It contains information about reading and or writing devices.

Column Name Data Type Remark
DeviceId serial  
Name text  
MediaTypeId integer  
StorageId integer  
DevMounts integer  
DevReadBytes bigint  
DevWriteBytes bigint  
DevReadBytesSinceCleaning bigint  
DevWriteBytesSinceCleaning bigint  
DevReadTime bigint  
DevWriteTime bigint  
DevReadTimeSinceCleaning bigint  
DevWriteTimeSinceCleaning bigint  
CleaningDate timestamp  
CleaningPeriod bigint  

Tape Volume

The number ob records depends on the “Maximum File Size” specified in the Device resource. This record allows Bareos to efficiently position close to any given file in a backup. For restoring a full Job, these records are not very important, but if you want to retrieve a single file that was written near the end of a 100GB backup, the JobMedia records can speed it up by orders of magnitude by permitting forward spacing files and blocks rather than reading the whole 100GB backup.

Other Volume

StartFile and StartBlock are both 32-bit integer values. However, as the position on a disk volume is specified in bytes, we need this to be a 64-bit value.

Therefore, the start position is calculated as:

StartPosition = StartFile * 4294967296 + StartBlock

The end position of a job on a volume can be determined by:

EndPosition = EndFile * 4294967296 + EndBlock

Be aware, that you can not assume, that the job size on a volume is EndPosition - StartPosition. When interleaving is used other jobs can also be stored between Start- and EndPosition.

EndPosition - StartPosition >= JobSizeOnThisMedia

Media (Volume)

The Media table contains one entry for each volume, that is each tape or file on which information is or was backed up. There is one volume record created for each of the NumVols specified in the Pool resource record.

Column Name Data Type Remark
MediaId serial Primary Key
VolumeName text Volume name
Slot integer Autochanger Slot number or zero
PoolId integer Link to Pool Record
MediaType text The MediaType supplied by the user
MediaTypeId integer The MediaTypeId
LabelType integer The type of label on the Volume
FirstWritten timestamp Time/date when first written
LastWritten timestamp Time/date when last written
LabelDate timestamp Time/date when tape labeled
VolJobs integer Number of jobs written to this media
VolFiles integer Number of files written to this media
VolBlocks integer Number of blocks written to this media
VolMounts integer Number of time media mounted
VolBytes bigint Number of bytes saved in Job
VolErrors integer Number of errors during Job
VolWrites integer Number of writes to media
VolCapacityBytes bigint Capacity estimate for this volume
VolStatus text
Status of media:
Full, Archive, Append, Recycle, Read-Only, Disabled, Error, Busy
Enabled smallint Whether or not Volume can be written
Recycle smallint Whether or not Bareos can recycle the Volumes: Yes, No
ActionOnPurge smallint What happens to a Volume after purging
VolRetention bigint 64 bit seconds until expiration
VolUseDureation bigint 64 bit seconds volume can be used
MaxVolJobs integer maximum jobs to put on Volume
MaxVolFiles integer maximume EOF marks to put on Volume
MaxVolBytes bigint Maximum bytes to put on this media
InChanger smallint Whether or not Volume in autochanger
StorageId integer Storage record ID
DeviceId integer Device record ID
MediaAddressing smallint Method of addressing media
VolReadTime bigint Time Reading Volume
VolWriteTime bigint Time Writing Volume
EndFile integer End File number of Volume
EndBlock bigint End block number of Volume
LocationId integer Location record ID
RecycleCount integer Number of times recycled
MinBlockSize integer Minimum block size on this media
MaxBlockSize integer Maximum block size on this media
InitialWrite timestamp When Volume first written
ScratchPoolId integer Id of Scratch Pool
RecyclePoolId integer Pool ID where to recycle Volume
EncryptionKey text Key used for encryptoion
Comment text User text field

Pool

The Pool table contains one entry for each media pool controlled by Bareos in this database.

Column Name Data Type Remark
PoolId serial Primary Key
Name text Pool Name
NumVols integer Number of Volumes in the Pool
MaxVols integer Maximum Volumes in the Pool
UseOnce smallint Use volume once
UseCatalog smallint Set to use catalog
AcceptAnyVolume smallint Accept any volume from Pool
VolRetention bigint 64 bit seconds to retain volume
VolUseDuration bigint 64 bit seconds volume can be used
MaxVolJobs integer max jobs on volume
MaxVolFiles integer max EOF marks to put on Volume
MaxVolBytes bigint max bytes to write on Volume
AutoPrune smallint yes or no for autopruning
Recycle smallint yes or no for allowing auto recycling of Volume
ActionOnPurge smallint Default Volume ActionOnPurge
PoolType text Backup, Copy, Cloned, Archive, Migration
LabelType integer Type of label ANSI/Bareos
LabelFormat text Label format
Enabled smallint Whether or not Volume can be written
ScratchPoolId integer Id of Scratch Pool
RecyclePoolId integer Pool ID where to recycle Volume
NextPoolId integer Pool ID of next Pool
MinBlockSize integer Pool ID of next Pool
MaxBlockSize integer Pool ID of next Pool
MigrationHighBytes bigint High water mark for migration
MigrationLowBytes bigint Low water mark for migration
MigrationTime bigint Time before migration

In the Media table one or more records exist for each of the Volumes contained in the Pool. The MediaType is defined by the administrator, and corresponds to the MediaType specified in the Director’s Storage definition record.

Client

The Client table contains one entry for each machine backed up by Bareos in this database. Normally the Name is a fully qualified domain name.

Column Name Data Type Remark
ClientId serial Primary Key
Name text File Services Name
UName text uname -a from Client (not yet used)
AutoPrune smallint yes or no for autopruning
FileRetention bigint 64 bit seconds to retain Files
JobRentention bigint 64 bit seconds to retain Job

Storage

The Storage table contains one entry for each Storage used.

Column Name Data Type Remark
StorageId serial Unique Id
Name text Resource name of Storage device
AutoChanger integer Set if it is an autochanger

Counters

The Counter table contains one entry for each permanent counter defined by the user.

Column Name Data Type Remark
Counter serial Counter name
MinValue integer Start/Min value for counter
MaxValue integer Max value for counter
CurrentValue integer Current counter value
WrapCounter text Name of another counter

Log

The Log table contains a log of all Job output.

Column Name Data Type Remark
LogId serial Primary Key
JobId integer Points to Job record
Time timestamp Time/date log record created
LogText text Log text

Location

The Location table defines where a Volume is physically.

Column Name Data Type Remark
LocationId serial Primary Key
Location text Text defining location
Cost integer Relative cost of obtaining Volume
Enabled smallint Whether or not Volume is enabled

LocationLog

The LocationLog table contains a log of all Job output.

Column Name Data Type Remark
LocLogId serial Primary Key
Date timestamp Time/date log record created
Comment text Time/date log record created
MediaId integer Points to Media record
LocationId integer Points to Location record
NewVolStaus integer enum: Full, Archive, Append, Recycle, Purged Read-only, Disabled, Error, Busy, Used, Cleaning
Enabled tinyint Whether or not Volume is enabled

Version

The Version table defines the Bareos database version number. Bareos checks this number before reading the database to ensure that it is compatible with the Bareos binary file.

Column Name Data Type Remark
VersionId integer Primary Key

BaseFiles

The BaseFiles table contains all the File references for a particular JobId that point to a Base file.

Column Name Data Type Remark
BaseId serial Primary Key
JobId integer Reference to Job
FileId bigint Reference to File
FileIndex integer File Index number
BaseJobId integer JobId of Base Job

For example they were previously saved and hence were not saved in the current JobId but in BaseJobId under FileId. FileIndex is the index of the file, and is used for optimization of Restore jobs to prevent the need to read the FileId record when creating the in memory tree. This record is not yet implemented.

UML Diagram of Database Schema

.. uml::
  :caption: UML Diagram of the Bareos Database Schema

  @startuml

  ' hide the spot
  hide circle

   'avoid problems with angled crows feet
  skinparam linetype ortho

  entity Path {
    **PathId** : SERIAL
    --
    Path : text
  }

  entity File {
    **FileId** : BIGSERIAL
    --
    FileIndex : INTEGER
    **JobId** : INTEGER
    **PathId** :INTEGER
    DeltaSeq : SMALLINT
    MarkId : INTEGER
    Fhinfo : NUMERIC(20)
    Fhnode : NUMERIC(20)
    LStat: TEXT
    Md5 : TEXT
    Name : TEXT
  }


  entity RestoreObject {
    **RestoreObjectId** : SERIAL
    --
    ObjectName : TEXT
    RestoreObject : BYTEA
    PluginName : TEXT
    ObjectLength : INTEGER
    ObjectFullLength : INTEGER
    ObjectIndex : INTEGER
    ObjectType : INTEGER
    FileIndex : INTEGER
    **JobId** : INTEGER
    ObjectCompression : INTEGER
  }

  entity Job {
    **JobId** : SERIAL
    --
    Job : TEXT
    Name : TEXT
    Type : CHAR(1)
    Level :CHAR(1)
    **ClientId** : INTEGER
    JobStatus : CHAR(1)
    SchedTime : TIMESTAMP
    StartTime : TIMESTAMP
    EndTime : TIMESTAMP
    RealEndTime : TIMESTAMP
    JobTDate : BIGINT
    VolSessionId : INTEGER
    volSessionTime : INTEGER
    JobFiles : INTEGER
    JobBytes : BIGINT
    ReadBytes : BIGINT
    JobErrors : INTEGER
    JobMissingFiles : INTEGER
    **PoolId** : INTEGER
    **FilesetId** : INTEGER
    **PriorJobid** : INTEGER
    PurgedFiles : SMALLINT
    HasBase : SMALLINT
    HasCache : SMALLINT
    Reviewed : SMALLINT
    Comment : TEXT
  }


  entity Location {
    **LocationId** : SERIAL
    --
    Location : TEXT
    Cost : INTEGER
    Enabled : SMALLINT
  }

  entity Fileset {
    **FileSetId** : SERIAL
    --
    Fileset : TEXT
    FileSetText : TEXT
    Md5 : TEXT
    CreateTime : TIMESTAMP
  }

  entity JobMedia {
    **JobMediaId** : SERIAL
    --
    **JobId** : INTEGER
    **MediaId** : INTEGER
    FirstIndex : INTEGER
    LastIndex : INTEGER
    StartFile : INTEGER
    EndFile : INTEGER
    StartBlock : BIGINT
    EndBlock : BIGINT
    JobBytes : NUMERIC(20)
    VolIndex : INTEGER
  }

  entity Media {
    **MediaId** : SERIAL
    --
    VolumeName : TEXT
    Slot : INTEGER
    **PoolId** : INTEGER
    MediaType : TEXT
    **MediaTypeId** : INTEGER
    LabelType : INTEGER
    FirstWritten : TIMESTAMP
    LastWritten : TIMESTAMP
    LabelDate : TIMESTAMP
    VolJobs : INTEGER
    VolFiles : INTEGER
    VolBlocks : INTEGER
    VolMounts : INTEGER
    VolBytes : BIGINT
    VolErrors : INTEGER
    VolWrites : INTEGER
    VolCapacityBytes : BIGINT
    VolStatus : TEXT
    Enabled : SMALLINT
    Recycle : SMALLINT
    ActionOnPurge : SMALLINT
    VolRetention : BIGINT
    VolUseDuration : BIGINT
    MaxVolJobs : INTEGER
    MaxVolFiles : INTEGER
    MaxVolBytes : BIGINT
    InChanger : SMALLINT
    **StorageId** : INTEGER
    **DeviceId** : INTEGER
    MediaAddressing : SMALLINT
    VolReadTime : BIGINT
    VolWriteTime : BIGINT
    EndFile : INTEGER
    EndBlock : BIGINT
    **LocationId** : INTEGER
    RecycleCount : INTEGER
    MinBlockSize : INTEGER
    MaxBlockSize : INTEGER
    InitialWrite : TIMESTAMP
    **ScratchPoolId** : INTEGER
    **RecyclePoolId** : INTEGER
    EncryptionKey : TEXT
    Comment : TEXT
  }

  entity MediaType {
    **MediaTypeId** : SERIAL
    --
    MediaType : TEXT
    ReadOnly : INTEGER
  }

  entity Storage {
    **StorageId** : SERIAL
    --
    Name : TEXT
    AutoChanger : INTEGER
  }

  entity Device {
    **DeviceId** : SERIAL
    --
    Name : TEXT
    **MediaTypeId** : INTEGER
    **StorageId** : INTEGER
    DevMounts : INTEGER
    DevReadBytes : BIGINT
    DevWriteBytes : BIGINT
    DevReadBytesSinceCleaning : BIGINT
    DevWriteBytesSinceCleaning : BIGINT
    DevReadTime : BIGINT
    DevWriteTime : BIGINT
    DevReadTimeSinceCleaning : BIGINT
    DevWriteTimeSinceCleaning : BIGINT
    CleaningDate : TIMESTAMP
    CleaningPeriod : BIGINT
  }

  entity Pool {
    **PoolId** : SERIAL
    --
    Name : TEXT
    NumVols : INTEGER
    MaxVols : INTEGER
    UseOnce : SMALLINT
    UseCatalog : SMALLINT
    AcceptAnyVolume : SMALLINT
    VolRetention : BIGINT
    VolUseDuration : BIGINT
    MaxVolJobs : INTEGER
    MaxVolFiles : INTEGER
    MaxVolBytes : BIGINT
    AutoPrune : SMALLINT
    Recycle : SMALLINT
    ActionOnPurge : SMALLINT
    PoolType : TEXT
    LabelType : INTEGER
    LabelFormat : TEXT
    Enabled : SMALLINT
    **ScratchPoolId** : INTEGER
    **RecyclePoolId** : INTEGER
    **NextPoolId** : INTEGER
    MinBlockSize : INTEGER
    MaxBlockSize : INTEGER
    MigrationHighBytes : BIGINT
    MigrationLowBytes : BIGINT
    MigrationTime : BIGINT
  }


  entity Client {
    **ClientId** : SERIAL
    --
    Name : TEXT
    UName : TEXT
    AutoPrune : SMALLINT
    FileRetention : BIGINT
    JobRetention : BIGINT
  }

  entity Log {
    **LogId** : SERIAL
    --
    **JobId** : INTEGER
    Time : TIMESTAMP
    LogText : TEXT
  }

  entity LocationLog {
    **LocLogId** : SERIAL
    --
    Date : TIMESTAMP
    Comment : TEXT
    **MediaId** : INTEGER
    LocationId : INTEGER
    NewVolStatus : TEXT
    newenabled : SMALLINT
  }

  entity basefiles {
    **BaseId** : SERIAL
    --
    **JobId** : INTEGER
    FileId : BIGINT
    FileIndex : INTEGER
    BaseJobId : INTEGER
  }

  entity PathHierarchy {
    **PathId** : INTEGER
    --
    PPathId : INTEGER
  }

  entity PathVisibility {
    **PathId** : INTEGER
    **JobId** : INTEGER
    --
    Size : BIGINT
    Files : INTEGER
  }

  entity Quota {
    **ClientId** : INTEGER
    --
    GraceTime : BIGINT
    QuotaLimit : BIGINT
  }

  entity NDMPLevelMap {
    **ClientId** : INTEGER
    **FilesetId** : INTEGER
    FileSystem : TEXT
    --
    DumpLevel : INTEGER
  }

  entity NDMPJobEnvironment {
    **JobId** : INTEGER
    FileIndex : INTEGER
    EnvName : TEXT
    --
    EnvValue : TEXT
  }

  entity DeviceStats {
    **DeviceId** : INTEGER
    SampleTime : TIMESTAMP
    ReadTime : BIGINT
    WriteTime : BIGINT
    ReadBytes : BIGINT
    WriteBytes : BIGINT
    SpoolSize : BIGINT
    NumWaiting : SMALLINT
    NumWriters : SMALLINT
    **MediaId** : INTEGER
    VolCatBytes : BIGINT
    VolCatFiles : BIGINT
    VolCatBlocks : BIGINT
  }

  entity JobStats {
    **DeviceId** : INTEGER
    SampleTime : TIMESTAMP
    **JobId** : INTEGER
    JobFiles : INTEGER
    JobBytes : BIGINT
  }

  entity TapeAlerts {
    **DeviceId** : INTEGER
    SampleTime : TIMESTAMP
    AlertFlags : BIGINT
  }

  package unreferenced <<Rectangle>> {
  entity Status {
    **JobStatus** : CHAR(1)
    --
    JobStatusLong : TEXT
    Severity : INTEGER
  }

  entity counters {
    **Counter** : TEXT
    --
    MinValue : INTEGER
    MaxValue : INTEGER
    CurrentValue : INTEGER
    wrapcounter : TEXT
  }

  entity version {
    **VersionId** : INTEGER
    --
  }
  }

  Client }o-- Job
  Client }o-- Quota
  Client }o-- NDMPLevelMap

  Device }o-- TapeAlerts
  Device }o-- DeviceStats
  Device }o-- Media

  Fileset }o-- Job
  Fileset }o-- NDMPLevelMap

  Job  }o-- File
  Job  }o-- Job
  Job  }o-- JobMedia
  Job  }o-- Log
  Job  }o-- basefiles
  Job  }o-- PathVisibility
  Job  }o-- NDMPJobEnvironment
  Job  }o-- JobStats
  Job  }o-- RestoreObject

  Location }o-- Media

  Media }o-- JobMedia
  Media }o-- LocationLog
  Media }o-- DeviceStats

  MediaType }o-- Device
  MediaType }o-- Media

  Path }o-- File
  Path }o-- PathHierarchy
  Path }o-- PathVisibility

  Pool }o-- Job
  Pool }o-- Pool
  Pool }o--|{ Media

  Storage }o-- Device
  Storage }o-- Media

  @enduml