469 lines
19 KiB
HTML
469 lines
19 KiB
HTML
<!DOCTYPE html>
|
|
<html><head>
|
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
|
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
|
|
<link href="sqlite.css" rel="stylesheet">
|
|
<title>The SQLite OS Interface or "VFS"</title>
|
|
<!-- path= -->
|
|
</head>
|
|
<body>
|
|
<div class=nosearch>
|
|
<a href="index.html">
|
|
<img class="logo" src="images/sqlite370_banner.gif" alt="SQLite" border="0">
|
|
</a>
|
|
<div><!-- IE hack to prevent disappearing logo --></div>
|
|
<div class="tagline desktoponly">
|
|
Small. Fast. Reliable.<br>Choose any three.
|
|
</div>
|
|
<div class="menu mainmenu">
|
|
<ul>
|
|
<li><a href="index.html">Home</a>
|
|
<li class='mobileonly'><a href="javascript:void(0)" onclick='toggle_div("submenu")'>Menu</a>
|
|
<li class='wideonly'><a href='about.html'>About</a>
|
|
<li class='desktoponly'><a href="docs.html">Documentation</a>
|
|
<li class='desktoponly'><a href="download.html">Download</a>
|
|
<li class='wideonly'><a href='copyright.html'>License</a>
|
|
<li class='desktoponly'><a href="support.html">Support</a>
|
|
<li class='desktoponly'><a href="prosupport.html">Purchase</a>
|
|
<li class='search' id='search_menubutton'>
|
|
<a href="javascript:void(0)" onclick='toggle_search()'>Search</a>
|
|
</ul>
|
|
</div>
|
|
<div class="menu submenu" id="submenu">
|
|
<ul>
|
|
<li><a href='about.html'>About</a>
|
|
<li><a href='docs.html'>Documentation</a>
|
|
<li><a href='download.html'>Download</a>
|
|
<li><a href='support.html'>Support</a>
|
|
<li><a href='prosupport.html'>Purchase</a>
|
|
</ul>
|
|
</div>
|
|
<div class="searchmenu" id="searchmenu">
|
|
<form method="GET" action="search">
|
|
<select name="s" id="searchtype">
|
|
<option value="d">Search Documentation</option>
|
|
<option value="c">Search Changelog</option>
|
|
</select>
|
|
<input type="text" name="q" id="searchbox" value="">
|
|
<input type="submit" value="Go">
|
|
</form>
|
|
</div>
|
|
</div>
|
|
<script>
|
|
function toggle_div(nm) {
|
|
var w = document.getElementById(nm);
|
|
if( w.style.display=="block" ){
|
|
w.style.display = "none";
|
|
}else{
|
|
w.style.display = "block";
|
|
}
|
|
}
|
|
function toggle_search() {
|
|
var w = document.getElementById("searchmenu");
|
|
if( w.style.display=="block" ){
|
|
w.style.display = "none";
|
|
} else {
|
|
w.style.display = "block";
|
|
setTimeout(function(){
|
|
document.getElementById("searchbox").focus()
|
|
}, 30);
|
|
}
|
|
}
|
|
function div_off(nm){document.getElementById(nm).style.display="none";}
|
|
window.onbeforeunload = function(e){div_off("submenu");}
|
|
/* Disable the Search feature if we are not operating from CGI, since */
|
|
/* Search is accomplished using CGI and will not work without it. */
|
|
if( !location.origin || !location.origin.match || !location.origin.match(/http/) ){
|
|
document.getElementById("search_menubutton").style.display = "none";
|
|
}
|
|
/* Used by the Hide/Show button beside syntax diagrams, to toggle the */
|
|
function hideorshow(btn,obj){
|
|
var x = document.getElementById(obj);
|
|
var b = document.getElementById(btn);
|
|
if( x.style.display!='none' ){
|
|
x.style.display = 'none';
|
|
b.innerHTML='show';
|
|
}else{
|
|
x.style.display = '';
|
|
b.innerHTML='hide';
|
|
}
|
|
return false;
|
|
}
|
|
</script>
|
|
</div>
|
|
<div class=fancy>
|
|
<div class=nosearch>
|
|
<div class="fancy_title">
|
|
The SQLite OS Interface or "VFS"
|
|
</div>
|
|
<div class="fancy_toc">
|
|
<a onclick="toggle_toc()">
|
|
<span class="fancy_toc_mark" id="toc_mk">►</span>
|
|
Table Of Contents
|
|
</a>
|
|
<div id="toc_sub"><div class="fancy-toc1"><a href="#introduction">1. Introduction</a></div>
|
|
<div class="fancy-toc1"><a href="#the_vfs_in_relation_to_the_rest_of_sqlite">2. The VFS In Relation To The Rest Of SQLite</a></div>
|
|
<div class="fancy-toc1"><a href="#multiple_vfses">3. Multiple VFSes</a></div>
|
|
<div class="fancy-toc2"><a href="#standard_unix_vfses">3.1. Standard Unix VFSes</a></div>
|
|
<div class="fancy-toc2"><a href="#standard_windows_vfses">3.2. Standard Windows VFSes</a></div>
|
|
<div class="fancy-toc2"><a href="#specifying_which_vfs_to_use">3.3. Specifying Which VFS To Use</a></div>
|
|
<div class="fancy-toc2"><a href="#vfs_shims">3.4. VFS Shims</a></div>
|
|
<div class="fancy-toc2"><a href="#other_example_vfses">3.5. Other Example VFSes</a></div>
|
|
<div class="fancy-toc1"><a href="#vfs_implementations">4. VFS Implementations</a></div>
|
|
</div>
|
|
</div>
|
|
<script>
|
|
function toggle_toc(){
|
|
var sub = document.getElementById("toc_sub")
|
|
var mk = document.getElementById("toc_mk")
|
|
if( sub.style.display!="block" ){
|
|
sub.style.display = "block";
|
|
mk.innerHTML = "▼";
|
|
} else {
|
|
sub.style.display = "none";
|
|
mk.innerHTML = "►";
|
|
}
|
|
}
|
|
</script>
|
|
</div>
|
|
|
|
|
|
|
|
|
|
|
|
<h1 id="introduction"><span>1. </span>Introduction</h1>
|
|
|
|
<p>
|
|
This article describes the SQLite OS portability layer or "VFS" - the
|
|
module at the bottom of the SQLite implementation stack
|
|
that provides portability across operating systems.
|
|
</p>
|
|
|
|
<h1 id="the_vfs_in_relation_to_the_rest_of_sqlite"><span>2. </span>The VFS In Relation To The Rest Of SQLite</h1>
|
|
|
|
<div>
|
|
<img src="images/vfs1.gif" align="right" hspace="10">
|
|
</div>
|
|
|
|
<p>
|
|
The internal organization of the SQLite library can be viewed as the
|
|
stack of modules shown to the right.
|
|
The Tokenizer, Parser, and Code Generator components are used to
|
|
process SQL statements and convert them into executable programs
|
|
in a virtual machine language or byte code.
|
|
Roughly speaking, these top three layers implement
|
|
<a href="c3ref/prepare.html">sqlite3_prepare_v2()</a>. The byte code generated by the top three
|
|
layers is a <a href="c3ref/stmt.html">prepared statement</a>.
|
|
The Virtual Machine module is responsible for running the SQL statement
|
|
byte code. The B-Tree module organizes a database file into multiple
|
|
key/value stores with ordered keys and logarithmic performance.
|
|
The Pager module is responsible for loading pages of the database
|
|
file into memory, for implementing and controlling transactions, and
|
|
for creating and maintaining the journal files that prevent database
|
|
corruption following a crash or power failure.
|
|
The OS Interface is a thin abstraction that provides a common set of
|
|
routines for adapting SQLite to run on different operating systems.
|
|
Roughly speaking, the bottom four layers implement
|
|
<a href="c3ref/step.html">sqlite3_step()</a>.
|
|
</p>
|
|
|
|
<p>
|
|
This article is about the bottom layer.
|
|
</p>
|
|
|
|
<p>The OS Interface - also called the "VFS" - is what makes SQLite
|
|
portable across operating systems. Whenever any of the other modules
|
|
in SQLite needs to communicate with the operating
|
|
system, they invoke methods in the VFS. The VFS then invokes the
|
|
operating-specific code needed to satisfy the request.
|
|
Hence, porting SQLite to a new
|
|
operating system is simply a matter of writing a new OS interface layer
|
|
or "VFS".</p>
|
|
|
|
<h1 id="multiple_vfses"><span>3. </span>Multiple VFSes</h1>
|
|
|
|
<p>
|
|
The standard SQLite source tree contains built-in VFSes for unix
|
|
and windows. Alternative VFSes can be
|
|
added at start-time or run-time using the
|
|
<a href="c3ref/vfs_find.html">sqlite3_vfs_register()</a> interface.
|
|
</p>
|
|
|
|
<p>
|
|
Multiple VFSes can be registered at the same time.
|
|
Each VFS has a unique names.
|
|
Separate <a href="c3ref/sqlite3.html">database connections</a> within the same process can be using
|
|
different VFSes at the same time. For that matter, if a single
|
|
database connection has multiple database files open using
|
|
the <a href="lang_attach.html">ATTACH</a> command, then each attached database might be using a
|
|
different VFS.
|
|
</p>
|
|
|
|
<h2 id="standard_unix_vfses"><span>3.1. </span>Standard Unix VFSes</h2>
|
|
|
|
<p>
|
|
Unix builds come with multiple built-in VFSes. The default VFS
|
|
for unix is called "unix" and is used in most applications.
|
|
Other VFSes that might be found in unix (depending on compile-time
|
|
options) include:
|
|
</p>
|
|
|
|
<ol>
|
|
<li><p><b>unix-dotfile</b> - uses dot-file locking rather than
|
|
POSIX advisory locks.
|
|
</p></li><li><p><b>unix-excl</b> - obtains and holds an exclusive lock on
|
|
database files, preventing other processes from accessing the
|
|
database. Also keeps the <a href="walformat.html#shm">wal-index</a> in heap rather than in
|
|
shared memory.
|
|
</p></li><li><p><b>unix-none</b> - all file locking operations are no-ops.
|
|
</p></li><li><p><b>unix-namedsem</b> - uses named semaphores for file locking.
|
|
VXWorks only.
|
|
</p></li></ol>
|
|
|
|
<p>
|
|
The various unix VFSes differ only in the way they handle file locking -
|
|
they share most of their implementation in common with one another and
|
|
are all located in the same SQLite source file:
|
|
<a href="http://www.sqlite.org/src/doc/trunk/src/os_unix.c">os_unix.c</a>.
|
|
Note that except for "unix" and "unix-excl", the various unix VFSes all
|
|
use incompatible locking implementations. If two processes are accessing
|
|
the same SQLite database using different unix VFSes, they may
|
|
not see each others locks and may end up interfering with one another,
|
|
resulting in database corruption. The "unix-none" VFS in particular
|
|
does no locking at all and will easily result in database corruption if
|
|
used by two or more database connections at the same time.
|
|
Programmers are encouraged to use only "unix" or "unix-excl" unless
|
|
there is a compelling reason to do otherwise.
|
|
</p>
|
|
|
|
<h2 id="standard_windows_vfses"><span>3.2. </span>Standard Windows VFSes</h2>
|
|
|
|
|
|
<p>
|
|
Windows builds also come with multiple built-in VFSes. The default
|
|
Windows VFS is called "win32" and is used in most applications.
|
|
Other VFSes that might be found on windows builds include:
|
|
</p>
|
|
|
|
<ol>
|
|
<li><p><b>win32-longpath</b> - like "win32" except that pathnames can
|
|
be up to 65534 bytes in length, whereas pathnames max out at
|
|
1040 bytes in "win32".
|
|
</p></li><li><p><b>win32-none</b> - all file locking operations are no-ops.
|
|
</p></li><li><p><b>win32-longpath-none</b> - combination of "win32-longpath"
|
|
and "win32-none" - long pathnames are supported and all lock
|
|
operations are no-ops.
|
|
</p></li></ol>
|
|
|
|
<p>As with unix, most of the code for the various Windows VFSes is shared.
|
|
|
|
</p><h2 id="specifying_which_vfs_to_use"><span>3.3. </span>Specifying Which VFS To Use</h2>
|
|
|
|
<p>
|
|
There is always one VFS which is the default VFS. On unix systems,
|
|
the "unix" VFS comes up as the default and on windows it is "win32".
|
|
If no other actions are taken, new database connections will make use
|
|
of the default VFS.
|
|
</p>
|
|
|
|
<p>
|
|
The default VFS can be changed by registering or re-registering the
|
|
VFS using the <a href="c3ref/vfs_find.html">sqlite3_vfs_register()</a> interface with a second parameter
|
|
of 1. Hence, if a (unix) process wants to always use the "unix-nolock" VFS
|
|
in place of "unix", the following code would work:
|
|
</p>
|
|
|
|
<blockquote><pre>
|
|
sqlite3_vfs_register(sqlite3_vfs_find("unix-nolock"), 1);
|
|
</pre></blockquote>
|
|
|
|
<p>
|
|
An alternate VFS can also be specified as the 4th parameter to the
|
|
<a href="c3ref/open.html">sqlite3_open_v2()</a> function. For example:
|
|
</p>
|
|
|
|
<blockquote><pre>
|
|
int rc = sqlite3_open_v2("demo.db", &db, SQLITE_OPEN_READWRITE, "unix-nolock");
|
|
</pre></blockquote>
|
|
|
|
<p>
|
|
Finally, if <a href="uri.html">URI filenames</a> have been enabled, then the alternative
|
|
VFS can be specified using the "vfs=" parameter on the URI. This technique
|
|
works with <a href="c3ref/open.html">sqlite3_open()</a>, <a href="c3ref/open.html">sqlite3_open16()</a>, <a href="c3ref/open.html">sqlite3_open_v2()</a>, and
|
|
when a new database is <a href="lang_attach.html">ATTACH</a>-ed to an existing database connection.
|
|
For example:
|
|
</p>
|
|
|
|
<blockquote><pre>
|
|
ATTACH 'file:demo2.db?vfs=unix-none' AS demo2;
|
|
</pre></blockquote>
|
|
|
|
<p>
|
|
The VFS specified by a URI has the highest priority. After that comes
|
|
a VFS specified as the fourth argument to <a href="c3ref/open.html">sqlite3_open_v2()</a>. The
|
|
default VFS is used if no VFS is specified otherwise.
|
|
</p>
|
|
|
|
<a name="shim"></a>
|
|
|
|
<h2 id="vfs_shims"><span>3.4. </span>VFS Shims</h2>
|
|
|
|
<p>
|
|
From the point of view of the uppers layers of the SQLite stack, each
|
|
open database file uses exactly one VFS.
|
|
But in practice, a particular VFS might
|
|
just be a thin wrapper around another VFS that does the real work.
|
|
We call a wrapper VFS a "shim".
|
|
</p>
|
|
|
|
<p>
|
|
A simple example of a shim is the "vfstrace" VFS. This is a VFS
|
|
(implemented in the
|
|
<a href="http://www.sqlite.org/src/doc/trunk/src/test_vfstrace.c">test_vfstrace.c</a>
|
|
source file) that writes a message associated with each VFS method call
|
|
into a log file, then passes control off to another VFS to do the actual
|
|
work.
|
|
</p>
|
|
|
|
<h2 id="other_example_vfses"><span>3.5. </span>Other Example VFSes</h2>
|
|
|
|
<p>
|
|
The following are other VFS implementations available in the public
|
|
SQLite source tree:
|
|
</p>
|
|
|
|
<ul>
|
|
<li><p>
|
|
<a href="http://www.sqlite.org/src/file/ext/misc/appendvfs.c">appendvfs.c</a> -
|
|
This VFS allows an SQLite database to be appended to the end of some
|
|
other file. This can be used, for example, to append an SQLite database
|
|
onto the end of an executable such that the executable where it can
|
|
then be easily located by the executable when needed. The
|
|
<a href="cli.html">command-line shell</a> will use this VFS if launched with the --append
|
|
option.
|
|
|
|
</p></li><li><p>
|
|
<a href="http://www.sqlite.org/src/doc/trunk/src/test_demovfs.c">test_demovfs.c</a> -
|
|
This file implements a very simple VFS named "demo" that uses POSIX
|
|
functions such as
|
|
open(), read(), write(), fsync(), close(), fsync(), sleep(), time(),
|
|
and so forth. This VFS only works on unix systems. But it is not
|
|
intended as a replacement for the standard "unix" VFS used by default
|
|
on unix platforms. The "demo" VFS is deliberately kept very simple
|
|
so that it can be used as a learning aid or as template for building
|
|
other VFSes or for porting SQLite to new operating systems.
|
|
|
|
</p></li><li><p>
|
|
<a href="http://www.sqlite.org/src/doc/trunk/src/test_quota.c">test_quota.c</a> -
|
|
This file implements a shim called "quota" that enforces cumulative
|
|
file size limits on a collection of database files. An auxiliary
|
|
interface is used to define "quota groups". A quota group is a
|
|
set of files (database files, journals, and temporary files) whose
|
|
names all match a <a href="lang_expr.html#glob">GLOB</a> pattern. The sum of the sizes of all files
|
|
in each quota group is tracked, and if that sum exceeds a threshold
|
|
defined for the quota group, a callback function is invoked. That
|
|
callback can either increase the threshold or cause the operation
|
|
that would have exceeded the quota to fail with an
|
|
<a href="rescode.html#full">SQLITE_FULL</a> error. One of the uses of this shim is used to enforce
|
|
resource limits on application databases in Firefox.
|
|
|
|
</p></li><li><p>
|
|
<a href="http://www.sqlite.org/src/doc/trunk/src/test_multiplex.c">test_multiplex.c</a> -
|
|
This file implements a shim that allows database files to exceed the
|
|
maximum file size of the underlying filesystem. This shim presents
|
|
an interface to the upper six layers of SQLite that makes it look like
|
|
very large files are being used, when in reality each such large file
|
|
is split up into many smaller files on the underlying system.
|
|
This shim has been used, for example, to allow databases to grow
|
|
larger than 2 gibibytes on FAT16 filesystems.
|
|
|
|
</p></li><li><p>
|
|
<a href="http://www.sqlite.org/src/doc/trunk/src/test_onefile.c">test_onefile.c</a> -
|
|
This file implements a demonstration VFS named "fs" that shows how SQLite
|
|
can be used on an embedded device that lacks a filesystem. Content is
|
|
written directly to the underlying media. A VFS derived from this
|
|
demonstration code could be used by a gadget with a limited amount of
|
|
flash memory to make SQLite behave as the filesystem for the flash memory
|
|
on the device.
|
|
|
|
</p></li><li><p>
|
|
<a href="http://www.sqlite.org/src/doc/trunk/src/test_journal.c">test_journal.c</a> -
|
|
This file implements a shim used during SQLite testing that verifies that
|
|
the database and rollback journal are written in the correct order and
|
|
are "synced" at appropriate times in order to guarantee that the database
|
|
can recover from a power lose are hard reset at any time. The shim
|
|
checks several invariants on the operation of databases and rollback
|
|
journals and raises exceptions if any of those invariants are violated.
|
|
These invariants, in turn, assure that the database is always recoverable.
|
|
Running a large suite of test cases using this shim provides added
|
|
assurance that SQLite databases will not be damaged by unexpected
|
|
power failures or device resets.
|
|
|
|
</p></li><li><p>
|
|
<a href="http://www.sqlite.org/src/doc/trunk/src/test_vfs.c">test_vfs.c</a> -
|
|
This file implements a shim that can be used to simulate filesystem faults.
|
|
This shim is used during testing to verify that SQLite responses sanely
|
|
to hardware malfunctions or to other error conditions such as running out
|
|
of filesystem space that are difficult to test on a real system.
|
|
</p></li></ul>
|
|
|
|
<p>
|
|
There are other VFS implementations both in the core SQLite source code
|
|
library and in available extensions. The list above is not meant to be
|
|
exhaustive but merely representative of the kinds of features that can
|
|
be realized using the VFS interface.
|
|
</p>
|
|
|
|
<h1 id="vfs_implementations"><span>4. </span>VFS Implementations</h1>
|
|
|
|
<p>
|
|
A new VFS is implemented by subclassing three objects:
|
|
</p>
|
|
|
|
<ul>
|
|
<li><a href="c3ref/vfs.html">sqlite3_vfs</a>
|
|
</li><li><a href="c3ref/io_methods.html">sqlite3_io_methods</a>
|
|
</li><li><a href="c3ref/file.html">sqlite3_file</a>
|
|
</li></ul>
|
|
|
|
<p>
|
|
An <a href="c3ref/vfs.html">sqlite3_vfs</a> object defines the name of the VFS and the core
|
|
methods that implement the interface to the operating system, such
|
|
as checking for existence of files, deleting files, creating files
|
|
and opening and for reading and/or writing, converting filenames
|
|
into their canonical form. The <a href="c3ref/vfs.html">sqlite3_vfs</a> object also contains
|
|
methods for obtaining randomness from the operating system, for
|
|
suspending a process (sleeping) and for finding the current date and
|
|
time.
|
|
</p>
|
|
|
|
<p>
|
|
The <a href="c3ref/file.html">sqlite3_file</a> object represents an open file.
|
|
The xOpen method of <a href="c3ref/vfs.html">sqlite3_vfs</a> constructs an <a href="c3ref/file.html">sqlite3_file</a>
|
|
object when the file is opened. The <a href="c3ref/file.html">sqlite3_file</a> keeps track
|
|
of the state of the file while it is opened.
|
|
</p>
|
|
|
|
<p>
|
|
The <a href="c3ref/io_methods.html">sqlite3_io_methods</a> object holds the methods used to interact
|
|
with an open file. Each <a href="c3ref/file.html">sqlite3_file</a> contains a pointer to
|
|
an <a href="c3ref/io_methods.html">sqlite3_io_methods</a> object that is appropriate for the file it
|
|
represents. The <a href="c3ref/io_methods.html">sqlite3_io_methods</a> object contains methods to do
|
|
things such as read and write from the file, to truncate the file,
|
|
to flush any changes to persistent storage, to find the size of the
|
|
file, to lock and unlock the file, and to close file and destroy
|
|
the <a href="c3ref/file.html">sqlite3_file</a> object.
|
|
</p>
|
|
|
|
<p>
|
|
Writing the code for a new VFS involves constructing a subclass for
|
|
the <a href="c3ref/vfs.html">sqlite3_vfs</a> object and then registering that VFS object using
|
|
a call to <a href="c3ref/vfs_find.html">sqlite3_vfs_register()</a>. The VFS implementation also
|
|
provides subclasses for <a href="c3ref/file.html">sqlite3_file</a> and <a href="c3ref/io_methods.html">sqlite3_io_methods</a> but
|
|
those objects are not registered directly with SQLite. Instead, the
|
|
<a href="c3ref/file.html">sqlite3_file</a> object is returned from the xOpen method of
|
|
<a href="c3ref/vfs.html">sqlite3_vfs</a> and the <a href="c3ref/file.html">sqlite3_file</a> object points to an instance
|
|
of the <a href="c3ref/io_methods.html">sqlite3_io_methods</a> object.
|
|
</p>
|
|
|