# page_content

c2: Architecture of SQLite
c5: arch.html
c4:
Introduction


This document describes the architecture of the SQLite library.
The information here is useful to those who want to understand or
modify the inner workings of SQLite.







A nearby diagram shows the main components of SQLite
and how they interoperate.  The text below
explains the roles of the various components.


Overview

SQLite works by compiling SQL text into bytecode, then running
that bytecode using a virtual machine.

The sqlite3_prepare_v2() and related interfaces act as a compiler
for converting SQL text into bytecode.  The sqlite3_stmt object is
a container for a single bytecode program using to implement a single
SQL statement.  The sqlite3_step() interface passes a bytecode program
into the virtual machine, and runs the program until it either completes,
or forms a row of result to be returned, or hits a fatal error, or is
interrupted.

Interface

Much of the C-language Interface is found in source
files main.c, legacy.c, and
vdbeapi.c
though some routines are
scattered about in other files where they can have access to data 
structures with file scope.  
The sqlite3_get_table() routine is implemented in table.c.
The sqlite3_mprintf() routine is found in printf.c.
The sqlite3_complete() interface is in complete.c.
The TCL Interface is implemented by tclsqlite.c.

To avoid name collisions, all external
symbols in the SQLite library begin with the prefix sqlite3.
Those symbols that are intended for external use (in other words,
those symbols which form the API for SQLite) add an underscore, and
thus begin with sqlite3_.  Extension APIs sometimes add the
extension name prior to the underscore; for example:
sqlite3rbu_ or sqlite3session_.

Tokenizer

When a string containing SQL statements is to be evaluated it is
first sent to the tokenizer.
The tokenizer breaks
the SQL text into tokens and hands those tokens
one by one to the parser.  The tokenizer is hand-coded in 
the file tokenize.c.

Note that in this design, the tokenizer calls the parser.  People
who are familiar with YACC and BISON may be accustomed to doing things the
other way around &mdash; having the parser call the tokenizer.  Having
the tokenizer call the parser is better, though, because it can be made
threadsafe and it runs faster.

Parser

The parser assigns meaning to tokens based on
their context.  The parser for SQLite is generated using the
Lemon parser generator.
Lemon does the same job as YACC/BISON, but it uses
a different input syntax which is less error-prone.
Lemon also generates a parser which is reentrant and thread-safe.
And Lemon defines the concept of a non-terminal destructor so
that it does not leak memory when syntax errors are encountered.
The grammar file that drives Lemon and that defines the SQL language
that SQLite understands is found in parse.y.

Because
Lemon is a program not normally found on development machines, the
complete source code to Lemon (just one C file) is included in the
SQLite distribution in the "tool" subdirectory.


Code Generator

After the parser assembles tokens into a parse tree,
the code generator runs to analyze the parser tree and generate
bytecode that performs the work of the SQL statement.
The prepared statement object is a container for this bytecode.
There are many files in the code generator, including:
attach.c,
auth.c,
build.c,
delete.c,
expr.c,
insert.c,
pragma.c,
select.c,
trigger.c,
update.c,
vacuum.c,
where.c,
wherecode.c, and
whereexpr.c.
In these files is where most of the serious magic happens.
expr.c handles code generation for expressions.
where*.c handles code generation for WHERE clauses on
SELECT, UPDATE and DELETE statements.  The files attach.c,
delete.c, insert.c, select.c, 
trigger.c
update.c, and vacuum.c handle the code generation
for SQL statements with the same names.  (Each of these files calls routines
in expr.c and where.c as necessary.)  All other
SQL statements are coded out of build.c.
The auth.c file implements the functionality of
sqlite3_set_authorizer().

The code generator, and especially the logic in where*.c
and in select.c, is sometimes called the
query planner.  For any particular SQL statement, there might be
hundreds, thousands, or millions of different algorithms to compute
the answer.  The query planner is an AI that strives to select the
best algorithm from these millions of choices.

Bytecode Engine

The bytecode program created by the code generator is run by
a virtual machine.

The virtual machine itself is entirely contained in a single
source file vdbe.c.  The
vdbe.h header file defines an interface
between the virtual machine and the rest of the SQLite library and
vdbeInt.h which defines structures and interfaces that
are private the virtual machine itself.
Various other vdbe*.c files are helpers to the virtual machine.
The vdbeaux.c file contains utilities used by the virtual
machine and interface modules used by the rest of the library to
construct VM programs.  The vdbeapi.c file contains external
interfaces to the virtual machine such as the 
sqlite3_bind_int() and sqlite3_step().  Individual values
(strings, integer, floating point numbers, and BLOBs) are stored
in an internal object named "Mem" which is implemented by
vdbemem.c.


SQLite implements SQL functions using callbacks to C-language routines.
Even the built-in SQL functions are implemented this way.  Most of
the built-in SQL functions (ex: abs(), count(),
substr(), and so forth) can be found in func.c source
file.
Date and time conversion functions are found in date.c.
Some functions such as coalesce() and typeof() are implemented
as bytecode directly by the code generator.


B-Tree

An SQLite database is maintained on disk using a B-tree implementation
found in the btree.c source file.  A separate B-tree is used for
each table and index in the database.  All B-trees are stored in the
same disk file.  The file format details are stable and well-defined and
are guaranteed to be compatible moving forward.

The interface to the B-tree subsystem and the rest of the SQLite library
is defined by the header file btree.h.


Page Cache

The B-tree module requests information from the disk in fixed-size
pages.  The default page_size is 4096 bytes but can be any power of
two between 512 and 65536 bytes.
The page cache is responsible for reading, writing, and
caching these pages.
The page cache also provides the rollback and atomic commit abstraction
and takes care of locking of the database file.  The
B-tree driver requests particular pages from the page cache and notifies
the page cache when it wants to modify pages or commit or rollback
changes. The page cache handles all the messy details of making sure
the requests are handled quickly, safely, and efficiently.

The primary page cache implementation is in the
pager.c file.  WAL mode logic is in the separate
wal.c.  In-memory caching is implemented by the
pcache.c and pcache1.c files.
The interface between page cache subsystem
and the rest of SQLite is defined by the header file pager.h.


OS Interface


In order to provide portability between across operating systems,
SQLite uses abstract object called the VFS.  Each VFS provides methods
for opening, read, writing, and closing files on disk, and for other
OS-specific task such as finding the current time, or obtaining randomness
to initialize the built-in pseudo-random number generator.
SQLite currently provides VFSes for unix (in the os_unix.c
file) and Windows (in the os_win.c file).


Utilities


Memory allocation, caseless string comparison routines, 
portable text-to-number conversion routines, and other utilities
are located in util.c.
Symbol tables used by the parser are maintained by hash tables found
in hash.c.  The utf.c source file contains Unicode
conversion subroutines.
SQLite has its own private implementation of 
printf() (with
some extensions) in printf.c and its own
pseudo-random number generator (PRNG) in random.c.


Test Code


Files in the "src/" folder of the source tree whose names begin with
test are for testing only and are not included in a standard
build of the library.


c2: SQLite Is Transactional
c5: transactional.html
c4:
SQLite is Transactional

A transactional database is one in which all changes and queries
appear to be
Atomic, Consistent, Isolated, and Durable
(ACID).
SQLite implements 
serializable
transactions that are atomic, consistent, isolated, and durable,
even if the transaction is interrupted by a program crash, an
operating system crash, or a power failure to the computer.



We here restate and amplify the previous sentence for emphasis:
All changes within a single transaction in SQLite either occur
completely or not at all, even if the act of writing the change
out to the disk is interrupted by

a program crash,
an operating system crash, or
a power failure.




The claim of the previous paragraph is extensively checked in the
SQLite regression test suite using a special test harness that 
simulates the effects on a database file of operating system crashes 
and power failures.



Additional information


c2: Why Is SQLite Coded In C
c5: whyc.html
c4:
1. C Is Best


Note: Sections 2.0 and 3.0 of this article were added in response
to comments on 
Hacker News and
Reddit.



Since its inception on 2000-05-29, SQLite has been implemented in generic C.
C was and continues to be the best language for implementing a software
library like SQLite.  There are no plans to recode SQLite in any other
programming language at this time.


The reasons why C is the best language to implement SQLite include:



 Performance
 Compatibility
 Low-dependency
 Stability


1.1. Performance

An intensively used low-level library like SQLite needs to be fast.
(And SQLite is fast, see Internal Versus External BLOBs and
35% Faster Than The Filesystem for example.)

C is a great language for writing fast code.  C is sometimes
described as "portable assembly language".  It enables to developers
to code as close to the underlying hardware as possible while still
remaining portable across platforms.

Other programming languages sometimes claim to be "as fast as C".
But no other language claims to be faster than C for general-purpose
programming, because none are.

1.2. Compatibility

Nearly all systems have the ability to call libraries
written in C.  This is not true of other implementation languages.

So, for example, Android applications written in Java are able to
invoke SQLite (through an adaptor).  Maybe it would have been more
convenient for Android if SQLite had been coded in Java as that would
make the interface simpler.  However, on iPhone applications are coded
in Objective-C or Swift, neither of which have the ability to call
libraries written in Java.  Thus, SQLite would be unusable on iPhones
had it been written in Java.

1.3. Low-Dependency

Libraries written in C do not have a huge run-time dependency.
In its minimum configuration, SQLite requires only the following
routines from the standard C library:






 memcmp()
 memcpy()
 memmove()
 memset()


&nbsp;&nbsp;&nbsp;


 strcmp()
 strlen()
 strncmp()







In a more complete build, SQLite also uses library routines like
malloc() and free() and operating system interfaces for opening, reading,
writing, and closing files.  But even then, the number of dependencies
is very small.  Other "modern" language, in contrast, often require
multi-megabyte runtimes loaded with thousands and thousands of interfaces.

1.4. Stability


The C language is old and boring.
It is a well-known and well-understood language.
This is exactly what one wants when developing a module like SQLite.
Writing a small, fast, and reliable database engine is hard enough as it
is without the implementation language changing out from under you with
each update to the implementation language specification.

2. Why Isn't SQLite Coded In An Object-Oriented Language?


Some programmers cannot imagine developing a complex system like
SQLite in a language that is not "object oriented".  So why is
SQLite not coded in C++ or Java?



Libraries written in C++ or Java can generally only be used by
applications written in the same language. It is difficult to
get an application written in Haskell or Java to invoke a library
written in C++.  On the other hand, libraries written in C are
callable from any programming language.


Object-Oriented is a design pattern, not a programming language.
You can do object-oriented programming in any language you want,
including assembly language.  Some languages (ex: C++ or Java) make
object-oriented easier.  But you can still do object-oriented programming
in languages like C.


Object-oriented is not the only valid design pattern.
Many programmers have been taught to think purely in terms of
objects.  And, to be fair, objects are often a good way to
decompose a problem.  But objects are not the only way, and are
not always the best way to decompose a problem.  Sometimes good old
procedural code is easier to write, easier to maintain and understand,
and faster than object-oriented code.


When SQLite was first being developed, Java was a young and immature
language.  C++ was older, but was undergoing such growing pains that
it was difficult to find any two C++ compilers that worked the same
way.  So C was definitely a better choice back when SQLite was first
being developed.  The situation is less stark now, but there is little
to no benefit in recoding SQLite at this point.


3. Why Isn't SQLite Coded In A "Safe" Language?


There has lately been a lot of interest in "safe" programming languages
like Rust or Go in which it is impossible, or is at least difficult, to make
common programming errors like memory leaks or array overruns.  So the
question often arises as to why SQLite is not coded in a "safe" language.



None of the safe programming languages existed for the first 10 years
of SQLite's existence.  SQLite could be recoded in Go or Rust, but doing
so would probably introduce far more bugs than would be fixed, and it
seems also likely to result in slower code.


Safe programming languages solve the easy problems: memory leaks,
use-after-free errors, array overruns, etc.  Safe languages provide
no help beyond ordinary C code in solving the rather more difficult
problem of computing a correct answer to an SQL statement.


Safe languages are often touted for helping to prevent
security vulnerabilities. True enough, but SQLite is
not a particularly security-sensitive library.  If an application is
running untrusted and unverified SQL, then it already has much bigger
security issues (SQL injection) that no "safe" language will fix.

It is true that applications sometimes import complete binary SQLite
database files from untrusted sources, and such imports could present a
possible attack vector.  However, those code paths in SQLite are
limited and are extremely well tested.  And pre-validation routines 
are available to applications that want to read untrusted databases
that can help detect possible attacks prior to use.


Some "safe" languages (ex: Go) dislike the use of assert().  But
the use of assert() is a vital part of keeping SQLite maintainable.
The lack of assert() in Go is a show-stopper as far as the developers
of SQLite are concerned.  See the The Use Of assert() In SQLite
article for additional information.


Safe languages insert additional machine branches to do things like
verify that array accesses are in-bounds.  In correct code, those
branches are never taken.  That means that the machine code cannot
be 100% branch tested, which is an important component of SQLite's
quality strategy.


Safe languages usually want to abort if they encounter an out-of-memory
(OOM) situation.  SQLite is designed to recover gracefully from an OOM.
It is unclear how this could be accomplished in the current crop of
safe languages.


All of the existing safe languages are new.  The developers of SQLite
applaud the efforts of computer language researchers in trying to
develop languages that are easier to program safely.  We encourage these
efforts to continue.  Be we ourselves are more interested in old and
boring languages when it comes to implementing SQLite.



All that said, it is possible that SQLite might
one day be recoded in Rust.  Recoding SQLite in Go is unlikely
since Go hates assert().  But Rust is a possibility.  Some
preconditions that must occur before SQLite is recoded in Rust
include:



 Rust needs to mature a little more, stop changing so fast, and
     move further toward being old and boring.
 Rust needs to demonstrate that it can be used to create general-purpose
     libraries that are callable from all other programming languages.
 Rust needs to demonstrate that it can produce object code that
     works on obscure embedded devices, including devices that lack
     an operating system.
 Rust needs to pick up the necessary tooling that enables one to
     do 100% branch coverage testing of the compiled binaries.
 Rust needs a mechanism to recover gracefully from OOM errors.
 Rust needs to demonstrate that it can do the kinds of work that
     C does in SQLite without a significant speed penalty.



If you are a "rustacean" and feel that Rust already meets the
preconditions listed above, and that SQLite should be recoded in
Rust, then you are welcomed and encouraged
to contact the SQLite developers privately
and argue your case.


c2: Constraint Conflict Resolution in SQLite
c5: conflict.html
c4:
Constraint Conflict Resolution in SQLite


In most SQL databases, if you have a UNIQUE, NOT NULL, or
CHECK constraint on
a table and you try to do an UPDATE or INSERT that violates
the constraint, the database will abort the operation in
progress, back out any prior changes associated with the same
UPDATE or INSERT statement, and return an error.
This is the default behavior of SQLite, though SQLite also allows one to
define alternative ways for dealing with constraint violations.
This article describes those alternatives and how to use them.


Conflict Resolution Algorithms


SQLite defines five constraint conflict resolution algorithms
as follows:



ROLLBACK
When a constraint violation occurs, an immediate ROLLBACK
occurs, thus ending the current transaction, and the command aborts
with a return code of SQLITE_CONSTRAINT.  If no transaction is
active (other than the implied transaction that is created on every
command) then this algorithm works the same as ABORT.

ABORT
When a constraint violation occurs, the command backs out
any prior changes it might have made and aborts with a return code
of SQLITE_CONSTRAINT.  But no ROLLBACK is executed so changes
from prior commands within the same transaction
are preserved.  This is the default behavior for SQLite.

FAIL
When a constraint violation occurs, the command aborts with a
return code SQLITE_CONSTRAINT.  But any changes to the database that
the command made prior to encountering the constraint violation
are preserved and are not backed out.  For example, if an UPDATE
statement encountered a constraint violation on the 100th row that
it attempts to update, then the first 99 row changes are preserved
but change to rows 100 and beyond never occur.

IGNORE
When a constraint violation occurs, the one row that contains
the constraint violation is not inserted or changed.  But the command
continues executing normally.  Other rows before and after the row that
contained the constraint violation continue to be inserted or updated
normally.  No error is returned.

REPLACE
When a UNIQUE constraint violation occurs, the pre-existing row
that caused the constraint violation is removed prior to inserting
or updating the current row.  Thus the insert or update always occurs.
The command continues executing normally.  No error is returned.


c2: SQLite FTS5 Extension
c5: fts5.html
c4:
1. Overview of FTS5

FTS5 is an SQLite virtual table module that provides 
full-text search
functionality to database applications. In their most elementary form, 
full-text search engines allow the user to efficiently search a large 
collection of documents for the subset that contain one or more instances of a
search term. The search functionality provided to world wide web users by
Google is, among other things, a full-text search
engine, as it allows users to search for all documents on the web that contain,
for example, the term "fts5".

To use FTS5, the user creates an FTS5 virtual table with one or more
columns. For example:

CREATE VIRTUAL TABLE email USING fts5(sender, title, body);


It is an error to add types, constraints or PRIMARY KEY declarations to 
a CREATE VIRTUAL TABLE statement used to create an FTS5 table. Once created,
an FTS5 table may be populated using INSERT, UPDATE or DELETE statements
like any other table. Like any other table with no PRIMARY KEY declaration, an
FTS5 table has an implicit INTEGER PRIMARY KEY field named rowid. 

Not shown in the example above is that there are also 
various options that may be provided to FTS5 as
part of the CREATE VIRTUAL TABLE statement to configure various aspects of the
new table. These may be used to modify the way in which the FTS5 table extracts
terms from documents and queries, to create extra indexes on disk to speed up
prefix queries, or to create an FTS5 table that acts as an index on content
stored elsewhere.

Once populated, there are three ways to execute a full-text query against
the contents of an FTS5 table:

 Using a MATCH operator in the WHERE clause of a SELECT statement, or
     Using an equals ("=") operator in the WHERE clause of a SELECT statement, or
     using the table-valued function syntax.


If using the MATCH or = operators, the expression to the left of the MATCH
   operator is usually the name of the FTS5 table (the exception is when 
   specifying a column-filter). The expression on the right
   must be a text value specifying the term to search for. For the table-valued
   function syntax, the term to search for is specified as the first table argument.
   For example:

-- Query for all rows that contain at least once instance of the term
-- "fts5" (in any column). The following three queries are equivalent.
SELECT * FROM email WHERE email MATCH 'fts5';
SELECT * FROM email WHERE email = 'fts5';
SELECT * FROM email('fts5');


 By default, FTS5 full-text searches are case-independent. Like any other
SQL query that does not contain an ORDER BY clause, the example above returns
results in an arbitrary order. To sort results by relevance (most to least
relevant), an ORDER BY may be added to a full-text query as follows:

-- Query for all rows that contain at least once instance of the term
-- "fts5" (in any column). Return results in order from best to worst
-- match.  
SELECT * FROM email WHERE email MATCH 'fts5' ORDER BY rank;


 As well as the column values and rowid of a matching row, an application
may use FTS5 auxiliary functions to retrieve extra information regarding
the matched row. For example, an auxiliary function may be used to retrieve
a copy of a column value for a matched row with all instances of the matched
term surrounded by html &lt;b&gt;&lt;/b&gt; tags. Auxiliary functions are
invoked in the same way as SQLite scalar functions, except that the name
of the FTS5 table is specified as the first argument. For example:

-- Query for rows that match "fts5". Return a copy of the "body" column
-- of each row with the matches surrounded by &lt;b&gt;&lt;/b&gt; tags.
SELECT highlight(email, 2, '&lt;b&gt;', '&lt;/b&gt;') FROM email('fts5');


A description of the available auxiliary functions, and more details
regarding configuration of the special "rank" column, are 
available below. Custom auxiliary functions may also be implemented in C and registered with
FTS5, just as custom SQL functions may be registered with the SQLite core.

 As well as searching for all rows that contain a term, FTS5 allows 
the user to search for rows that contain:


   any terms that begin with a specified prefix,
   "phrases" - sequences of terms or prefix terms that must feature in a
       document for it to match the query, 
   sets of terms, prefix terms or phrases that appear within a specified
       proximity of each other (these are called "NEAR queries"), or
   boolean combinations of any of the above.


 Such advanced searches are requested by providing a more complicated 
FTS5 query string as the text to the right of the MATCH operator (or =
operator, or as the first argument to a table-valued function syntax). The 
full query syntax is described here.


2. Compiling and Using FTS5

2.1. Building FTS5 as part of SQLite

As of version 3.9.0 (2015-10-14),
FTS5 is included as part of the SQLite amalgamation.
If using one of the two autoconf build system, FTS5 is
enabled by specifying the "--enable-fts5" option when running the configure
script.  (FTS5 is currently disabled by default for the
source-tree configure script and enabled by default for
the amalgamation configure script, but these defaults might
change in the future.)

Or, if sqlite3.c is compiled using some other build system, by arranging for
the SQLITE_ENABLE_FTS5 pre-processor symbol to be defined.

2.2. Building a Loadable Extension

Alternatively, FTS5 may be built as a loadable extension.

The canonical FTS5 source code consists of a series of *.c and other files
in the "ext/fts5" directory of the SQLite source tree. A build process reduces
this to just two files - "fts5.c" and "fts5.h" - which may be used to build an
SQLite loadable extension.


   Obtain the latest SQLite code from fossil.
   Create a Makefile as described in How To Compile SQLite.
   Build the "fts5.c" target. Which also creates fts5.h.


$ wget -c http://www.sqlite.org/src/tarball/SQLite-trunk.tgz?uuid=trunk -O SQLite-trunk.tgz
.... output ...
$ tar -xzf SQLite-trunk.tgz
$ cd SQLite-trunk
$ ./configure && make fts5.c
... lots of output ...
$ ls fts5.&#91;ch]
fts5.c        fts5.h



  The code in "fts5.c" may then be compiled into a loadable extension or
  statically linked into an application as described in 
  Compiling Loadable Extensions. There are two entry points defined, both
  of which do the same thing:


   sqlite3_fts_init
   sqlite3_fts5_init



  The other file, "fts5.h", is not required to compile the FTS5 extension. 
  It is used by applications that implement custom FTS5 tokenizers or auxiliary functions.


3. Full-text Query Syntax


The following block contains a summary of the FTS query syntax in BNF form.
A detailed explanation follows.

&lt;phrase&gt;    := string &#91;*]
&lt;phrase&gt;    := &lt;phrase&gt; + &lt;phrase&gt;
&lt;neargroup&gt; := NEAR ( &lt;phrase&gt; &lt;phrase&gt; ... &#91;, N] )
&lt;query&gt;     := &#91; &#91;-] &lt;colspec&gt; :] &#91;&#94;] &lt;phrase&gt;
&lt;query&gt;     := &#91; &#91;-] &lt;colspec&gt; :] &lt;neargroup&gt;
&lt;query&gt;     := &#91; &#91;-] &lt;colspec&gt; :] ( &lt;query&gt; )
&lt;query&gt;     := &lt;query&gt; AND &lt;query&gt;
&lt;query&gt;     := &lt;query&gt; OR &lt;query&gt;
&lt;query&gt;     := &lt;query&gt; NOT &lt;query&gt;
&lt;colspec&gt;   := colname
&lt;colspec&gt;   := { colname1 colname2 ... }



3.1. FTS5 Strings

Within an FTS expression a string may be specified in one of two ways:


   By enclosing it in double quotes ("). Within a string, any embedded
       double quote characters may be escaped SQL-style - by adding a second
       double-quote character.

   As an FTS5 bareword that is not "AND", "OR" or "NOT" (case sensitive). 
       An FTS5 bareword is a string of one or more consecutive characters that
       are all either:
       
       
          Non-ASCII range characters (i.e. unicode codepoints greater 
              than 127), or 
          One of the 52 upper and lower case ASCII characters, or
          One of the 10 decimal digit ASCII characters, or
          The underscore character (unicode codepoint 96).
          The substitute character (unicode codepoint 26).
       

       Strings that include any other characters must be quoted. Characters
       that are not currently allowed in barewords, are not quote characters and
       do not currently serve any special purpose in FTS5 query expressions may
       at some point in the future be allowed in barewords or used to implement
       new query functionality. This means that queries that are currently
       syntax errors because they include such a character outside of a quoted
       string may be interpreted differently by some future version of FTS5.



3.2. FTS5 Phrases

FTS queries are made up of phrases. A phrase is an ordered list of 
one or more tokens. A string is transformed into a phrase by passing it to
the FTS table tokenizer. Two phrases can be concatenated into a single 
large phrase using the "+" operator. For example, assuming the tokenizer
module being used tokenizes the input "one.two.three" to three separate
tokens, the following four queries all specify the same phrase:

... MATCH '"one two three"'
... MATCH 'one + two + three'
... MATCH '"one two" + three'
... MATCH 'one.two.three'



A phrase matches a document if the document contains at least one sub-sequence
of tokens that matches the sequence of tokens that make up the phrase.


3.3. FTS5 Prefix Queries

If a "*" character follows a string within an FTS expression, then the final
token extracted from the string is marked as a prefix token. As you
might expect, a prefix token matches any document token of which it is a 
prefix. For example, the first two queries in the following block will match
any document that contains the token "one" immediately followed by the token
"two" and then any token that begins with "thr".

... MATCH '"one two thr" * '
... MATCH 'one + two + thr*'
... MATCH '"one two thr*"'      -- May not work as expected!


The final query in the block above may not work as expected. Because the
"*" character is inside the double-quotes, it will be passed to the tokenizer,
which will likely discard it (or perhaps, depending on the specific tokenizer
in use, include it as part of the final token) instead of recognizing it as
a special FTS character.




3.4. FTS5 Initial Token Queries

If a "&#94;" character appears immediately before a phrase that is not part of a
NEAR query, then that phrase only matches a document only if it starts at the
first token in a column. The "&#94;" syntax may be combined with a 
column filter, but may not be inserted into the middle of
a phrase.

... MATCH '&#94;one'              -- first token in any column must be "one"
... MATCH '&#94; one + two'       -- phrase "one two" must appear at start of a column
... MATCH '&#94; "one two"'       -- same as previous 
... MATCH 'a : &#94;two'          -- first token of column "a" must be "two"
... MATCH 'NEAR(&#94;one, two)'   -- syntax error! 
... MATCH 'one + &#94;two'        -- syntax error! 
... MATCH '"&#94;one two"'        -- May not work as expected!



3.5. FTS5 NEAR Queries

Two or more phrases may be grouped into a NEAR group. A NEAR group
is specified by the token "NEAR" (case sensitive) followed by an open
parenthesis character, followed by two or more whitespace separated phrases, optionally followed by a comma and the numeric parameter N, followed by
a close parenthesis. For example:

... MATCH 'NEAR("one two" "three four", 10)'
... MATCH 'NEAR("one two" thr* + four)'


If no N parameter is supplied, it defaults to 10. A NEAR group
matches a document if the document contains at least one clump of tokens that: 

 
   contains at least one instance of each phrase, and 
   for which the number of tokens between the end of the first phrase 
       and the beginning of the last phrase in the clump is less than or equal to N.


For example:

CREATE VIRTUAL TABLE f USING fts5(x);
INSERT INTO f(rowid, x) VALUES(1, 'A B C D x x x E F x');

... MATCH 'NEAR(e d, 4)';                      -- Matches!
... MATCH 'NEAR(e d, 3)';                      -- Matches!
... MATCH 'NEAR(e d, 2)';                      -- Does not match!

... MATCH 'NEAR("c d" "e f", 3)';              -- Matches!
... MATCH 'NEAR("c"   "e f", 3)';              -- Does not match!

... MATCH 'NEAR(a d e, 6)';                    -- Matches!
... MATCH 'NEAR(a d e, 5)';                    -- Does not match!

... MATCH 'NEAR("a b c d" "b c" "e f", 4)';    -- Matches!
... MATCH 'NEAR("a b c d" "b c" "e f", 3)';    -- Does not match!



3.6. FTS5 Column Filters


A single phrase or NEAR group may be restricted to matching text within a
specified column of the FTS table by prefixing it with the column name 
followed by a colon character. Or to a set of columns by prefixing it
with a whitespace separated list of column names enclosed in parenthesis
("curly brackets") followed by a colon character. Column names may be specified
using either of the two forms described for strings above. Unlike strings that
are part of phrases, column names are not passed to the tokenizer module.
Column names are case-insensitive in the usual way for SQLite column names -
upper/lower case equivalence is understood for ASCII-range characters only.

... MATCH 'colname : NEAR("one two" "three four", 10)'
... MATCH '"colname" : one + two + three'

... MATCH '{col1 col2} : NEAR("one two" "three four", 10)'
... MATCH '{col2 col1 col3} : one + two + three'



If a column filter specification is preceded by a "-" character, then
it is interpreted as a list of column not to match against. For example:

-- Search for matches in all columns except "colname"
... MATCH '- colname : NEAR("one two" "three four", 10)'

-- Search for matches in all columns except "col1", "col2" and "col3"
... MATCH '- {col2 col1 col3} : one + two + three'



Column filter specifications may also be applied to arbitrary expressions
enclosed in parenthesis. In this case the column filter applies to all 
phrases within the expression. Nested column filter operations may only 
further restrict the subset of columns matched, they can not be used to 
re-enable filtered columns. For example:

-- The following are equivalent:
... MATCH '{a b} : ( {b c} : "hello" AND "world" )'
... MATCH '(b : "hello") AND ({a b} : "world")'



Finally, a column filter for a single column may be specified by using
the column name as the LHS of a MATCH operator (instead of the usual
table name). For example:

-- Given the following table
CREATE VIRTUAL TABLE ft USING fts5(a, b, c);

-- The following are equivalent
SELECT * FROM ft WHERE b MATCH 'uvw AND xyz';
SELECT * FROM ft WHERE ft MATCH 'b : (uvw AND xyz)';

-- This query cannot match any rows (since all columns are filtered out): 
SELECT * FROM ft WHERE b MATCH 'a : xyz';



3.7. FTS5 Boolean Operators


Phrases and NEAR groups may be arranged into expressions using boolean
operators. In order of precedence, from highest (tightest grouping) to
lowest (loosest grouping), the operators are:


  Operator Function

  &lt;query1&gt; NOT &lt;query2&gt; 
      Matches if query1 matches and query2 does not match.

  &lt;query1&gt; AND &lt;query2&gt; 
      Matches if both query1 and query2 match.

  &lt;query1&gt; OR &lt;query2&gt; 
      Matches if either query1 or query2 match.




Parenthesis may be used to group expressions in order to modify operator
precedence in the usual ways. For example:

-- Matches documents that contain at least one instance of either "one"
-- or "two", but do not contain any instances of token "three".
... MATCH 'one OR two NOT three'

-- Match all documents that contain the token "two" but not "three", or
-- contain the token "one".
... MATCH 'one OR (two NOT three)'



Phrases and NEAR groups may also be connected by implicit AND operators.
For simplicity, these are not shown in the BNF grammar above. Essentially, any
sequence of phrases or NEAR groups (including those restricted to matching
specified columns) separated only by whitespace are handled as if there were an
implicit AND operator between each pair of phrases or NEAR groups. Implicit
AND operators are never inserted after or before an expression enclosed in
parenthesis. For example:

... MATCH 'one two three'         -- 'one AND two AND three'
... MATCH 'three "one two"'       -- 'three AND "one two"'
... MATCH 'NEAR(one two) three'   -- 'NEAR(one two) AND three'
... MATCH 'one OR two three'      -- 'one OR two AND three'

... MATCH '(one OR two) three'    -- Syntax error!
... MATCH 'func(one two)'         -- Syntax error!



4. FTS5 Table Creation and Initialization

Each argument specified as part of a "CREATE VIRTUAL TABLE ... USING fts5 
..." statement is either a column declaration or a configuration option. A
column declaration consists of one or more whitespace separated FTS5
barewords or string literals quoted in any manner acceptable to SQLite.

The first string or bareword in a column declaration is the column name. It
is an error to attempt to name an fts5 table column "rowid" or "rank", or to
assign the same name to a column as is used by the table itself. This is not
supported.

Each subsequent string or bareword in a column declaration is a column
option that modifies the behaviour of that column. Column options are
case-independent. Unlike the SQLite core, FTS5 considers unrecognized column
options to be errors. Currently, the only option recognized is 
"UNINDEXED" (see below).

A configuration option consists of an FTS5 bareword - the option name -
followed by an "=" character, followed by the option value. The option value is
specified using either a single FTS5 bareword or a string literal, again quoted
in any manner acceptable to the SQLite core. For example:

CREATE VIRTUAL TABLE mail USING fts5(sender, title, body, tokenize = 'porter ascii');


 There are currently the following configuration options:


   The "tokenize" option, used to configure a custom tokenizer.
   The "prefix" option, used to add prefix indexes
       to an FTS5 table.
   The "content" option, used to make the FTS5 table an 
       external content or contentless table.
   The "content_rowid" option, used to set the rowid field of an 
       external content table.
   The "columnsize" option, used to configure
       whether or not the size in tokens of each value in the FTS5 table is
       stored separately within the database.
   The "detail" option. This option may be used 
       to reduce the size of the FTS index on disk by omitting some information
       from it.  



4.1. The UNINDEXED column option

The contents of columns qualified with the UNINDEXED column option are not
added to the FTS index. This means that for the purposes of MATCH queries and
FTS5 auxiliary functions, the column contains no matchable tokens. 

For example, to avoid adding the contents of the "uuid" field to the FTS
index:
CREATE VIRTUAL TABLE customers USING fts5(name, addr, uuid UNINDEXED);



4.2. Prefix Indexes

 By default, FTS5 maintains a single index recording the location of each
token instance within the document set. This means that querying for complete
tokens is fast, as it requires a single lookup, but querying for a prefix 
token can be slow, as it requires a range scan. For example, to query for
the prefix token "abc*" requires a range scan of all tokens greater than
or equal to "abc" and less than "abd".

 A prefix index is a separate index that records the location of all
instances of prefix tokens of a certain length in characters used to speed
up queries for prefix tokens. For example, optimizing a query for prefix
token "abc*" requires a prefix index of three-character prefixes.

 To add prefix indexes to an FTS5 table, the "prefix" option is set to
either a single positive integer or a text value containing a white-space
separated list of one or more positive integer values. A prefix index is
created for each integer specified. If more than one "prefix" option is
specified as part of a single CREATE VIRTUAL TABLE statement, all apply.

-- Two ways to create an FTS5 table that maintains prefix indexes for
-- two and three character prefix tokens.
CREATE VIRTUAL TABLE ft USING fts5(a, b, prefix='2 3');
CREATE VIRTUAL TABLE ft USING fts5(a, b, prefix=2, prefix=3);



4.3. Tokenizers

 The CREATE VIRTUAL TABLE "tokenize" option is used to configure the
specific tokenizer used by the FTS5 table. The option argument must be either
an FTS5 bareword, or an SQL text literal. The text of the argument is itself
treated as a white-space series of one or more FTS5 barewords or SQL text
literals. The first of these is the name of the tokenizer to use. The second
and subsequent list elements, if they exist, are arguments passed to the
tokenizer implementation.

 Unlike option values and column names, SQL text literals intended as
tokenizers must be quoted using single quote characters. For example:

-- The following are all equivalent
CREATE VIRTUAL TABLE t1 USING fts5(x, tokenize = 'porter ascii');
CREATE VIRTUAL TABLE t1 USING fts5(x, tokenize = "porter ascii");
CREATE VIRTUAL TABLE t1 USING fts5(x, tokenize = "'porter' 'ascii'");
CREATE VIRTUAL TABLE t1 USING fts5(x, tokenize = '''porter'' ''ascii''');

-- But this will fail:
CREATE VIRTUAL TABLE t1 USING fts5(x, tokenize = '"porter" "ascii"');

-- This will fail too:
CREATE VIRTUAL TABLE t1 USING fts5(x, tokenize = 'porter' 'ascii');




FTS5 features three built-in tokenizer modules, described in subsequent
sections:


   The unicode61 tokenizer, based on the Unicode 6.1 standard. This
       is the default.

   The ascii tokenizer, which assumes all characters outside of
  the ASCII codepoint range (0-127) are to be treated as token characters.

   The porter tokenizer, which implements the 
porter stemming algorithm.


 It is also possible to create custom tokenizers for FTS5. The API for doing so is described here.

4.3.1. Unicode61 Tokenizer

 The unicode tokenizer classifies all unicode characters as either 
"separator" or "token" characters. By default all space and punctuation
characters, as defined by Unicode 6.1, are considered separators, and all 
other characters as token characters. More specifically, all unicode 
characters assigned to a 

general category beginning with "L" or "N" (letters and numbers,
specifically) or to category "Co" ("other, private use") are considered tokens.
All other characters are separators.
 
Each contiguous run of one or more token characters is considered to be a
token. The tokenizer is case-insensitive according to the rules defined by
Unicode 6.1.

 By default, diacritics are removed from all Latin script characters. This
means, for example, that "A", "a", "&#192;", "&#224;", "&#194;" and "&#226;"
are all considered to be equivalent.

 Any arguments following "unicode61" in the token specification are treated
as a list of alternating option names and values. Unicode61 supports the
following options:


   Option  Usage
   remove_diacritics
  This option should be set to "0", "1" or "2". The default value is "1".
  If it is set to "1" or "2", then diacritics are removed from Latin script
  characters as described above. However, if it is set to "1", then diacritics
  are not removed in the fairly uncommon case where a single unicode codepoint
  is used to represent a character with more that one diacritic. For example,
  diacritics are not removed from codepoint 0x1ED9 ("LATIN SMALL LETTER O WITH
  CIRCUMFLEX AND DOT BELOW"). This is technically a bug, but cannot be fixed
  without creating backwards compatibility problems. If this option is set to
  "2", then diacritics are correctly removed from all Latin characters.

   categories
  This option may be used to modify the set of Unicode general categories
  that are considered to correspond to token characters. The argument must
  consist of a space separated list of two-character general category
  abbreviations (e.g. "Lu" or "Nd"), or of the same with the second character
  replaced with an asterisk ("*"), interpreted as a glob pattern. The default
  value is "L* N* Co".

   tokenchars
   This option is used to specify additional unicode characters that 
  should be considered token characters, even if they are white-space or
  punctuation characters according to Unicode 6.1. All characters in the
  string that this option is set to are considered token characters.

   separators
   This option is used to specify additional unicode characters that 
  should be considered as separator characters, even if they are token
  characters according to Unicode 6.1. All characters in the string that 
  this option is set to are considered separators.


 For example:

-- Create an FTS5 table that does not remove diacritics from Latin
-- script characters, and that considers hyphens and underscore characters
-- to be part of tokens. 
CREATE VIRTUAL TABLE ft USING fts5(a, b, 
    tokenize = "unicode61 remove_diacritics 0 tokenchars '-_'"
);


 or:

-- Create an FTS5 table that, as well as the default token character classes,
-- considers characters in class "Mn" to be token characters.
CREATE VIRTUAL TABLE ft USING fts5(a, b, 
    tokenize = "unicode61 categories 'L* N* Co Mn'"
);


 The fts5 unicode61 tokenizer is byte-for-byte compatible with the fts3/4
unicode61 tokenizer.

4.3.2. Ascii Tokenizer

 The Ascii tokenizer is similar to the Unicode61 tokenizer, except that:


   All non-ASCII characters (those with codepoints greater than 127) are
  always considered token characters. If any non-ASCII characters are specified
  as part of the separators option, they are ignored.  

   Case-folding is only performed for ASCII characters. So while "A" and
  "a" are considered to be equivalent, "&#195;" and "&#227;" are distinct.

   The remove_diacritics option is not supported.


 For example:

-- Create an FTS5 table that uses the ascii tokenizer, but does not
-- consider numeric characters to be part of tokens.
CREATE VIRTUAL TABLE ft USING fts5(a, b, 
    tokenize = "ascii separators '0123456789'"
);


4.3.3. Porter Tokenizer

 The porter tokenizer is a wrapper tokenizer. It takes the output of some
other tokenizer and applies the 
porter stemming algorithm
to each token before it returns it to FTS5. This allows search terms like
"correction" to match similar words such as "corrected" or "correcting". The
porter stemmer algorithm is designed for use with English language terms 
only - using it with other languages may or may not improve search utility.

 By default, the porter tokenizer operates as a wrapper around the default
tokenizer (unicode61). Or, if one or more extra arguments are added to the
"tokenize" option following "porter", they are treated as a specification for
the underlying tokenizer that the porter stemmer uses. For example:

-- Two ways to create an FTS5 table that uses the porter tokenizer to
-- stem the output of the default tokenizer (unicode61). 
CREATE VIRTUAL TABLE t1 USING fts5(x, tokenize = porter); 
CREATE VIRTUAL TABLE t1 USING fts5(x, tokenize = 'porter unicode61');

-- A porter tokenizer used to stem the output of the unicode61 tokenizer,
-- with diacritics removed before stemming.
CREATE VIRTUAL TABLE t1 USING fts5(x, tokenize = 'porter unicode61 remove_diacritics 1');



4.4. External Content and Contentless Tables


Normally, when a row is inserted into an FTS5 table, as well as the various
full-text index entries and other data a copy of the row is stored in a private
table managed by the FTS5 module. When column values are requested from the
FTS5 table by the user or by an auxiliary function implementation, they are
read from this private table. The "content" option may be used to create an
FTS5 table that stores only FTS full-text index entries. Because the column
values themselves are usually much larger than the associated full-text index
entries, this can save significant database space.


There are two ways to use the "content" option:

   By setting it to an empty string to create a contentless FTS5 table. In
       this case FTS5 assumes that the original column values are unavailable
       to it when processing queries. Full-text queries and some auxiliary
       functions can still be used, but no column values apart from the rowid
       may be read from the table.

   By setting it to the name of a database object (table, virtual table or
       view) that may be queried by FTS5 at any time to retrieve the column
       values. This is known as an "external content" table. In this case all
       FTS5 functionality may be used, but it is the responsibility of the user
       to ensure that the contents of the full-text index are consistent with
       the named database object. If they are not, query results may be
       unpredictable.  



4.4.1. Contentless Tables

 A contentless FTS5 table is created by setting the "content" option to
an empty string. For example:

CREATE VIRTUAL TABLE f1 USING fts5(a, b, c, content='');


 Contentless FTS5 tables do not support UPDATE or DELETE statements, or
INSERT statements that do not supply a non-NULL value for the rowid field.
Contentless tables do not support REPLACE conflict handling. REPLACE 
and INSERT OR REPLACE statements are treated as regular INSERT statements.
Rows may be deleted from a contentless table using an FTS5 delete command.

 Attempting to read any column value except the rowid from a contentless
FTS5 table returns an SQL NULL value.


4.4.2. External Content Tables

 An external content FTS5 table is created by setting the content 
option to the name of a table, virtual table or view (hereafter the "content
table") within the same database. Whenever column values are required by
FTS5, it queries the content table as follows, with the rowid of the row
for which values are required bound to the SQL variable:

SELECT &lt;content_rowid&gt;, &lt;cols&gt; FROM &lt;content&gt; WHERE &lt;content_rowid&gt; = ?;


 In the above, &lt;content&gt; is replaced by the name of the content table.
By default, &lt;content_rowid&gt; is replaced by the literal text "rowid". Or,
if the "content_rowid" option is set within the CREATE VIRTUAL TABLE statement,
by the value of that option. &lt;cols&gt; is replaced by a comma-separated list
of the FTS5 table column names. For example:

-- If the database schema is: 
CREATE TABLE tbl (a, b, c, d INTEGER PRIMARY KEY);
CREATE VIRTUAL TABLE fts USING fts5(a, c, content=tbl, content_rowid=d);

-- Fts5 may issue queries such as:
SELECT d, a, c FROM tbl WHERE d = ?;


 The content table may also be queried as follows:

SELECT &lt;content_rowid&gt;, &lt;cols&gt; FROM &lt;content&gt; ORDER BY &lt;content_rowid&gt; ASC;
SELECT &lt;content_rowid&gt;, &lt;cols&gt; FROM &lt;content&gt; ORDER BY &lt;content_rowid&gt; DESC;


 It is still the responsibility of the user to ensure that the contents of
an external content FTS5 table are kept up to date with the content table. 
One way to do this is with triggers. For example:

-- Create a table. And an external content fts5 table to index it.
CREATE TABLE tbl(a INTEGER PRIMARY KEY, b, c);
CREATE VIRTUAL TABLE fts_idx USING fts5(b, c, content='tbl', content_rowid='a');

-- Triggers to keep the FTS index up to date.
CREATE TRIGGER tbl_ai AFTER INSERT ON tbl BEGIN
  INSERT INTO fts_idx(rowid, b, c) VALUES (new.a, new.b, new.c);
END;
CREATE TRIGGER tbl_ad AFTER DELETE ON tbl BEGIN
  INSERT INTO fts_idx(fts_idx, rowid, b, c) VALUES('delete', old.a, old.b, old.c);
END;
CREATE TRIGGER tbl_au AFTER UPDATE ON tbl BEGIN
  INSERT INTO fts_idx(fts_idx, rowid, b, c) VALUES('delete', old.a, old.b, old.c);
  INSERT INTO fts_idx(rowid, b, c) VALUES (new.a, new.b, new.c);
END;


 Like contentless tables, external content tables do not support REPLACE
conflict handling. Any operations that specify REPLACE conflict handling are
handled using ABORT.


4.5. The Columnsize Option

Normally, FTS5 maintains a special backing table within the database that
stores the size of each column value in tokens inserted into the main FTS5
table in a separate table. This backing table is used by the
xColumnSize API function, which is in turn used by
the built-in bm25 ranking function (and is likely to be useful 
to other ranking functions as well).

In order to save space, this backing table may be omitted by setting the
columnsize option to zero. For example:

-- A table without the xColumnSize() values stored on disk:
CREATE VIRTUAL TABLE ft USING fts5(a, b, c, columnsize=0);

-- Three equivalent ways of creating a table that does store the
-- xColumnSize() values on disk:
CREATE VIRTUAL TABLE ft USING fts5(a, b, c);
CREATE VIRTUAL TABLE ft USING fts5(a, b, c, columnsize=1);
CREATE VIRTUAL TABLE ft USING fts5(a, b, columnsize='1', c);


 It is an error to set the columnsize option to any value other than
0 or 1.

 If an FTS5 table is configured with columnsize=0 but is not a
contentless table, the xColumnSize API function
still works, but runs much more slowly. In this case, instead of reading
the value to return directly from the database, it reads the text value
itself and count the tokens within it on demand.

Or, if the table is also a contentless table,
then the following apply:


   The xColumnSize API always returns -1. There is no way to determine 
       the number of tokens in a value stored within a contentless FTS5 table
       configured with columnsize=0.

   Each inserted row must be accompanied by an explicitly specified rowid
       value. If a contentless table is configured with columnsize=0,
       attempting to insert a NULL value into the rowid is an SQLITE_MISMATCH
       error.

   All queries on the table must be full-text queries. In other words,
       they must use the MATCH or = operator with the table-name column as the
       left-hand operand, or else use the table-valued function syntax. Any
       query that is not a full-text query results in an error.


 The name of the table in which the xColumnSize values are stored
(unless columnsize=0 is specified) is "&lt;name&gt;_docsize", where 
&lt;name&gt; is the name of the FTS5 table itself. The 
sqlite3_analyzer
tool may be used on an existing database in order to determine how much
space might be saved by recreating an FTS5 table using columnsize=0.


4.6. The Detail Option

 For each term in a document, the FTS index maintained by FTS5 
stores the rowid of the document, the column number of the column that contains
the term and the offset of the term within the column value. The "detail"
option may be used to omit some of this information. This reduces the space
that the index consumes within the database file, but also reduces the
capability and efficiency of the system.

 The detail option may be set to "full" (the default value), "column" or
"none". For example:

-- The following two lines are equivalent (because the default value
-- of "detail" is "full". 
CREATE VIRTUAL TABLE ft1 USING fts5(a, b, c);
CREATE VIRTUAL TABLE ft1 USING fts5(a, b, c, detail=full);

CREATE VIRTUAL TABLE ft2 USING fts5(a, b, c, detail=column);
CREATE VIRTUAL TABLE ft3 USING fts5(a, b, c, detail=none);


If the detail option is set to column, then for each term the FTS
index records the rowid and column number only, omitting the term offset
information. This results in the following restrictions:


   NEAR queries are not available.
   Phrase queries are not available.
   Assuming the table is not also a 
  contentless table, the 
  xInstCount, xInst, 
  xPhraseFirst and xPhraseNext
  are slower than usual. This is because instead of reading the required data
  directly from the FTS index they have to load and tokenize the document text 
  on demand.
   If the table is also a contentless table, the xInstCount, xInst, 
  xPhraseFirst and xPhraseNext APIs behave as if the current row contains no
  phrase matches at all (i.e. xInstCount() returns 0).

  
If the detail option is set to none, then for each term the FTS
index records just the rowid is stored. Both column and offset information
are omitted. As well as the restrictions itemized above for detail=column
mode, this imposes the following extra limitations:


   Column filter queries are not available.
   Assuming the table is not also a contentless table, the 
  xPhraseFirstColumn and 
  xPhraseNextColumn are slower than usual. 

   If the table is also a contentless table, the xPhraseFirstColumn and
  xPhraseNextColumn APIs behave as if the current row contains no phrase
  matches at all (i.e. xPhraseFirstColumn() sets the iterator to EOF).


 In one test that indexed a large set of emails (1636 MiB on disk), the FTS
index was 743 MiB on disk with detail=full, 340 MiB with detail=column and 134
MiB with detail=none.


5.  Auxiliary Functions 

 Auxiliary functions are similar to SQL scalar functions,
except that they may only be used within full-text queries (those that use
the MATCH operator) on an FTS5 table. Their results are calculated based not
only on the arguments passed to them, but also on the current match and 
matched row. For example, an auxiliary function may return a numeric value
indicating the accuracy of the match (see the bm25() function), 
or a fragment of text from the matched row that contains one or more 
instances of the search terms (see the snippet() function).

To invoke an auxiliary function, the name of the FTS5 table should be
specified as the first argument. Other arguments may follow the first,
depending on the specific auxiliary function being invoked. For example, to
invoke the "highlight" function:

SELECT highlight(email, 2, '&lt;b&gt;', '&lt;/b&gt;') FROM email WHERE email MATCH 'fts5'


The built-in auxiliary functions provided as part of FTS5 are described in
the following section. Applications may also implement 
custom auxiliary functions in C.

5.1. Built-in Auxiliary Functions

 FTS5 provides three built-in auxiliary functions:


   The bm25() auxiliary function returns a real value
       reflecting the accuracy of the current match. Better matches are
       assigned numerically lower values.

   The highlight() auxiliary function returns a copy
       of the text from one of the columns of the current match with each
       instance of a queried term within the result surrounded by specified
       markup (for example "&lt;b&gt;" and "&lt;/b&gt;").

   The snippet() auxiliary function selects a short
       fragment of text from one of the columns of the matched row and returns
       it with each instance of a queried term surrounded by markup in
       the same manner as the highlight() function. The fragment of text is
       selected so as to maximize the number of queried terms it contains.



5.1.1. The bm25() function

 The built-in auxiliary function bm25() returns a real value indicating
how well the current row matches the full-text query. The better the match,
the numerically smaller the value returned. A query such as the following may
be used to return matches in order from best to worst match:

SELECT * FROM fts WHERE fts MATCH ? ORDER BY bm25(fts)


 In order to calculate a documents score, the full-text query is separated
    into its component phrases. The bm25 score for document D and 
    query Q is then calculated as follows:

 

 In the above, nPhrase is the number of phrases in the query.
    |D| is the number of tokens in the current document, and
    avgdl is the average number of tokens in all documents within the
    FTS5 table.  k1 and b are both constants,
    hard-coded at 1.2 and 0.75 respectively.

 The "-1" term at the start of the formula is not found in most
implementations of the BM25 algorithm. Without it, a better match is assigned
a numerically higher BM25 score. Since the default sorting order is
"ascending", this means that appending "ORDER BY bm25(fts)" to a query would
cause results to be returned in order from worst to best. The "DESC" keyword
would be required in order to return the best matches first. In order to
avoid this pitfall, the FTS5 implementation of BM25 multiplies the result
by -1 before returning it, ensuring that better matches are assigned
numerically lower scores.

 IDF(qi) is the inverse-document-frequency of query 
    phrase i. It is calculated as follows, where N is the total
    number of rows in the FTS5 table and n(qi) is the total
    number of rows that contain at least one instance of phrase i:

 

 Finally, f(qi,D) is the phrase frequency of phrase 
i. By default, this is simply the number of occurrences of the phrase
within the current row. However, by passing extra real value arguments to 
the bm25() SQL function, each column of the table may be assigned a different
weight and the phrase frequency calculated as follows:

 

 where wc is the weight assigned to column c and
n(qi,c) is the number of occurrences of phrase i in
column c of the current row. The first argument passed to bm25()
following the table name is the weight assigned to the leftmost column of
the FTS5 table. The second is the weight assigned to the second leftmost
column, and so on. If there are not enough arguments for all table columns,
remaining columns are assigned a weight of 1.0. If there are too many 
trailing arguments, the extras are ignored. For example:

-- Assuming the following schema:
CREATE VIRTUAL TABLE email USING fts5(sender, title, body);

-- Return results in bm25 order, with each phrase hit in the "sender"
-- column considered the equal of 10 hits in the "body" column, and
-- each hit in the "title" column considered as valuable as 5 hits in
-- the "body" column.
SELECT * FROM email WHERE email MATCH ? ORDER BY bm25(email, 10.0, 5.0);


Refer to wikipedia for 
more information regarding
BM25 and its variants.


5.1.2. The highlight() function

 The highlight() function returns a copy of the text from a specified 
column of the current row with extra markup text inserted to mark the start 
and end of phrase matches. 

The highlight() must be invoked with exactly three arguments following 
the table name. To be interpreted as follows:


   An integer indicating the index of the FTS table column to read the 
       text from. Columns are numbered from left to right starting at zero.

   The text to insert before each phrase match.

   The text to insert after each phrase match.


For example:

-- Return a copy of the text from the leftmost column of the current
-- row, with phrase matches marked using html "b" tags.
SELECT highlight(fts, 0, '&lt;b&gt;', '&lt;/b&gt;') FROM fts WHERE fts MATCH ?


In cases where two or more phrase instances overlap (share one or more
tokens in common), a single open and close marker is inserted for each set
of overlapping phrases. For example:

-- Assuming this:
CREATE VIRTUAL TABLE ft USING fts5(a);
INSERT INTO ft VALUES('a b c x c d e');
INSERT INTO ft VALUES('a b c c d e');
INSERT INTO ft VALUES('a b c d e');

-- The following SELECT statement returns these three rows:
--   '&#91;a b c&#93; x &#91;c d e&#93;'
--   '&#91;a b c&#93; &#91;c d e&#93;'
--   '&#91;a b c d e&#93;'
SELECT highlight(ft, 0, '&#91;', '&#93;') FROM ft WHERE ft MATCH 'a+b+c AND c+d+e';



5.1.3. The snippet() function

The snippet() function is similar to highlight(), except that instead of
returning entire column values, it automatically selects and extracts a
short fragment of document text to process and return. The snippet() function
must be passed five parameters following the table name argument:


   An integer indicating the index of the FTS table column to select
       the returned text from. Columns are numbered from left to right 
       starting at zero. A negative value indicates that the column should
       be automatically selected.

   The text to insert before each phrase match within the returned text.

   The text to insert after each phrase match within the returned text.

   The text to add to the start or end of the selected text to indicate
       that the returned text does not occur at the start or end of its column,
       respectively.

   The maximum number of tokens in the returned text. This must be greater
       than zero and equal to or less than 64. 



5.2. Sorting by Auxiliary Function Results

 All FTS5 tables feature a special hidden column named "rank". If the
current query is not a full-text query (i.e. if it does not include a MATCH
operator), the value of the "rank" column is always NULL. Otherwise, in a
full-text query, column rank contains by default the same value as would be
returned by executing the bm25() auxiliary function with no trailing 
arguments.

 The difference between reading from the rank column and using the bm25()
function directly within the query is only significant when sorting by the
returned value. In this case, using "rank" is faster than using bm25().

-- The following queries are logically equivalent. But the second may
-- be faster, particularly if the caller abandons the query before
-- all rows have been returned (or if the queries were modified to 
-- include LIMIT clauses).
SELECT * FROM fts WHERE fts MATCH ? ORDER BY bm25(fts);
SELECT * FROM fts WHERE fts MATCH ? ORDER BY rank;


 Instead of using bm25() with no trailing arguments, the specific auxiliary
function mapped to the rank column may be configured either on a per-query
basis, or by setting a different persistent default for the FTS table.

 In order to change the mapping of the rank column for a single query, 
a term similar to either of the following is added to the WHERE clause of a 
query:

rank MATCH 'auxiliary-function-name(arg1, arg2, ...)'
rank = 'auxiliary-function-name(arg1, arg2, ...)'


 The right-hand-side of the MATCH or = operator must be a constant
expression that evaluates to a string consisting of the auxiliary function to
invoke, followed by zero or more comma separated arguments within parenthesis.
Arguments must be SQL literals. For example:

-- The following queries are logically equivalent. But the second may
-- be faster. See above. 
SELECT * FROM fts WHERE fts MATCH ? ORDER BY bm25(fts, 10.0, 5.0);
SELECT * FROM fts WHERE fts MATCH ? AND rank MATCH 'bm25(10.0, 5.0)' ORDER BY rank;


 The table-valued function syntax may also be used to specify an alternative
ranking function. In this case the text describing the ranking function should
be specified as the second table-valued function argument. The following three
queries are equivalent:

SELECT * FROM fts WHERE fts MATCH ? AND rank MATCH 'bm25(10.0, 5.0)' ORDER BY rank;
SELECT * FROM fts WHERE fts = ? AND rank = 'bm25(10.0, 5.0)' ORDER BY rank;
SELECT * FROM fts WHERE fts(?, 'bm25(10.0, 5.0)') ORDER BY rank;


 The default mapping of the rank column for a table may be modified 
using the FTS5 rank configuration option.

6. Special INSERT Commands


6.1. The 'automerge' Configuration Option


  Instead of using a single data structure on disk to store the full-text
  index, FTS5 uses a series of b-trees. Each time a new transaction is
  committed, a new b-tree containing the contents of the committed transaction
  is written into the database file. When the full-text index is queried, each
  b-tree must be queried individually and the results merged before being
  returned to the user.


  In order to prevent the number of b-trees in the database from becoming too
  large (slowing down queries), smaller b-trees are periodically merged into
  single larger b-trees containing the same data. By default, this happens
  automatically within INSERT, UPDATE or DELETE statements that modify the
  full-text index. The 'automerge' parameter determines how many smaller
  b-trees are merged together at a time. Setting it to a small value can
  speed up queries (as they have to query and merge the results from fewer 
  b-trees), but can also slow down writing to the database (as each INSERT,
  UPDATE or DELETE statement has to do more work as part of the automatic
  merging process).


  Each of the b-trees that make up the full-text index is assigned to a "level"
  based on its size. Level-0 b-trees are the smallest, as they contain the
  contents of a single transaction. Higher level b-trees are the result of
  merging two or more level-0 b-trees together and so they are larger. FTS5
  begins to merge b-trees together once there exist M or more b-trees 
  with the same level, where M is the value of the 'automerge' 
  parameter.


  The maximum allowed value for the 'automerge' parameter is 16. The default
  value is 4. Setting the 'automerge' parameter to 0 disables the automatic 
  incremental merging of b-trees altogether.

INSERT INTO ft(ft, rank) VALUES('automerge', 8);


6.2. The 'crisismerge' Configuration Option

The 'crisismerge' option is similar to 'automerge', in that it determines
how and how often the component b-trees that make up the full-text index are
merged together. Once there exist C or more b-trees on a single level
within the full-text index, where C is the value of the 'crisismerge'
option, all b-trees on the level are immediately merged into a single b-tree.

The difference between this option and the 'automerge' option is that when
the 'automerge' limit is reached FTS5 only begins to merge the b-trees
together. Most of the work is performed as part of subsequent INSERT, 
UPDATE or DELETE operations. Whereas when the 'crisismerge' limit is reached,
the offending b-trees are all merged immediately. This means that an INSERT,
UPDATE or DELETE that triggers a crisis-merge may take a long time to 
complete.

The default 'crisismerge' value is 16. There is no maximum limit. Attempting
to set the 'crisismerge' parameter to a value of 0 or 1 is equivalent to
setting it to the default value (16). It is an error to attempt to set the
'crisismerge' option to a negative value.

INSERT INTO ft(ft, rank) VALUES('crisismerge', 16);



6.3. The 'delete' Command

 This command is only available with external content and contentless tables. It
is used to delete the index entries associated with a single row from the
full-text index. This command and the delete-all
command are the only ways to remove entries from the full-text index of a
contentless table.

 In order to use this command to delete a row, the text value 'delete' 
must be inserted into the special column with the same name as the table.
The rowid of the row to delete is inserted into the rowid column. The
values inserted into the other columns must match the values currently
stored in the table. For example:

-- Insert a row with rowid=14 into the fts5 table.
INSERT INTO ft(rowid, a, b, c) VALUES(14, $a, $b, $c);

-- Remove the same row from the fts5 table.
INSERT INTO ft(ft, rowid, a, b, c) VALUES('delete', 14, $a, $b, $c);


 If the values "inserted" into the text columns as part of a 'delete'
command are not the same as those currently stored within the table, the
results may be unpredictable.

 The reason for this is easy to understand: When a document is inserted
into the FTS5 table, an entry is added to the full-text index to record the
position of each token within the new document. When a document is removed,
the original data is required in order to determine the set of entries that
need to be removed from the full-text index. So if the data supplied to FTS5
when a row is deleted using this command is different from that used to
determine the set of token instances when it was inserted, some full-text 
index entries may not be correctly deleted, or FTS5 may try to remove index 
entries that do not exist. This can leave the full-text index in an
unpredictable state, making future query results unreliable.


6.4. The 'delete-all' Command

 This command is only available with external content and contentless tables. It
deletes all entries from the full-text index.

INSERT INTO ft(ft) VALUES('delete-all');


6.5. The 'integrity-check' Command

 This command is used to verify that the full-text index is consistent 
with the contents of the FTS5 table or content 
table. It is not available with contentless tables.

The integrity-check command is invoked by inserting the text value
'integrity-check' into the special column with the same name as the FTS5
table. For example:

INSERT INTO ft(ft) VALUES('integrity-check');


If the full-text index is consistent with the contents of the table, the
INSERT used to invoke the integrity-check command succeeds. Or, if any
discrepancy is found, it fails with an SQLITE_CORRUPT_VTAB error.


6.6. The 'merge' Command

INSERT INTO ft(ft, rank) VALUES('merge', 500);


 This command merges b-tree structures together until roughly N pages
of merged data have been written to the database, where N is the absolute
value of the parameter specified as part of the 'merge' command. The size of
each page is as configured by the FTS5 pgsz option.

 If the parameter is a positive value, B-tree structures are only eligible
for merging if one of the following is true:


   There are U or more such b-trees on a
       single level (see the documentation for the FTS5 automerge option
       for an explanation of b-tree levels), where U is the value assigned
       to the FTS5 usermerge option option.
   A merge has already been started (perhaps by a 'merge' command that
       specified a negative parameter).


 It is possible to tell whether or not the 'merge' command found any 
b-trees to merge together by checking the value returned by the
sqlite3_total_changes() API before and after the command is executed. If
the difference between the two values is 2 or greater, then work was performed.
If the difference is less than 2, then the 'merge' command was a no-op. In this
case there is no reason to execute the same 'merge' command again, at least
until after the FTS table is next updated.

 If the parameter is negative, and there are B-tree structures on more than
one level within the FTS index, all B-tree structures are assigned to the same
level before the merge operation is commenced. Additionally, if the parameter
is negative, the value of the usermerge configuration option is not 
respected - as few as two b-trees from the same level may be merged together.

 The above means that executing the 'merge' command with a negative
parameter until the before and after difference in the return value of
sqlite3_total_changes() is less than two optimizes the FTS index in the
same way as the FTS5 optimize command. However, if a new b-tree is added
to the FTS index while this process is ongoing, FTS5 will move the new 
b-tree to the same level as the existing b-trees and restart the merge. To
avoid this, only the first call to 'merge' should specify a negative parameter.
Each subsequent call to 'merge' should specify a positive value so that the
merge started by the first call is run to completion even if new b-trees are
added to the FTS index.


6.7. The 'optimize' Command

This command merges all individual b-trees that currently make up the
full-text index into a single large b-tree structure. This ensures that the
full-text index consumes the minimum space within the database and is in the
fastest form to query.

Refer to the documentation for the FTS5 automerge option for more details
regarding the relationship between the full-text index and its component
b-trees.

INSERT INTO ft(ft) VALUES('optimize');


Because it reorganizes the entire FTS index, the optimize command can 
take a long time to run. The FTS5 merge command can be used to divide
the work of optimizing the FTS index into multiple steps. To do this:


   Invoke the 'merge' command once with the parameter set to -N, then
   Invoke the 'merge' command zero or more times with the parameter set to N.


where N is the number of pages of data to merge within each invocation of
the merge command. The application should stop invoking merge when the
difference in the value returned by the sqlite3_total_changes() function before
and after the merge command drops to below two. The merge commands may be
issued as part of the same or separate transactions, and by the same or
different database clients. Refer to the documentation for the 
merge command for further details.


6.8. The 'pgsz' Configuration Option

 This command is used to set the persistent "pgsz" option.

 The full-text index maintained by FTS5 is stored as a series of fixed-size
blobs in a database table. It is not strictly necessary for all blobs that make
up a full-text index to be the same size. The pgsz option determines the size
of all blobs created by subsequent index writers. The default value is 1000.

INSERT INTO ft(ft, rank) VALUES('pgsz', 4072);



6.9. The 'rank' Configuration Option

 This command is used to set the persistent "rank" option.

 The rank option is used to change the default auxiliary function mapping
for the rank column. The option should be set to a text value in the same
format as described for "rank MATCH ?" terms 
above. For example:

INSERT INTO ft(ft, rank) VALUES('rank', 'bm25(10.0, 5.0)');



6.10. The 'rebuild' Command

 This command first deletes the entire full-text index, then rebuilds it
based on the contents of the table or content
table.  It is not available with contentless
tables.

INSERT INTO ft(ft) VALUES('rebuild');



6.11. The 'usermerge' Configuration Option

 This command is used to set the persistent "usermerge" option.

 The usermerge option is similar to the automerge and crisismerge options.
It is the minimum number of b-tree segments that will be merged together by
a 'merge' command with a positive parameter. For example:

INSERT INTO ft(ft, rank) VALUES('usermerge', 4);


 The default value of the usermerge option is 4. The minimum allowed value
is 2, and the maximum 16.


7. Extending FTS5

FTS5 features APIs allowing it to be extended by:


   Adding new auxiliary functions implemented in C, and
   Adding new tokenizers, also implemented in C.


 The built-in tokenizers and auxiliary functions described in this
document are all implemented using the publicly available API described
below.

 Before a new auxiliary function or tokenizer implementation may be 
registered with FTS5, an application must obtain a pointer to the "fts5_api"
structure. There is one fts5_api structure for each database connection with
which the FTS5 extension is registered. To obtain the pointer, the application
invokes the SQL user-defined function fts5() with a single argument.  That
argument must be set to a pointer to a pointer to an fts5_api object
using the sqlite3_bind_pointer() interface.
The following example code demonstrates the technique:

/*
** Return a pointer to the fts5_api pointer for database connection db.
** If an error occurs, return NULL and leave an error in the database 
** handle (accessible using sqlite3_errcode()/errmsg()).
*/
fts5_api *fts5_api_from_db(sqlite3 *db){
  fts5_api *pRet = 0;
  sqlite3_stmt *pStmt = 0;

  if( SQLITE_OK==sqlite3_prepare(db, "SELECT fts5(?1)", -1, &pStmt, 0) ){
    sqlite3_bind_pointer(pStmt, (void*)&pRet, "fts5_api_ptr", NULL);
    sqlite3_step(pStmt);
  }
  sqlite3_finalize(pStmt);
  return pRet;
}


Backwards Compatibility Warning:
Prior to SQLite version 3.20.0 (2017-08-01), the fts5() worked slightly
differently.  Older applications that extend FTS5 must be revised to use 
the new technique shown above.

 The fts5_api structure is defined as follows. It exposes three methods, 
one each for registering new auxiliary functions and tokenizers, and one for
retrieving existing tokenizer. The latter is intended to facilitate the
implementation of "tokenizer wrappers" similar to the built-in
porter tokenizer.

typedef struct fts5_api fts5_api;
struct fts5_api {
  int iVersion;                   /* Currently always set to 2 */

  /* Create a new tokenizer */
  int (*xCreateTokenizer)(
    fts5_api *pApi,
    const char *zName,
    void *pContext,
    fts5_tokenizer *pTokenizer,
    void (*xDestroy)(void*)
  );

  /* Find an existing tokenizer */
  int (*xFindTokenizer)(
    fts5_api *pApi,
    const char *zName,
    void **ppContext,
    fts5_tokenizer *pTokenizer
  );

  /* Create a new auxiliary function */
  int (*xCreateFunction)(
    fts5_api *pApi,
    const char *zName,
    void *pContext,
    fts5_extension_function xFunction,
    void (*xDestroy)(void*)
  );
};


 To invoke a method of the fts5_api object, the fts5_api pointer itself
should be passed as the methods first argument followed by the other, method
specific, arguments. For example:

rc = pFts5Api->xCreateTokenizer(pFts5Api, ... other args ...);


 The fts5_api structure methods are described individually in the following
sections.


7.1. Custom Tokenizers

 To create a custom tokenizer, an application must implement three
functions: a tokenizer constructor (xCreate), a destructor (xDelete) and a
function to do the actual tokenization (xTokenize). The type of each
function is as for the member variables of the fts5_tokenizer struct:

typedef struct Fts5Tokenizer Fts5Tokenizer;
typedef struct fts5_tokenizer fts5_tokenizer;
struct fts5_tokenizer {
  int (*xCreate)(void*, const char **azArg, int nArg, Fts5Tokenizer **ppOut);
  void (*xDelete)(Fts5Tokenizer*);
  int (*xTokenize)(Fts5Tokenizer*, 
      void *pCtx,
      int flags,            /* Mask of FTS5_TOKENIZE_* flags */
      const char *pText, int nText, 
      int (*xToken)(
        void *pCtx,         /* Copy of 2nd argument to xTokenize() */
        int tflags,         /* Mask of FTS5_TOKEN_* flags */
        const char *pToken, /* Pointer to buffer containing token */
        int nToken,         /* Size of token in bytes */
        int iStart,         /* Byte offset of token within input text */
        int iEnd            /* Byte offset of end of token within input text */
      )
  );
};

/* Flags that may be passed as the third argument to xTokenize() */
#define FTS5_TOKENIZE_QUERY     0x0001
#define FTS5_TOKENIZE_PREFIX    0x0002
#define FTS5_TOKENIZE_DOCUMENT  0x0004
#define FTS5_TOKENIZE_AUX       0x0008

/* Flags that may be passed by the tokenizer implementation back to FTS5
** as the third argument to the supplied xToken callback. */
#define FTS5_TOKEN_COLOCATED    0x0001      /* Same position as prev. token */


 The implementation is registered with the FTS5 module by calling the
xCreateTokenizer() method of the fts5_api object. If there is already a
tokenizer with the same name, it is replaced.  If a non-NULL xDestroy parameter
is passed to xCreateTokenizer(), it is invoked with a copy of the pContext
pointer passed as the only argument when the database handle is closed or when
the tokenizer is replaced.

 If successful, xCreateTokenizer() returns SQLITE_OK. Otherwise, it
returns an SQLite error code. In this case the xDestroy function is not 
invoked.

 When an FTS5 table uses the custom tokenizer, the FTS5 core calls xCreate()
once to create a tokenizer, then xTokenize() zero or more times to tokenize
strings, then xDelete() to free any resources allocated by xCreate(). More
specifically:


xCreate:
   This function is used to allocate and initialize a tokenizer instance.
   A tokenizer instance is required to actually tokenize text.

   The first argument passed to this function is a copy of the (void*)
   pointer provided by the application when the fts5_tokenizer object
   was registered with FTS5 (the third argument to xCreateTokenizer()). 
   The second and third arguments are an array of nul-terminated strings
   containing the tokenizer arguments, if any, specified following the
   tokenizer name as part of the CREATE VIRTUAL TABLE statement used
   to create the FTS5 table.

   The final argument is an output variable. If successful, (*ppOut) 
   should be set to point to the new tokenizer handle and SQLITE_OK
   returned. If an error occurs, some value other than SQLITE_OK should
   be returned. In this case, fts5 assumes that the final value of *ppOut 
   is undefined.

 xDelete:
   This function is invoked to delete a tokenizer handle previously
   allocated using xCreate(). Fts5 guarantees that this function will
   be invoked exactly once for each successful call to xCreate().

 xTokenize:
   This function is expected to tokenize the nText byte string indicated 
   by argument pText. pText may or may not be nul-terminated. The first
   argument passed to this function is a pointer to an Fts5Tokenizer object
   returned by an earlier call to xCreate().

   The second argument indicates the reason that FTS5 is requesting
   tokenization of the supplied text. This is always one of the following
   four values:

    FTS5_TOKENIZE_DOCUMENT - A document is being inserted into
            or removed from the FTS table. The tokenizer is being invoked to
            determine the set of tokens to add to (or delete from) the
            FTS index.

        FTS5_TOKENIZE_QUERY - A MATCH query is being executed 
            against the FTS index. The tokenizer is being called to tokenize 
            a bareword or quoted string specified as part of the query.

        (FTS5_TOKENIZE_QUERY | FTS5_TOKENIZE_PREFIX) - Same as
            FTS5_TOKENIZE_QUERY, except that the bareword or quoted string is
            followed by a "*" character, indicating that the last token
            returned by the tokenizer will be treated as a token prefix.

        FTS5_TOKENIZE_AUX - The tokenizer is being invoked to 
            satisfy an fts5_api.xTokenize() request made by an auxiliary
            function. Or an fts5_api.xColumnSize() request made by the same
            on a columnsize=0 database.  
   

   For each token in the input string, the supplied callback xToken() must
   be invoked. The first argument to it should be a copy of the pointer
   passed as the second argument to xTokenize(). The third and fourth
   arguments are a pointer to a buffer containing the token text, and the
   size of the token in bytes. The 4th and 5th arguments are the byte offsets
   of the first byte of and first byte immediately following the text from
   which the token is derived within the input.

   The second argument passed to the xToken() callback ("tflags") should
   normally be set to 0. The exception is if the tokenizer supports 
   synonyms. In this case see the discussion below for details.

   FTS5 assumes the xToken() callback is invoked for each token in the 
   order that they occur within the input text.

   If an xToken() callback returns any value other than SQLITE_OK, then
   the tokenization should be abandoned and the xTokenize() method should
   immediately return a copy of the xToken() return value. Or, if the
   input buffer is exhausted, xTokenize() should return SQLITE_OK. Finally,
   if an error occurs with the xTokenize() implementation itself, it
   may abandon the tokenization and return any error code other than
   SQLITE_OK or SQLITE_DONE.

7.1.1. Synonym Support

   Custom tokenizers may also support synonyms. Consider a case in which a
   user wishes to query for a phrase such as "first place". Using the 
   built-in tokenizers, the FTS5 query 'first + place' will match instances
   of "first place" within the document set, but not alternative forms
   such as "1st place". In some applications, it would be better to match
   all instances of "first place" or "1st place" regardless of which form
   the user specified in the MATCH query text.

   There are several ways to approach this in FTS5:

    By mapping all synonyms to a single token. In this case, using
            the above example, this means that the tokenizer returns the
            same token for inputs "first" and "1st". Say that token is in
            fact "first", so that when the user inserts the document "I won
            1st place" entries are added to the index for tokens "i", "won",
            "first" and "place". If the user then queries for '1st + place',
            the tokenizer substitutes "first" for "1st" and the query works
            as expected.

        By querying the index for all synonyms of each query term
            separately. In this case, when tokenizing query text, the
            tokenizer may provide multiple synonyms for a single term 
            within the document. FTS5 then queries the index for each 
            synonym individually. For example, faced with the query:

   ... MATCH 'first place'


            the tokenizer offers both "1st" and "first" as synonyms for the
            first token in the MATCH query and FTS5 effectively runs a query 
            similar to:

   ... MATCH '(first OR 1st) place'


            except that, for the purposes of auxiliary functions, the query
            still appears to contain just two phrases - "(first OR 1st)" 
            being treated as a single phrase.

        By adding multiple synonyms for a single term to the FTS index.
            Using this method, when tokenizing document text, the tokenizer
            provides multiple synonyms for each token. So that when a 
            document such as "I won first place" is tokenized, entries are
            added to the FTS index for "i", "won", "first", "1st" and
            "place".

            This way, even if the tokenizer does not provide synonyms
            when tokenizing query text (it should not - to do so would be
            inefficient), it doesn't matter if the user queries for 
            'first + place' or '1st + place', as there are entries in the
            FTS index corresponding to both forms of the first token.
   

   Whether it is parsing document or query text, any call to xToken that
   specifies a tflags argument with the FTS5_TOKEN_COLOCATED bit
   is considered to supply a synonym for the previous token. For example,
   when parsing the document "I won first place", a tokenizer that supports
   synonyms would call xToken() 5 times, as follows:

   xToken(pCtx, 0, "i",                      1,  0,  1);
xToken(pCtx, 0, "won",                    3,  2,  5);
xToken(pCtx, 0, "first",                  5,  6, 11);
xToken(pCtx, FTS5_TOKEN_COLOCATED, "1st", 3,  6, 11);
xToken(pCtx, 0, "place",                  5, 12, 17);


   It is an error to specify the FTS5_TOKEN_COLOCATED flag the first time
   xToken() is called. Multiple synonyms may be specified for a single token
   by making multiple calls to xToken(FTS5_TOKEN_COLOCATED) in sequence. 
   There is no limit to the number of synonyms that may be provided for a
   single token.

   In many cases, method (1) above is the best approach. It does not add 
   extra data to the FTS index or require FTS5 to query for multiple terms,
   so it is efficient in terms of disk space and query speed. However, it
   does not support prefix queries very well. If, as suggested above, the
   token "first" is substituted for "1st" by the tokenizer, then the query:

   ... MATCH '1s*'


   will not match documents that contain the token "1st" (as the tokenizer
   will probably not map "1s" to any prefix of "first").

   For full prefix support, method (3) may be preferred. In this case, 
   because the index contains entries for both "first" and "1st", prefix
   queries such as 'fi*' or '1s*' will match correctly. However, because
   extra entries are added to the FTS index, this method uses more space
   within the database.

   Method (2) offers a midpoint between (1) and (3). Using this method,
   a query such as '1s*' will match documents that contain the literal 
   token "1st", but not "first" (assuming the tokenizer is not able to
   provide synonyms for prefixes). However, a non-prefix query like '1st'
   will match against "1st" and "first". This method does not require
   extra disk space, as no extra entries are added to the FTS index. 
   On the other hand, it may require more CPU cycles to run MATCH queries,
   as separate queries of the FTS index are required for each synonym.

   When using methods (2) or (3), it is important that the tokenizer only
   provide synonyms when tokenizing document text (method (2)) or query
   text (method (3)), not both. Doing so will not cause any errors, but is
   inefficient.




7.2. Custom Auxiliary Functions

 Implementing a custom auxiliary function is similar to implementing a
scalar SQL function. The implementation
should be a C function of type fts5_extension_function, defined as follows:

typedef struct Fts5ExtensionApi Fts5ExtensionApi;
typedef struct Fts5Context Fts5Context;
typedef struct Fts5PhraseIter Fts5PhraseIter;

typedef void (*fts5_extension_function)(
  const Fts5ExtensionApi *pApi,   /* API offered by current FTS version */
  Fts5Context *pFts,              /* First arg to pass to pApi functions */
  sqlite3_context *pCtx,          /* Context for returning result/error */
  int nVal,                       /* Number of values in apVal[] array */
  sqlite3_value **apVal           /* Array of trailing arguments */
);


 The implementation is registered with the FTS5 module by calling the
xCreateFunction() method of the fts5_api object. If there is already an
auxiliary function with the same name, it is replaced by the new function.
If a non-NULL xDestroy parameter is passed to xCreateFunction(), it is invoked
with a copy of the pContext pointer passed as the only argument when the
database handle is closed or when the registered auxiliary function is
replaced.

 If successful, xCreateFunction() returns SQLITE_OK. Otherwise, it
returns an SQLite error code. In this case the xDestroy function is not 
invoked.

 The final three arguments passed to the auxiliary function callback are
similar to the three arguments passed to the implementation of a scalar SQL
function. All arguments except the first passed to the auxiliary function are
available to the implementation in the apVal&#91;&#93; array. The
implementation should return a result or error via the content handle pCtx.

 The first argument passed to an auxiliary function callback is a pointer
to a structure containing methods that may be invoked in order to obtain
information regarding the current query or row. The second argument is an
opaque handle that should be passed as the first argument to any such method 
invocation. For example, the following auxiliary function definition returns
the total number of tokens in all columns of the current row:

/*
** Implementation of an auxiliary function that returns the number
** of tokens in the current row (including all columns).
*/
static void column_size_imp(
  const Fts5ExtensionApi *pApi,
  Fts5Context *pFts,
  sqlite3_context *pCtx,
  int nVal,
  sqlite3_value **apVal
){
  int rc;
  int nToken;
  rc = pApi->xColumnSize(pFts, -1, &nToken);
  if( rc==SQLITE_OK ){
    sqlite3_result_int(pCtx, nToken);
  }else{
    sqlite3_result_error_code(pCtx, rc);
  }
}


The following section describes the API offered to auxiliary function
implementations in detail. Further examples may be found in the "fts5_aux.c"
file of the source code.


7.2.1. 
  Custom Auxiliary Functions API Reference


struct Fts5ExtensionApi {
  int iVersion;                   /* Currently always set to 3 */

  void *(*xUserData)(Fts5Context*);

  int (*xColumnCount)(Fts5Context*);
  int (*xRowCount)(Fts5Context*, sqlite3_int64 *pnRow);
  int (*xColumnTotalSize)(Fts5Context*, int iCol, sqlite3_int64 *pnToken);

  int (*xTokenize)(Fts5Context*, 
    const char *pText, int nText, /* Text to tokenize */
    void *pCtx,                   /* Context passed to xToken() */
    int (*xToken)(void*, int, const char*, int, int, int)       /* Callback */
  );

  int (*xPhraseCount)(Fts5Context*);
  int (*xPhraseSize)(Fts5Context*, int iPhrase);

  int (*xInstCount)(Fts5Context*, int *pnInst);
  int (*xInst)(Fts5Context*, int iIdx, int *piPhrase, int *piCol, int *piOff);

  sqlite3_int64 (*xRowid)(Fts5Context*);
  int (*xColumnText)(Fts5Context*, int iCol, const char **pz, int *pn);
  int (*xColumnSize)(Fts5Context*, int iCol, int *pnToken);

  int (*xQueryPhrase)(Fts5Context*, int iPhrase, void *pUserData,
    int(*)(const Fts5ExtensionApi*,Fts5Context*,void*)
  );
  int (*xSetAuxdata)(Fts5Context*, void *pAux, void(*xDelete)(void*));
  void *(*xGetAuxdata)(Fts5Context*, int bClear);

  int (*xPhraseFirst)(Fts5Context*, int iPhrase, Fts5PhraseIter*, int*, int*);
  void (*xPhraseNext)(Fts5Context*, Fts5PhraseIter*, int *piCol, int *piOff);

  int (*xPhraseFirstColumn)(Fts5Context*, int iPhrase, Fts5PhraseIter*, int*);
  void (*xPhraseNextColumn)(Fts5Context*, Fts5PhraseIter*, int *piCol);
};




void *(*xUserData)(Fts5Context*)

Return a copy of the context pointer the extension function was 
   registered with.



int (*xColumnTotalSize)(Fts5Context*, int iCol, sqlite3_int64 *pnToken)

If parameter iCol is less than zero, set output variable *pnToken
   to the total number of tokens in the FTS5 table. Or, if iCol is
   non-negative but less than the number of columns in the table, return
   the total number of tokens in column iCol, considering all rows in 
   the FTS5 table.



   If parameter iCol is greater than or equal to the number of columns
   in the table, SQLITE_RANGE is returned. Or, if an error occurs (e.g.
   an OOM condition or IO error), an appropriate SQLite error code is 
   returned.



int (*xColumnCount)(Fts5Context*)

Return the number of columns in the table.



int (*xColumnSize)(Fts5Context*, int iCol, int *pnToken)

If parameter iCol is less than zero, set output variable *pnToken
   to the total number of tokens in the current row. Or, if iCol is
   non-negative but less than the number of columns in the table, set
   *pnToken to the number of tokens in column iCol of the current row.



   If parameter iCol is greater than or equal to the number of columns
   in the table, SQLITE_RANGE is returned. Or, if an error occurs (e.g.
   an OOM condition or IO error), an appropriate SQLite error code is 
   returned.



   This function may be quite inefficient if used with an FTS5 table
   created with the "columnsize=0" option.



int (*xColumnText)(Fts5Context*, int iCol, const char **pz, int *pn)

This function attempts to retrieve the text of column iCol of the
   current document. If successful, (*pz) is set to point to a buffer
   containing the text in utf-8 encoding, (*pn) is set to the size in bytes
   (not characters) of the buffer and SQLITE_OK is returned. Otherwise,
   if an error occurs, an SQLite error code is returned and the final values
   of (*pz) and (*pn) are undefined.



int (*xPhraseCount)(Fts5Context*)

Returns the number of phrases in the current query expression.



int (*xPhraseSize)(Fts5Context*, int iPhrase)

Returns the number of tokens in phrase iPhrase of the query. Phrases
   are numbered starting from zero.



int (*xInstCount)(Fts5Context*, int *pnInst)

Set *pnInst to the total number of occurrences of all phrases within
   the query within the current row. Return SQLITE_OK if successful, or
   an error code (i.e. SQLITE_NOMEM) if an error occurs.



   This API can be quite slow if used with an FTS5 table created with the
   "detail=none" or "detail=column" option. If the FTS5 table is created 
   with either "detail=none" or "detail=column" and "content=" option 
   (i.e. if it is a contentless table), then this API always returns 0.



int (*xInst)(Fts5Context*, int iIdx, int *piPhrase, int *piCol, int *piOff)

Query for the details of phrase match iIdx within the current row.
   Phrase matches are numbered starting from zero, so the iIdx argument
   should be greater than or equal to zero and smaller than the value
   output by xInstCount().



   Usually, output parameter *piPhrase is set to the phrase number, *piCol
   to the column in which it occurs and *piOff the token offset of the
   first token of the phrase. Returns SQLITE_OK if successful, or an error
   code (i.e. SQLITE_NOMEM) if an error occurs.



   This API can be quite slow if used with an FTS5 table created with the
   "detail=none" or "detail=column" option.



sqlite3_int64 (*xRowid)(Fts5Context*)

Returns the rowid of the current row.



int (*xTokenize)(Fts5Context*, 
    const char *pText, int nText, 
    void *pCtx,                   
    int (*xToken)(void*, int, const char*, int, int, int)       
)

Tokenize text using the tokenizer belonging to the FTS5 table.



int (*xQueryPhrase)(Fts5Context*, int iPhrase, void *pUserData,
    int(*)(const Fts5ExtensionApi*,Fts5Context*,void*)
)

This API function is used to query the FTS table for phrase iPhrase
   of the current query. Specifically, a query equivalent to:


... FROM ftstable WHERE ftstable MATCH $p ORDER BY rowid



   with $p set to a phrase equivalent to the phrase iPhrase of the
   current query is executed. Any column filter that applies to
   phrase iPhrase of the current query is included in $p. For each 
   row visited, the callback function passed as the fourth argument 
   is invoked. The context and API objects passed to the callback 
   function may be used to access the properties of each matched row.
   Invoking Api.xUserData() returns a copy of the pointer passed as 
   the third argument to pUserData.



   If the callback function returns any value other than SQLITE_OK, the
   query is abandoned and the xQueryPhrase function returns immediately.
   If the returned value is SQLITE_DONE, xQueryPhrase returns SQLITE_OK.
   Otherwise, the error code is propagated upwards.



   If the query runs to completion without incident, SQLITE_OK is returned.
   Or, if some error occurs before the query completes or is aborted by
   the callback, an SQLite error code is returned.



int (*xSetAuxdata)(Fts5Context*, void *pAux, void(*xDelete)(void*))

Save the pointer passed as the second argument as the extension function's 
   "auxiliary data". The pointer may then be retrieved by the current or any
   future invocation of the same fts5 extension function made as part of
   the same MATCH query using the xGetAuxdata() API.



   Each extension function is allocated a single auxiliary data slot for
   each FTS query (MATCH expression). If the extension function is invoked 
   more than once for a single FTS query, then all invocations share a 
   single auxiliary data context.



   If there is already an auxiliary data pointer when this function is
   invoked, then it is replaced by the new pointer. If an xDelete callback
   was specified along with the original pointer, it is invoked at this
   point.



   The xDelete callback, if one is specified, is also invoked on the
   auxiliary data pointer after the FTS5 query has finished.



   If an error (e.g. an OOM condition) occurs within this function,
   the auxiliary data is set to NULL and an error code returned. If the
   xDelete parameter was not NULL, it is invoked on the auxiliary data
   pointer before returning.



void *(*xGetAuxdata)(Fts5Context*, int bClear)

Returns the current auxiliary data pointer for the fts5 extension 
   function. See the xSetAuxdata() method for details.



   If the bClear argument is non-zero, then the auxiliary data is cleared
   (set to NULL) before this function returns. In this case the xDelete,
   if any, is not invoked.



int (*xRowCount)(Fts5Context*, sqlite3_int64 *pnRow)

This function is used to retrieve the total number of rows in the table.
   In other words, the same value that would be returned by:


SELECT count(*) FROM ftstable;



int (*xPhraseFirst)(Fts5Context*, int iPhrase, Fts5PhraseIter*, int*, int*)

This function is used, along with type Fts5PhraseIter and the xPhraseNext
   method, to iterate through all instances of a single query phrase within
   the current row. This is the same information as is accessible via the
   xInstCount/xInst APIs. While the xInstCount/xInst APIs are more convenient
   to use, this API may be faster under some circumstances. To iterate 
   through instances of phrase iPhrase, use the following code:


Fts5PhraseIter iter;
int iCol, iOff;
for(pApi->xPhraseFirst(pFts, iPhrase, &iter, &iCol, &iOff);
    iCol>=0;
    pApi->xPhraseNext(pFts, &iter, &iCol, &iOff)
){
  // An instance of phrase iPhrase at offset iOff of column iCol
}



   The Fts5PhraseIter structure is defined above. Applications should not
   modify this structure directly - it should only be used as shown above
   with the xPhraseFirst() and xPhraseNext() API methods (and by
   xPhraseFirstColumn() and xPhraseNextColumn() as illustrated below).



   This API can be quite slow if used with an FTS5 table created with the
   "detail=none" or "detail=column" option. If the FTS5 table is created 
   with either "detail=none" or "detail=column" and "content=" option 
   (i.e. if it is a contentless table), then this API always iterates
   through an empty set (all calls to xPhraseFirst() set iCol to -1).



void (*xPhraseNext)(Fts5Context*, Fts5PhraseIter*, int *piCol, int *piOff)

See xPhraseFirst above.



int (*xPhraseFirstColumn)(Fts5Context*, int iPhrase, Fts5PhraseIter*, int*)

This function and xPhraseNextColumn() are similar to the xPhraseFirst()
   and xPhraseNext() APIs described above. The difference is that instead
   of iterating through all instances of a phrase in the current row, these
   APIs are used to iterate through the set of columns in the current row
   that contain one or more instances of a specified phrase. For example:


Fts5PhraseIter iter;
int iCol;
for(pApi->xPhraseFirstColumn(pFts, iPhrase, &iter, &iCol);
    iCol>=0;
    pApi->xPhraseNextColumn(pFts, &iter, &iCol)
){
  // Column iCol contains at least one instance of phrase iPhrase
}



   This API can be quite slow if used with an FTS5 table created with the
   "detail=none" option. If the FTS5 table is created with either 
   "detail=none" "content=" option (i.e. if it is a contentless table), 
   then this API always iterates through an empty set (all calls to 
   xPhraseFirstColumn() set iCol to -1).



   The information accessed using this API and its companion
   xPhraseFirstColumn() may also be obtained using xPhraseFirst/xPhraseNext
   (or xInst/xInstCount). The chief advantage of this API is that it is
   significantly more efficient than those alternatives when used with
   "detail=column" tables.



void (*xPhraseNextColumn)(Fts5Context*, Fts5PhraseIter*, int *piCol)

See xPhraseFirstColumn above.






8. The fts5vocab Virtual Table Module

 The fts5vocab virtual table module allows users to extract information from
an FTS5 full-text index directly. The fts5vocab module is a part of FTS5 - it 
is available whenever FTS5 is.

 Each fts5vocab table is associated with a single FTS5 table. An fts5vocab
table is usually created by specifying two arguments in place of column names
in the CREATE VIRTUAL TABLE statement - the name of the associated FTS5 table
and the type of fts5vocab table. Currently there are three types of fts5vocab
table; "row", "col" and "instance". Unless the fts5vocab table is created
within the "temp" database, it must be part of the same database as the
associated FTS5 table.

-- Create an fts5vocab "row" table to query the full-text index belonging
-- to FTS5 table "ft1".
CREATE VIRTUAL TABLE ft1_v USING fts5vocab('ft1', 'row');

-- Create an fts5vocab "col" table to query the full-text index belonging
-- to FTS5 table "ft2".
CREATE VIRTUAL TABLE ft2_v USING fts5vocab(ft2, col);

-- Create an fts5vocab "instance" table to query the full-text index
-- belonging to FTS5 table "ft3".
CREATE VIRTUAL TABLE ft3_v USING fts5vocab(ft3, instance);


 If an fts5vocab table is created in the temp database, it may be associated
with an FTS5 table in any attached database. In order to attach the fts5vocab
table to an FTS5 table located in a database other than "temp", the name of the
database is inserted before the FTS5 table name in the CREATE VIRTUAL TABLE 
arguments. For example:

-- Create an fts5vocab "row" table to query the full-text index belonging
-- to FTS5 table "ft1" in database "main".
CREATE VIRTUAL TABLE temp.ft1_v USING fts5vocab(main, 'ft1', 'row');

-- Create an fts5vocab "col" table to query the full-text index belonging
-- to FTS5 table "ft2" in attached database "aux".
CREATE VIRTUAL TABLE temp.ft2_v USING fts5vocab('aux', ft2, col);

-- Create an fts5vocab "instance" table to query the full-text index 
-- belonging to FTS5 table "ft3" in attached database "other".
CREATE VIRTUAL TABLE temp.ft2_v USING fts5vocab('aux', ft3, 'instance');


 Specifying three arguments when creating an fts5vocab table in any database
other than "temp" results in an error.

 An fts5vocab table of type "row" contains one row for each distinct term
in the associated FTS5 table. The table columns are as follows:


  ColumnContents
  term The term, as stored in the FTS5 index.
  doc  The number of rows that contain at least one instance of the term.
  cnt  The total number of instances of the term in the entire FTS5 table.


 An fts5vocab table of type "col" contains one row for each distinct term/column
combination in the associated FTS5 table. Table columns are as follows:


  ColumnContents
  term The term, as stored in the FTS5 index.
  col  The name of the FTS5 table column that contains the term.
  doc  The number of rows in the FTS5 table for which column $col
                   contains at least one instance of the term.
  cnt  The total number of instances of the term that appear in
                   column $col of the FTS5 table (considering all rows). 


 An fts5vocab table of type "instance" contains one row for each term
instance stored in the associated FTS index. Assuming the FTS5 table is
created with the 'detail' option set to 'full', table columns are as follows:


  ColumnContents
  term   The term, as stored in the FTS5 index.
  doc    The rowid of the document that contains the term instance.
  col    The name of the column that contains the term instance.
  offset The index of the term instance within its column. Terms 
                     are numbered in order of occurrence starting from 0.


 If the FTS5 table is created with the 'detail' option set to 'col', then
the offset column of an instance virtual table always contains NULL.
In this case there is one row in the table for each unique term/doc/col 
combination. Or, if the FTS5 table is created with 'detail' set to 'none',
then both offset and col always contain NULL values. For
detail=none FTS5 tables, there is one row in the fts5vocab table for each
unique term/doc combination.

Example:

-- Assuming a database created using:
CREATE VIRTUAL TABLE ft1 USING fts5(c1, c2);
INSERT INTO ft1 VALUES('apple banana cherry', 'banana banana cherry');
INSERT INTO ft1 VALUES('cherry cherry cherry', 'date date date');

-- Then querying the following fts5vocab table (type "col") returns:
--
--    apple  | c1 | 1 | 1
--    banana | c1 | 1 | 1
--    banana | c2 | 1 | 2
--    cherry | c1 | 2 | 4
--    cherry | c2 | 1 | 1
--    date   | c3 | 1 | 3
--
CREATE VIRTUAL TABLE ft1_v_col USING fts5vocab(ft1, col);

-- Querying an fts5vocab table of type "row" returns:
--
--    apple  | 1 | 1
--    banana | 1 | 3
--    cherry | 2 | 5
--    date   | 1 | 3
--
CREATE VIRTUAL TABLE ft1_v_row USING fts5vocab(ft1, row);

-- And, for type "instance"
INSERT INTO ft1 VALUES('apple banana cherry', 'banana banana cherry');
INSERT INTO ft1 VALUES('cherry cherry cherry', 'date date date');
--
--    apple  | 1 | c1 | 0
--    banana | 1 | c1 | 1
--    banana | 1 | c2 | 0
--    banana | 1 | c2 | 1
--    cherry | 1 | c1 | 2
--    cherry | 1 | c2 | 2
--    cherry | 2 | c1 | 0
--    cherry | 2 | c1 | 1
--    cherry | 2 | c1 | 2
--    date   | 2 | c2 | 0
--    date   | 2 | c2 | 1
--    date   | 2 | c2 | 2
--
CREATE VIRTUAL TABLE ft1_v_instance USING fts5vocab(ft1, instance);





  Appendix A: Comparison with FTS3/4


 Also available is the similar but more mature FTS3/4 module. 
FTS5 is a new version of FTS4 that includes various fixes and solutions for 
problems that could not be fixed in FTS4 without sacrificing backwards 
compatibility. Some of these problems are 
described below.

 Application Porting Guide 

 In order to use FTS5 instead of FTS3 or FTS4, applications usually require
minimal modifications. Most of these fall into three categories - changes
required to the CREATE VIRTUAL TABLE statement used to create the FTS table,
changes required to SELECT queries used to execute queries against the table,
and changes required to applications that use FTS auxiliary functions.

 Changes to CREATE VIRTUAL TABLE statements 


 The module name must be changed from "fts3" or "fts4" to "fts5".

 All type information or constraint specifications must be removed from
     column definitions. FTS3/4 ignores everything following the column name in
     a column definition, FTS5 attempts to parse it (and will report an error
     if it fails to).

 The "matchinfo=fts3" option is not available. The 
     "columnsize=0" option is equivalent.

 The notindexed= option is not available. Adding UNINDEXED
     to the column definition is equivalent.

 The ICU tokenizer is not available.

 The compress=, uncompress= and languageid= options are not available.
     There is as of yet no equivalent for their functionality.


 -- FTS3/4 statement 
CREATE VIRTUAL TABLE t1 USING fts4(
  linkid INTEGER,
  header CHAR(20),
  text VARCHAR,
  notindexed=linkid,
  matchinfo=fts3,
  tokenizer=unicode61
);

 -- FTS5 equivalent (note - the "tokenizer=unicode61" option is not
 -- required as this is the default for FTS5 anyway)
CREATE VIRTUAL TABLE t1 USING fts5(
  linkid UNINDEXED,
  header,
  text,
  columnsize=0
);


 Changes to SELECT statements 


   The "docid" alias does not exist. Applications must use "rowid"
          instead.

   The behaviour of queries when a column-filter is specified both as
          part of the FTS query and by using a column as the LHS of a MATCH
          operator is slightly different. For a table with columns "a" and "b"
          and a query similar to:
... a MATCH 'b: string'

       FTS3/4 searches for matches in column "b". However, FTS5 always
          returns zero rows, as results are first filtered for column "b", then
          for column "a", leaving no results. In other words, in FTS3/4 the
          inner filter overrides the outer, in FTS5 both filters are applied.

   The FTS query syntax (right hand side of the MATCH operator) has
          changed in some ways. The FTS5 syntax is quite close to the FTS4
          "enhanced syntax". The main difference is that FTS5 is fussier 
          about unrecognized punctuation characters and similar within query
          strings. Most queries that work with FTS3/4 should also work with
          FTS5, and those that do not should return parse errors.


 Auxiliary Function Changes 

 FTS5 has no matchinfo() or offsets() function, and the snippet() function
is not as fully-featured as in FTS3/4. However, since FTS5 does provide 
an API allowing applications to create custom auxiliary functions, any
required functionality may be implemented within the application code.

 The set of built-in auxiliary functions provided by FTS5 may be 
improved upon in the future.

 Other Issues


   The functionality provided by the fts4aux module is now provided
          by fts5vocab. The schema of these two tables is slightly different.

   The FTS3/4 "merge=X,Y" command has been replaced by the 
          FTS5 merge command.

   The FTS3/4 "automerge=X" command has been replaced by the 
          FTS5 automerge option.



 
  Summary of Technical Differences 


FTS5 is similar to FTS3/4 in that the primary task of each is to maintain
an index mapping from each unique token to a list of instances of that token 
within a set of documents, where each instance is identified by the document 
in which it appears and its position within that document. For example:

-- Given the following SQL:
CREATE VIRTUAL TABLE ft USING fts5(a, b);
INSERT INTO ft(rowid, a, b) VALUES(1, 'X Y', 'Y Z');
INSERT INTO ft(rowid, a, b) VALUES(2, 'A Z', 'Y Y');

-- The FTS5 module creates the following mapping on disk:
A --&gt; (2, 0, 0)
X --&gt; (1, 0, 0)
Y --&gt; (1, 0, 1) (1, 1, 0) (2, 1, 0) (2, 1, 1)
Z --&gt; (1, 1, 1) (2, 0, 1)


In the example above, each triple identifies the location of a token
instance by rowid, column number (columns are numbered sequentially
starting at 0 from left to right) and position within the column value (the
first token in a column value is 0, the second is 1, and so on). Using this
index, FTS5 is able to provide timely answers to queries such as "the set
of all documents that contain the token 'A'", or "the set of all documents
that contain the sequence 'Y Z'". The list of instances associated with a
single token is called an "instance-list".

The principle difference between FTS3/4 and FTS5 is that in FTS3/4,
each instance-list is stored as a single large database record, whereas
in FTS5 large instance-lists are divided between multiple database records.
This has the following implications for dealing with large databases that
contain large lists:


   FTS5 is able to load instance-lists into memory incrementally in
       order to reduce memory usage and peak allocation size. FTS3/4 very
       often loads entire instance-lists into memory.

   When processing queries that feature more than one token, FTS5 is
       sometimes able to determine that the query can be answered by
       inspecting a subset of a large instance-list. FTS3/4 almost always
       has to traverse entire instance-lists.

   If an instance-list grows so large that it exceeds
       the SQLITE_MAX_LENGTH limit, FTS3/4 is unable to handle it. FTS5
       does not have this problem. 


For these reasons, many complex queries may use less memory and run faster 
using FTS5.

Some other ways in which FTS5 differs from FTS3/4 are:


   FTS5 supports "ORDER BY rank" for returning results in order of
       decreasing relevancy.

   FTS5 features an API allowing users to create custom auxiliary 
       functions for advanced ranking and text processing applications. The
       special "rank" column may be mapped to a custom auxiliary function
       so that adding "ORDER BY rank" to a query works as expected.

   FTS5 recognizes unicode separator characters and case equivalence by
       default. This is also possible using FTS3/4, but must be explicitly
       enabled.

   The query syntax has been revised where necessary to remove
       ambiguities and to make it possible to escape special characters
       in query terms.

   By default, FTS3/4 occasionally merges together two or more of the
       b-trees that make up its full-text index within an INSERT, UPDATE or
       DELETE statement executed by the user. This means that any operation
       on an FTS3/4 table may turn out to be surprisingly slow, as FTS3/4 
       may unpredictably choose to merge together two or more large b-trees
       within it. FTS5 uses incremental merging by default, which limits
       the amount of processing that may take place within any given 
       INSERT, UPDATE or DELETE operation.




  Appendix B: Shadow tables created by FTS5



When an FTS5 virtual table is created in a database, between 3 and 5 real
tables are created in the database. These are known as "shadow tables", and are
used by the virtual table module to store persistent data. They should not
be accessed directly by the user. Many other virtual table modules, including
FTS3 and rtree, also create and use shadow tables.

FTS5 creates the following shadow tables. In each case the actual table name
is based on the name of the FTS5 virtual table (in the following table, replace
&lt;name&gt; with the name of the virtual table to find the actual shadow
table name).


Table&nbsp;NameContents
&lt;name&gt;_data This table contains most of the full-text index data.
&lt;name&gt;_idx This table contains the remainder of the full-text
index data. It is almost always much smaller than the &lt;name&gt;_data table.
&lt;name&gt;_config Contains the values of persistent
configuration parameters.
&lt;name&gt;_content Contains the actual data inserted into the 
FTS5 table. This shadow table is not present for 
contentless or 
external content FTS5 tables.
&lt;name&gt;_docsize Contains the size of each column of each
row in the virtual table in tokens. This shadow table is not present if 
the "columnsize" option is set to 0.


c2: Code Of Conduct
c5: codeofconduct.html
c4:
1. Overview


The original document we put here was more of a 
Code of Ethics of the Project Founder. 
While we stand by those principles, they are not in line with the modern
technical meaning of a Code of Conduct and have hence been renamed.

2. Code of Conduct


The SQLite community has been a kind, welcoming and well-mannered community
since its inception nearly two decades ago. In that time many of the
qualities we've worked hard to bring to our project - safety, consistency,
usability and integrity among them - have become part of our community's
culture as well. Much like the care and effort we've put into securing the
longevity of SQLite, we'd like to do the same for that community by 
clarifying our goals and expectations for all involved. To that end
the SQLite project has officially adopted the 
Mozilla Community Participation Guidelines
as its Code of Conduct with the exception that the reporting
email address is changed to
codeofconduct@sqlite.org.
While we are not doing so in reaction to any current or ongoing issues,
we believe that this will be a helpful part of maintaining the long-term
sustainability of the project.


c2: SQLite Consortium Agreement
c5: consortium_agreement-20071201.html
c4:
SQLite Consortium Agreement


This SQLite Consortium Agreement ("Agreement") is made and
entered into as of the _____ day of ______________, 2007 ("Effective Date")
by and between ___________________ (the "Company"), and
Hipp, Wyrick & Company, Inc., a Georgia Corporation with
headquarters at 6200 Maple Cove Lane, Charlotte, NC ("Hwaci").

WHEREAS, Hwaci is the developer of a serverless, embeddable, public
domain SQL database engine available at 
http://www.sqlite.org/
("SQLite"); and

WHEREAS, Company wishes to support the independent development of 
the SQLite database engine to insure the continuing reliability, 
vitality, and independence of SQLite, and in connection therewith,
Company wishes to provide funds for ongoing research, development, 
and maintenance of the SQLite code base, operation of the SQLite 
website, protection and maintenance of the SQLite
trademark, as well as for promotion of SQLite at conferences and
trade shows; and

WHEREAS, Company wishes to receive benefits in the form of
services from Hwaci in consideration for their support of SQLite; and

WHEREAS, Company and Hwaci each wish SQLite to be supported and 
maintained as an independent, community-driven project, and to 
ensure the SQLite development team remains independent.

NOW, THEREFORE, in consideration of the mutual promises contained 
herein, the parties agree as follows:

1.0 Definitions

1.1 SQLite Developers
The "SQLite Developers" are the employees or contractors of Hwaci engaged 
in the development of SQLite, including without limitation the 
SQLite Architect.

1.2 SQLite Architect
The "SQLite Architect" is the SQLite Developer with ultimate 
responsibility and authority over changes and maintenance of the 
SQLite code.  The current SQLite Architect is the original designer
and developer of SQLite, D. Richard Hipp.   Hwaci may change the 
SQLite Architect from time to time upon approval by Company.

1.3 Consortium Member
A "Consortium Member" is a company or organization, including
but not limited to Company, which has entered into an agreement
with Hwaci which is substantially similar to this Agreement.


2.0 Agreement

2.1 Responsibilities And Obligations Of SQLite Developers

2.1.1 General Support
The SQLite Developers shall provide, without limit,
timely and accurate answers, advice, and instruction
in response to questions and comments from Company about SQLite
submitted by email or by telephone at any time day or night.


2.1.2 Debugging Support
The SQLite Developers shall at Company's request provide
Company with direct assistance
in debugging applications that are built upon or use SQLite.
The combined time for Debugging Support and Custom
Development for Company is limited to 330 hours per year.


2.1.3 Custom Development
The SQLite Developers shall at the request of Company
write new extensions or enhancements
to SQLite according to Company's specifications.  These extensions
and enhancements may be proprietary to Company or open source or
with the consent of both Company and the SQLite Architect may
be folded into the public domain SQLite source tree.
The time devoted to Custom Development for Company is limited 220 hours
per year.


2.1.4 Custom Builds
The SQLite Developers shall at Company's request provide Company
with specialized builds of SQLite according to Company's specifications.


2.1.5 Legacy Support
The SQLite Developers shall at Company's request provide support
and bug-fix patches for any historical version of SQLite.


2.1.6 Custom Testing
The SQLite Developers shall at Company's request set up and run
acceptance tests according Company's specifications that are specifically 
designed to exercise SQLite as configured and used by Company.
These acceptance tests shall be run and shall pass prior to any release
of SQLite.


2.1.7 Priority Bug Fixes
The SQLite Developers shall assign highest priority to fixing bugs
in SQLite that are reported by Company or in which Company has
expressed a specific interest.


2.1.8 Code Quality
The SQLite Developers shall at all times keep the SQLite source 
code well commented and documented and clearly structured and 
organized so that an experienced and competent programmer can
understand it and support it after no more than a few months of study.


2.1.9 Backwards Compatibility
The SQLite Developers recognize that Company uses SQLite 
in mission-critical applications and therefore shall work diligently 
to ensure continuing bug-free correct operation and backwards 
compatibility with prior releases.


2.1.10 Test Coverage
The SQLite Developers shall maintain the SQLite test suite such
that no less than 95% source code coverage is provided.


2.1.11 Website
The SQLite Developers shall maintain the SQLite website at
http://www.sqlite.org/
in good working order and all SQLite source code shall 
be made freely available at such website. 



2.2 Responsibilities And Obligations Of The SQLite Architect

2.2.1 Private Briefings
The SQLite Architect shall, at the request of Company, 
provide Company with private notification and briefings
of any new bugs or new features in SQLite that might 
effect Company's products.


2.2.2 Direction
The SQLite Architect shall coordinate the activities of SQLite
Developers and shall publish from time to time a roadmap of
proposed future changes and enhancements to SQLite.


2.2.3 Oversight
The SQLite Architect shall personally review all changes to 
the SQLite Source Code and take responsibility for the quality 
and correctness of those changes.


2.2.4 Continuity
The initial SQLite Architect shall be the original developer
of SQLite, D. Richard Hipp.  The role of SQLite Architect shall
not be assigned to another during the term of this Agreement except 
due to the disability of Dr. Hipp and with approval of Company.


2.3 Responsibilities And Obligations Of Hwaci

2.3.1 Employer
Hwaci shall act as the employer of the SQLite Developers and
SQLite Architect and shall make all tax and legal filings
and tax withholding required by employers.  Hwaci shall provide
Company with W-9 information upon request.


2.3.2 Responsible Party
All obligations of the SQLite Developers set forth in this
agreement are deemed obligations of Hwaci.  Hwaci shall recruit, 
employ, and supervise SQLite Developers in such a way that the
responsibilities and obligations of the SQLite Developers set 
forth in this agreement are upheld.

2.3.3 Multiple Developers
Hwaci shall recruit and employ as many SQLite Developers for as
many hours as can be reasonably achieved using the funds received 
from Consortium Members under this agreement.
So that support for SQLite will not be interrupted by the
disability or withdrawal of any one SQLite Developer, Hwaci will
keep at least two knowledgeable and competent SQLite Developers
on staff.

2.3.4 Mentorship
Hwaci shall work to encourage independent 
programmers from around the world
to become familiar with SQLite internals so that there will be a
pool of talent able to support SQLite in the future.


2.3.5 Audits
Hwaci shall keep and maintain complete and accurate records
of the use of development funds provided by Company
and shall allow Company, or its representative, a certified 
public accountant mutually acceptable to Hwaci and Company, 
during office hours and at reasonable intervals, no more than 
once every 12 months, to inspect and make extracts or copies
of such records solely for the purpose of ascertaining Hwaci's 
compliance with the objectives and requirements of this agreement.


2.3.6 Disaster Planning
Hwaci shall maintain backup copies of all
SQLite source files and documentation, current and historical,
at at least two separate locations separated from each
other and from the primary on-line SQLite repository
by at least 400 kilometers.


2.3.7 Trademark
Hwaci shall maintain ownership of the SQLite trademark and
the sqlite.org domain name and shall purchase bandwidth and
server space for the 
http://www.sqlite.org/ website.


2.3.8 No Take-overs or Buy-outs
Neither Hwaci nor the SQLite Architect shall relinquish development
control of SQLite during the term of this Agreement, by acquisition or merger 
or by any other means,
except with the consent of Company.


2.3.9 New Consortium Members
New Consortium Members may be accepted into the consortium from
time to time under identical terms as this agreement, or under
substantially similar terms that have been approved by existing
Consortium Members.

2.3.10 Adequate Staff
Hwaci shall recruit and employ a sufficient number of qualified
SQLite Developers to easily cover all custom development,
debugging, and general support service obligations for all
Consortium Members while still providing ample time for
the SQLite Developers to engage in general maintenance
and extension of SQLite.


2.3.11 Use Of Funds
Hwaci shall use the funds provided hereunder
solely for the development and maintenance of SQLite as set forth in
this Agreement.
Interest on funds received in advance and held in trust will
be reinvested and used for the same purposes as the principal.


2.4 Responsibilities And Obligations Of Company

2.4.1 Funding
Company shall provide funding for the ongoing support and
maintenance of SQLite as set forth in section 3.0 "Fees".


2.4.2 Copyright Disclaimer
Company acknowledges that SQLite source code and documentation
published on the SQLite website
is in the public domain and that nothing in this agreement shall change
that fact.


3.0 Fees

In consideration of the performance
by Hwaci, the SQLite Developers, and the SQLite Architect
of the obligations described herein,
Company shall pay Hwaci at least US $75,000 per year in advance
either annually, quarterly, or monthly, at Company's discretion.

4.0 Confidentiality

4.1 Definition of Confidential Information

"Confidential Information" means any Company proprietary information, 
technical data, trade secrets or know-how, including, but not limited to, 
research, product plans, products, services, customers, customer lists, 
markets, software, developments, inventions, processes, formulas, 
technology, designs, drawings, engineering, hardware configuration 
information, marketing, finances or other business information 
disclosed by Company either directly or indirectly in writing, 
orally or by drawings or inspection of parts or equipment.

4.2 Non-Use and Non-Disclosure.

Hwaci shall not, during or subsequent to the term of this Agreement,
use Company's Confidential Information for any purpose whatsoever
other than the performance of the Services or disclose 
Company's Confidential Information to any third party.  The parties 
acknowledge that Confidential Information will remain the sole
property of Company.  Hwaci shall take all reasonable precautions 
to prevent any unauthorized disclosure of Confidential Information 
including, but not limited to, having each employee or 
consultant of Hwaci, if any, with access to any 
Confidential Information, execute a nondisclosure agreement
containing provisions in Company's favor substantially similar 
to this Agreement.  Confidential Information does not include
information that: (i) is known to Hwaci at the time of 
disclosure to Hwaci by Company as evidenced by written 
records of Hwaci; (ii) has become publicly known and made 
generally available through no wrongful act of Hwaci; or 
(iii) has been received by Hwaci from a third party who is 
authorized to make such disclosure.

4.2.1 Disclosure Required by Law
In the event any Confidential Information is required to be disclosed 
by Hwaci under the terms of a valid and effective subpoena or order 
issued by a court of competent jurisdiction, or by a demand or 
information request from an executive or administrative agency 
or other governmental authority, Hwaci shall, unless prohibited
by the terms of a subpoena, order, or demand, promptly notify 
Company of the existence, terms and circumstances surrounding 
such demand or request, shall consult with Company on the 
advisability of taking legally available steps to resist 
or narrow such demand or request, and, if disclosure of 
such Confidential Information is required, shall exercise 
its reasonable best efforts to narrow the scope of disclosure 
and obtain an order or other reliable assurance that 
confidential treatment will be accorded to such 
Confidential Information.  To the extent that Hwaci
is prohibited from notifying Company of a subpoena, 
order or demand, by the terms of same, Hwaci shall exercise 
its reasonable efforts to narrow the scope of disclosure.

4.3 Return of Materials.

Upon the termination of this Agreement, or upon Company's 
earlier request, Hwaci shall deliver to Company all of
Company's property or Confidential Information that Hwaci 
may have in Hwaci's possession or control.

5.0 Intellectual Property

5.1 No Assignment

Company acknowledges that all copyrightable material, notes, 
records, drawings, designs, inventions, improvements, developments, 
discoveries and trade secrets made, generated, conceived, or
reduced to practice by Hwaci related to SQLite
will remain the property of Hwaci.  Nothing in this Agreement
will be construed to transfer any intellectual property right 
of Hwaci to Company.

5.2 Availability and Public Domain Dedication

The SQLite Developers and Hwaci shall, 
subject to their discretion as to the quality
and suitability of the SQLite source code and documentation
for public release, make 
the SQLite source code and documentation
publicly available as downloadable files
and make a public statement ceding all intellectual 
property rights, including but not limited to copyright
and patent rights, in the SQLite source code and documentation
to the public domain.  
To the extent that the SQLite Developers and Hwaci 
elect not to release the SQLite
source code and documentation
publicly, they shall provide copies thereof to 
Company and hereby grants to Company, under all of the 
SQLite Developers' and Hwaci's 
rights
including but not limited to copyright and patent rights,
in and to the SQLite source code and documentation, 
perpetual, irrevocable, 
worldwide, non-exclusive, sublicenseable license to use, 
copy, prepare derivative works of, publicly perform and 
display the SQLite source code and documentation 
and derivative works thereof.

5.3 Trademark

Hwaci shall use the name "SQLite" only to apply to the 
publicly available project known by such name as of the 
Effective Date.  Hwaci may in its discretion file such 
trademark applications or registrations as it deems 
appropriate to protect or record its rights therein, 
and may set such policies as it deems appropriate for 
licensing the use of the trademark.

6.0 Representations And Warranties

6.1 Intellectual Property Clearances.

Hwaci hereby represents and warrants that Hwaci shall 
enter into agreements with the SQLite Developers sufficient 
to enable Hwaci to undertake the obligations of Section 5.

6.2 Disclaimer.

THE WORK PRODUCT AND ALL MATERIAL PROVIDED BY HWACI AND 
COMPANY ARE PROVIDED "AS IS."  NEITHER PARTY MAKES ANY 
REPRESENTATIONS OR WARRANTIES OF ANY KIND, WHETHER ORAL
OR WRITTEN, WHETHER EXPRESS, IMPLIED, OR ARISING BY STATUTE, 
CUSTOM, COURSE OF DEALING OR TRADE USAGE, WITH RESPECT TO 
THE SUBJECT MATTER HEREOF, IN CONNECTION WITH THIS AGREEMENT.  
EACH PARTY SPECIFICALLY DISCLAIMS ANY AND ALL IMPLIED
WARRANTIES OR CONDITIONS OF TITLE, MERCHANTABILITY, 
FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT.

7.0 Term And Termination

7.1 Term.

This Agreement will commence on the Effective Date and 
will continue until 12 months after the Effective Date.
Thereafter, the parties may by mutual consent renew 
this Agreement subject to agreement on fees to be paid by 
Company for sponsorship for additional periods.

7.2 Termination.

If either party materially defaults in the performance 
of any of its material obligations hereunder and if any 
such default is not corrected within 30 days after notice 
in writing, then the non-defaulting party, at its option, 
may, in addition to any other remedies it may have, thereupon 
terminate this Agreement by giving written notice of 
termination to the defaulting party.

7.3 Survival.

Upon such termination all rights and duties of the
parties toward each other will cease except:  
Sections 4 (Confidentiality), 5 (Intellectual Property), 
and 8 (Miscellaneous) will survive termination of this Agreement.

8. Miscellaneous

8.1 Nonassignment/Binding Agreement.

The parties acknowledge that the unique nature of 
Hwaci's services are substantial consideration for the 
parties' entering into this Agreement.  Neither this 
Agreement nor any rights under this Agreement may be 
assigned or otherwise transferred by Hwaci, in whole 
or in part, whether voluntarily or by operation of law, 
without the prior written consent of Company, which 
consent will not be unreasonably withheld.  
Subject to the foregoing, this Agreement will be binding
upon and will inure to the benefit of the parties and
their respective successors and assigns. Any assignment 
in violation of the foregoing will be null and void.

8.2 Notices.

Any notice required or permitted under the terms of 
this Agreement or required by law must be in writing 
and must be:  (a) delivered in person; (b) sent by first 
class registered mail, or air mail, as appropriate; or 
(c) sent by overnight air courier, in each case properly 
posted and fully prepaid to the appropriate address set 
forth in the preamble to this Agreement.  Either party 
may change its address for notice by notice to the other 
party given in accordance with this Section.  Notices 
will be considered to have been given at the time of
actual delivery in person, three business days after 
deposit in the mail as set forth above, or one day after 
delivery to an overnight air courier service.

8.3 Waiver.

Any waiver of the provisions of this Agreement or 
of a party's rights or remedies under this Agreement 
must be in writing to be effective.  Failure, neglect, 
or delay by a party to enforce the provisions of this 
Agreement or its rights or remedies at any time, will
not be construed as a waiver of such party's rights 
under this Agreement and will not in any way affect 
the validity of the whole or any part of this Agreement 
or prejudice such party's right to take subsequent action. 
No exercise or enforcement by either party of any right
or remedy under this Agreement will preclude the 
enforcement by such party of any other right or remedy 
under this Agreement or that such party is entitled by 
law to enforce.

8.4 Severability.

If any term, condition, or provision in this Agreement 
is found to be invalid, unlawful or unenforceable to 
any extent, the parties shall endeavor in good faith 
to agree to such amendments that will preserve, as far
as possible, the intentions expressed in this Agreement. 
If the parties fail to agree on such an amendment, 
such invalid term, condition or provision will be 
severed from the remaining terms, conditions and
provisions, which will continue to be valid and 
enforceable to the fullest extent permitted by law.

8.5 Integration.

This Agreement contains the entire agreement of the 
parties with respect to the subject matter of this 
Agreement and supersedes all previous communications, 
representations, understandings and agreements, 
either oral or written, between the parties with 
respect to said subject matter.  This Agreement 
may not be amended, except by a writing signed by both parties.

8.6 Counterparts.

This Agreement may be executed in counterparts, 
each of which so executed will be deemed to be an 
original and such counterparts together will constitute 
one and the same agreement.

8.7 Governing Law.

This Agreement will be interpreted and construed in 
accordance with the laws of the State of North Carolina
and the United States of America, without regard to
conflict of law principles.  All disputes arising out 
of this Agreement will be subject to the exclusive 
jurisdiction of the state and federal courts located in 
North Carolina, and each party hereby consents to the 
personal jurisdiction thereof.

8.8 Independent Contractors.

It is the intention of the parties that Hwaci is an 
independent contractor.  Nothing in this Agreement will 
in any way be construed to constitute Hwaci or any of 
its employees as an agent, employee or representative 
of Company.

9.0 Signatures

The parties have executed this Agreement below to 
indicate their acceptance of its terms.



HWACI
By:
&nbsp;
Print Name:
&nbsp;
Title:
&nbsp;


COMPANY
By:
&nbsp;
Print Name:
&nbsp;
Title:
&nbsp;


c2: How SQLite Works
c5: howitworks.html
c4:
1. Background


SQLite is a software library
that translates high-level disk I/O requests generated by an application
into low-level I/O operations that
can be carried out by the operating system.  The application constructs 
high-level I/O requests using the
SQL language.
SQLite translates each high-level SQL statement into a sequence of
many low-level I/O requests (open a file, read a few bytes from a file,
write a few bytes into a file, etc.) that do the work requested by
the SQL.



An application program could do all its disk I/O by direct calls to
operating system I/O routines
or by using a key/value storage engine like 
Berkeley DB or
RocksDB (to name but two).
But there are advantages to using a higher-level interface based
on the SQL language.



SQL is a very high-level language.
A few lines of SQL can replace hundreds or thousands of lines of
procedural code.  SQL thus reduces
the amount of work needed to develop and maintain the application,
and thereby helps to reduce the number of bugs in the application.


SQL and SQLite are 
transactional.
The use of a transactional storage system makes it much easier to
reason about the behavior of the application, and to write applications
that are robust, even in the face of software bugs, hardware faults,
or power losses.


SQLite is often
faster than direct low-level I/O.
This is counterintuitive.  One would expect that a
high-level interface such as SQLite would impose a run-time penalty.
And, theoretically, that is correct.  But in practice, SQL-based
systems such as SQLite do so many behind-the-scenes optimizations
that an application developer would never have time to create and
maintain, that the SQL-based systems end up providing
a net performance gain.


1.1. SQLite Is Different From Most Other SQL Databases


There are many SQL-based database management systems available, besides
SQLite.  Common options include MySQL, PostgreSQL, and SQL-Server.
All these systems use the SQL langauge to communicate with the 
application, just like SQLite.  But these other systems different
from SQLite in important respects.



SQLite is a serverless software library, whereas the other
systems are client-server based.  With MySQL, PostgreSQL, SQL-Server,
and others, the application sends a message containing some SQL
over to a separate server thread or process. That separate
thread or process performs the requested I/O, then send the results
back to the application.  But there is no separate thread or
process with SQLite.  SQLite runs in the same address space as
the application, using the same program counter and heap storage.
SQLite does no interprocess communication (IPC).  When an application
sends an SQL statement into SQLite (by invoking a the appropriate
SQLite library subroutine), SQLite interprets the SQL
in the same thread as the caller.  When an SQLite API routine
returns, it does not leave behind any background tasks that
run separately from the application.


An SQLite database is a single ordinary file on disk
(with a well-defined file format).  With other
systems, a "database" is usually a large number of separate
files hidden away in obscure directories of the filesystem, or
even spread across multiple machines.  But with SQLite, a
complete database is just an ordinary disk file.


2. SQL Is A Programming Language


The best way to understand how SQL database engines work is to
think of SQL as a programming language, not as a "query language".
Each SQL statement is a separate program.  Applications construct
SQL program source files and send them to the database engine.
The database engine compiles the SQL source code into executable
form, runs that executable, then sends the result back to the
application.


While SQL is a programming language, it is different from other
programming languages like C, Javascript, Python, or Go in that
SQL is a 
declarative language
where the others are
imperative languages.
This is an important difference that has implications for the
design of the compiler used to translate program source text into an
executable format. However, those details 
should not detract from the fact that
SQL is really just another programming language.

2.1. Programming Language Processing Steps


All programming languages are processed in two steps:



Translate the program source text into an executable format.

Run the executable generated in the previous step in order to
carry out the desired action.



All programming languages uses those two basic steps.
The main difference is in the executable format.


"Compiled" languages like C++ and Rust translate
the source text into machine code that can be directly executed
by the underlying hardware.  There exist SQL database systems
that do the same with SQL - they translate each SQL statement
directly into machine code.  But that approach is uncommon and
is not the approach taken by SQLite.


Other languages like Java, Perl, Python, and TCL
typically translate the program source text into bytecode.
This bytecode is then run through an interpreter that
reads the bytecode and carries out the desired operations.
SQLite uses this bytecode approach.  If you preceed
any SQL statement with the "EXPLAIN" keyword in SQLite,
it will show you the bytecode that is generated rather
than run the bytecode.


Another approach is to translate the program source text
into a tree of objects in memory.  This tree is the "executable".
An interpret runs the executable by walking the tree.
This is the technique used by MySQL, PostgreSQL, and
SQL-Server.


Of course, not every language fits neatly into one of the
above catagories.  This applies to both SQL database engines
and more familiar imperative programming languages.  Javascript
is famous for using a hybrid execution model, where the code
is initially compiled into a tree of objects, but might be
further translating (using just-in-time compilation) down into
more efficient bytecode or machine code, as a means of boosting
performance.


The executable format really ends up being just an implementation
detail.  The key point is that all languages have a compiler
step which translates programs into an executable format and
a run step that executes the compiled program.


2.2. Compiling SQLite Programs


When an SQL program is submitted to SQLite, the first step is
to split the source text into "tokens".  A token might be:


A language keyword like "SELECT" or "UPDATE".
An identifier for a table or column or variable.
Punctuation characters like "," or "==" or ";".
Literal values: numeric or string constants.
Whitespace or comments.



Whitespace and comment tokens are discarded.  All other tokens
are fed into an
LALR(1) Parser
that analysis the structure of the input program and generates an
Abstract Syntax Tree (AST)
for the input program.


The parser forwards the AST on to the code generator.
The code generator is the heart of SQLite, and is where most of
the magic happens.
The code generator resolves symbolic names in the AST - matching
the names of columns and tables in the input SQL into actual
columns and tables of the database.  The code generator also does
various transformations on the AST to "optimize" it.  Finally
the code generator chooses appropriate algorithms to implement
the operations requested by the AST and constructs bytecode to
carry out those operations.


The bytecode generated by the code generator is called a
"prepared statement".  Translating SQL source text into a
prepared statement is analogous to converting a C++ program
into machine code by invoking gcc or clang.  Human-readable
source text (SQL or C++) goes in, and a machine readable executable
(bytecode or machine code) comes out.

3. Further Reading



The Atomic Commit document describes how SQLite implements
transactions.


The bytecode engine document has more information
on the bytecode format used by SQLite, and how to view and interpret
an SQLite prepared statement.


The SQLite query planner and
Next Generation Query Planner documents have further details
on the algorithms SQLite uses to implement SQL statements and
how it goes above choosing an appropriate algorithm for each
individual SQL statement.


c2: C/C++ Interface For SQLite Version 3 (old)
c5: capi3.html
c4:
Note:
This document was written in 2004 as a guide to helping programmers
move from using SQLite version 2 to SQLite version 3.  The information
in this document is still essentially correct, however there have been
many changes and enhancements over the years.  We recommend that the
following documents be used instead:

An Introduction To The SQLite C/C++ Interface
SQLite C/C++ Reference Guide




C/C++ Interface For SQLite Version 3

1.0 Overview


SQLite version 3.0 is a new version of SQLite, derived from
the SQLite 2.8.13 code base, but with an incompatible file format
and API.
SQLite version 3.0 was created to answer demand for the following features:



Support for UTF-16.
User-definable text collating sequences.
The ability to store BLOBs in indexed columns.



It was necessary to move to version 3.0 to implement these features because
each requires incompatible changes to the database file format.  Other
incompatible changes, such as a cleanup of the API, were introduced at the
same time under the theory that it is best to get your incompatible changes
out of the way all at once.  



The API for version 3.0 is similar to the version 2.X API,
but with some important changes.  Most noticeably, the "sqlite_"
prefix that occurs on the beginning of all API functions and data
structures are changed to "sqlite3_".  
This avoids confusion between the two APIs and allows linking against both
SQLite 2.X and SQLite 3.0 at the same time.



There is no agreement on what the C datatype for a UTF-16
string should be.  Therefore, SQLite uses a generic type of void*
to refer to UTF-16 strings.  Client software can cast the void* 
to whatever datatype is appropriate for their system.


2.0 C/C++ Interface


The API for SQLite 3.0 includes 83 separate functions in addition
to several data structures and #defines.  (A complete
API reference is provided as a separate 
document.)
Fortunately, the interface is not nearly as complex as its size implies.
Simple programs can still make do with only 3 functions:
sqlite3_open(), sqlite3_exec(), and sqlite3_close().
More control over the execution of the database engine is provided
using sqlite3_prepare_v2()
to compile an SQLite statement into byte code and
sqlite3_step() to execute that bytecode.
A family of routines with names beginning with 
sqlite3_column_
is used to extract information about the result set of a query.
Many interface functions come in pairs, with both a UTF-8 and
UTF-16 version.  And there is a collection of routines
used to implement user-defined SQL functions and user-defined
text collating sequences.



2.1 Opening and closing a database


   typedef struct sqlite3 sqlite3;
   int sqlite3_open(const char*, sqlite3**);
   int sqlite3_open16(const void*, sqlite3**);
   int sqlite3_close(sqlite3*);
   const char *sqlite3_errmsg(sqlite3*);
   const void *sqlite3_errmsg16(sqlite3*);
   int sqlite3_errcode(sqlite3*);



The sqlite3_open() routine returns an integer error code rather than
a pointer to the sqlite3 structure as the version 2 interface did.
The difference between sqlite3_open()
and sqlite3_open16() is that sqlite3_open16() takes UTF-16 (in host native
byte order) for the name of the database file.  If a new database file
needs to be created, then sqlite3_open16() sets the internal text
representation to UTF-16 whereas sqlite3_open() sets the text
representation to UTF-8.



The opening and/or creating of the database file is deferred until the
file is actually needed.  This allows options and parameters, such
as the native text representation and default page size, to be
set using PRAGMA statements.



The sqlite3_errcode() routine returns a result code for the most
recent major API call.  sqlite3_errmsg() returns an English-language
text error message for the most recent error.  The error message is
represented in UTF-8 and will be ephemeral - it could disappear on
the next call to any SQLite API function.  sqlite3_errmsg16() works like
sqlite3_errmsg() except that it returns the error message represented
as UTF-16 in host native byte order.



The error codes for SQLite version 3 are unchanged from version 2.
They are as follows:



#define SQLITE_OK           0   /* Successful result */
#define SQLITE_ERROR        1   /* SQL error or missing database */
#define SQLITE_INTERNAL     2   /* An internal logic error in SQLite */
#define SQLITE_PERM         3   /* Access permission denied */
#define SQLITE_ABORT        4   /* Callback routine requested an abort */
#define SQLITE_BUSY         5   /* The database file is locked */
#define SQLITE_LOCKED       6   /* A table in the database is locked */
#define SQLITE_NOMEM        7   /* A malloc() failed */
#define SQLITE_READONLY     8   /* Attempt to write a readonly database */
#define SQLITE_INTERRUPT    9   /* Operation terminated by sqlite_interrupt() */
#define SQLITE_IOERR       10   /* Some kind of disk I/O error occurred */
#define SQLITE_CORRUPT     11   /* The database disk image is malformed */
#define SQLITE_NOTFOUND    12   /* (Internal Only) Table or record not found */
#define SQLITE_FULL        13   /* Insertion failed because database is full */
#define SQLITE_CANTOPEN    14   /* Unable to open the database file */
#define SQLITE_PROTOCOL    15   /* Database lock protocol error */
#define SQLITE_EMPTY       16   /* (Internal Only) Database table is empty */
#define SQLITE_SCHEMA      17   /* The database schema changed */
#define SQLITE_TOOBIG      18   /* Too much data for one row of a table */
#define SQLITE_CONSTRAINT  19   /* Abort due to contraint violation */
#define SQLITE_MISMATCH    20   /* Data type mismatch */
#define SQLITE_MISUSE      21   /* Library used incorrectly */
#define SQLITE_NOLFS       22   /* Uses OS features not supported on host */
#define SQLITE_AUTH        23   /* Authorization denied */
#define SQLITE_ROW         100  /* sqlite_step() has another row ready */
#define SQLITE_DONE        101  /* sqlite_step() has finished executing */


2.2 Executing SQL statements


   typedef int (*sqlite_callback)(void*,int,char**, char**);
   int sqlite3_exec(sqlite3*, const char *sql, sqlite_callback, void*, char**);



The sqlite3_exec() function works much as it did in SQLite version 2.
Zero or more SQL statements specified in the second parameter are compiled
and executed.  Query results are returned to a callback routine.



In SQLite version 3, the sqlite3_exec routine is just a wrapper around
calls to the prepared statement interface.



   typedef struct sqlite3_stmt sqlite3_stmt;
   int sqlite3_prepare(sqlite3*, const char*, int, sqlite3_stmt**, const char**);
   int sqlite3_prepare16(sqlite3*, const void*, int, sqlite3_stmt**, const void**);
   int sqlite3_finalize(sqlite3_stmt*);
   int sqlite3_reset(sqlite3_stmt*);



The sqlite3_prepare interface compiles a single SQL statement into byte code
for later execution.  This interface is now the preferred way of accessing
the database.



The SQL statement is a UTF-8 string for sqlite3_prepare().
The sqlite3_prepare16() works the same way except
that it expects a UTF-16 string as SQL input.
Only the first SQL statement in the input string is compiled.
The fifth parameter is filled in with a pointer to the next (uncompiled)
SQLite statement in the input string, if any.
The sqlite3_finalize() routine deallocates a prepared SQL statement.
All prepared statements must be finalized before the database can be
closed.
The sqlite3_reset() routine resets a prepared SQL statement so that it
can be executed again.



The SQL statement may contain tokens of the form "?" or "?nnn" or ":aaa"
where "nnn" is an integer and "aaa" is an identifier.
Such tokens represent unspecified literal values (or "wildcards")
to be filled in later by the 
sqlite3_bind interface.
Each wildcard has an associated number which is its sequence in the
statement or the "nnn" in the case of a "?nnn" form. 
It is allowed for the same wildcard
to occur more than once in the same SQL statement, in which case
all instance of that wildcard will be filled in with the same value.
Unbound wildcards have a value of NULL.



   int sqlite3_bind_blob(sqlite3_stmt*, int, const void*, int n, void(*)(void*));
   int sqlite3_bind_double(sqlite3_stmt*, int, double);
   int sqlite3_bind_int(sqlite3_stmt*, int, int);
   int sqlite3_bind_int64(sqlite3_stmt*, int, long long int);
   int sqlite3_bind_null(sqlite3_stmt*, int);
   int sqlite3_bind_text(sqlite3_stmt*, int, const char*, int n, void(*)(void*));
   int sqlite3_bind_text16(sqlite3_stmt*, int, const void*, int n, void(*)(void*));
   int sqlite3_bind_value(sqlite3_stmt*, int, const sqlite3_value*);



There is an assortment of sqlite3_bind routines used to assign values
to wildcards in a prepared SQL statement.  Unbound wildcards
are interpreted as NULLs.  Bindings are not reset by sqlite3_reset().
But wildcards can be rebound to new values after an sqlite3_reset().



After an SQL statement has been prepared (and optionally bound), it
is executed using:



   int sqlite3_step(sqlite3_stmt*);



The sqlite3_step() routine return SQLITE_ROW if it is returning a single
row of the result set, or SQLITE_DONE if execution has completed, either
normally or due to an error.  It might also return SQLITE_BUSY if it is
unable to open the database file.  If the return value is SQLITE_ROW, then
the following routines can be used to extract information about that row
of the result set:



   const void *sqlite3_column_blob(sqlite3_stmt*, int iCol);
   int sqlite3_column_bytes(sqlite3_stmt*, int iCol);
   int sqlite3_column_bytes16(sqlite3_stmt*, int iCol);
   int sqlite3_column_count(sqlite3_stmt*);
   const char *sqlite3_column_decltype(sqlite3_stmt *, int iCol);
   const void *sqlite3_column_decltype16(sqlite3_stmt *, int iCol);
   double sqlite3_column_double(sqlite3_stmt*, int iCol);
   int sqlite3_column_int(sqlite3_stmt*, int iCol);
   long long int sqlite3_column_int64(sqlite3_stmt*, int iCol);
   const char *sqlite3_column_name(sqlite3_stmt*, int iCol);
   const void *sqlite3_column_name16(sqlite3_stmt*, int iCol);
   const unsigned char *sqlite3_column_text(sqlite3_stmt*, int iCol);
   const void *sqlite3_column_text16(sqlite3_stmt*, int iCol);
   int sqlite3_column_type(sqlite3_stmt*, int iCol);



The sqlite3_column_count()
function returns the number of columns in
the results set.  sqlite3_column_count() can be called at any time after
sqlite3_prepare_v2().  sqlite3_data_count() works similarly to
sqlite3_column_count() except that it only works following sqlite3_step().
If the previous call to sqlite3_step() returned SQLITE_DONE or an error code,
then sqlite3_data_count() will return 0 whereas sqlite3_column_count() will
continue to return the number of columns in the result set.


Returned data is examined using the other 
sqlite3_column_***() functions, 
all of which take a column number as their second parameter. Columns are
zero-indexed from left to right. Note that this is different to parameters,
which are indexed starting at one.



The sqlite3_column_type() function returns the
datatype for the value in the Nth column.  The return value is one
of these:



   #define SQLITE_INTEGER  1
   #define SQLITE_FLOAT    2
   #define SQLITE_TEXT     3
   #define SQLITE_BLOB     4
   #define SQLITE_NULL     5



The sqlite3_column_decltype() routine returns text which is the
declared type of the column in the CREATE TABLE statement.  For an
expression, the return type is an empty string.  sqlite3_column_name()
returns the name of the Nth column.  sqlite3_column_bytes() returns
the number of bytes in a column that has type BLOB or the number of bytes
in a TEXT string with UTF-8 encoding.  sqlite3_column_bytes16() returns
the same value for BLOBs but for TEXT strings returns the number of bytes
in a UTF-16 encoding.
sqlite3_column_blob() return BLOB data.  
sqlite3_column_text() return TEXT data as UTF-8.
sqlite3_column_text16() return TEXT data as UTF-16.
sqlite3_column_int() return INTEGER data in the host machines native
integer format.
sqlite3_column_int64() returns 64-bit INTEGER data.
Finally, sqlite3_column_double() return floating point data.



It is not necessary to retrieve data in the format specify by
sqlite3_column_type().  If a different format is requested, the data
is converted automatically.



Data format conversions can invalidate the pointer returned by
prior calls to sqlite3_column_blob(), sqlite3_column_text(), and/or
sqlite3_column_text16().  Pointers might be invalided in the following
cases:



The initial content is a BLOB and sqlite3_column_text() 
or sqlite3_column_text16()
is called.  A zero-terminator might need to be added to the string.


The initial content is UTF-8 text and sqlite3_column_bytes16() or
sqlite3_column_text16() is called.  The content must be converted to UTF-16.


The initial content is UTF-16 text and sqlite3_column_bytes() or
sqlite3_column_text() is called.  The content must be converted to UTF-8.



Note that conversions between UTF-16be and UTF-16le 
are always done in place and do
not invalidate a prior pointer, though of course the content of the buffer
that the prior pointer points to will have been modified.  Other kinds
of conversion are done in place when it is possible, but sometime it is
not possible and in those cases prior pointers are invalidated.  



The safest and easiest to remember policy is this: assume that any
result from

sqlite3_column_blob(),
sqlite3_column_text(), or
sqlite3_column_text16()

is invalided by subsequent calls to 

sqlite3_column_bytes(),
sqlite3_column_bytes16(),
sqlite3_column_text(), or
sqlite3_column_text16().

This means that you should always call sqlite3_column_bytes() or
sqlite3_column_bytes16() before calling sqlite3_column_blob(),
sqlite3_column_text(), or sqlite3_column_text16().


2.3 User-defined functions


User defined functions can be created using the following routine:



   typedef struct sqlite3_value sqlite3_value;
   int sqlite3_create_function(
     sqlite3 *,
     const char *zFunctionName,
     int nArg,
     int eTextRep,
     void*,
     void (*xFunc)(sqlite3_context*,int,sqlite3_value**),
     void (*xStep)(sqlite3_context*,int,sqlite3_value**),
     void (*xFinal)(sqlite3_context*)
   );
   int sqlite3_create_function16(
     sqlite3*,
     const void *zFunctionName,
     int nArg,
     int eTextRep,
     void*,
     void (*xFunc)(sqlite3_context*,int,sqlite3_value**),
     void (*xStep)(sqlite3_context*,int,sqlite3_value**),
     void (*xFinal)(sqlite3_context*)
   );
   #define SQLITE_UTF8     1
   #define SQLITE_UTF16    2
   #define SQLITE_UTF16BE  3
   #define SQLITE_UTF16LE  4
   #define SQLITE_ANY      5



The nArg parameter specifies the number of arguments to the function.
A value of 0 indicates that any number of arguments is allowed.  The
eTextRep parameter specifies what representation text values are expected
to be in for arguments to this function.  The value of this parameter should
be one of the parameters defined above.  SQLite version 3 allows multiple
implementations of the same function using different text representations.
The database engine chooses the function that minimization the number
of text conversions required.



Normal functions specify only xFunc and leave xStep and xFinal set to NULL.
Aggregate functions specify xStep and xFinal and leave xFunc set to NULL.
There is no separate sqlite3_create_aggregate() API.



The function name is specified in UTF-8.  A separate sqlite3_create_function16()
API works the same as sqlite_create_function()
except that the function name is specified in UTF-16 host byte order.



Notice that the parameters to functions are now pointers to sqlite3_value
structures instead of pointers to strings as in SQLite version 2.X.
The following routines are used to extract useful information from these
"values":



   const void *sqlite3_value_blob(sqlite3_value*);
   int sqlite3_value_bytes(sqlite3_value*);
   int sqlite3_value_bytes16(sqlite3_value*);
   double sqlite3_value_double(sqlite3_value*);
   int sqlite3_value_int(sqlite3_value*);
   long long int sqlite3_value_int64(sqlite3_value*);
   const unsigned char *sqlite3_value_text(sqlite3_value*);
   const void *sqlite3_value_text16(sqlite3_value*);
   int sqlite3_value_type(sqlite3_value*);



Function implementations use the following APIs to acquire context and
to report results:



   void *sqlite3_aggregate_context(sqlite3_context*, int nbyte);
   void *sqlite3_user_data(sqlite3_context*);
   void sqlite3_result_blob(sqlite3_context*, const void*, int n, void(*)(void*));
   void sqlite3_result_double(sqlite3_context*, double);
   void sqlite3_result_error(sqlite3_context*, const char*, int);
   void sqlite3_result_error16(sqlite3_context*, const void*, int);
   void sqlite3_result_int(sqlite3_context*, int);
   void sqlite3_result_int64(sqlite3_context*, long long int);
   void sqlite3_result_null(sqlite3_context*);
   void sqlite3_result_text(sqlite3_context*, const char*, int n, void(*)(void*));
   void sqlite3_result_text16(sqlite3_context*, const void*, int n, void(*)(void*));
   void sqlite3_result_value(sqlite3_context*, sqlite3_value*);
   void *sqlite3_get_auxdata(sqlite3_context*, int);
   void sqlite3_set_auxdata(sqlite3_context*, int, void*, void (*)(void*));


2.4 User-defined collating sequences


The following routines are used to implement user-defined
collating sequences:



   sqlite3_create_collation(sqlite3*, const char *zName, int eTextRep, void*,
      int(*xCompare)(void*,int,const void*,int,const void*));
   sqlite3_create_collation16(sqlite3*, const void *zName, int eTextRep, void*,
      int(*xCompare)(void*,int,const void*,int,const void*));
   sqlite3_collation_needed(sqlite3*, void*, 
      void(*)(void*,sqlite3*,int eTextRep,const char*));
   sqlite3_collation_needed16(sqlite3*, void*,
      void(*)(void*,sqlite3*,int eTextRep,const void*));



The sqlite3_create_collation() function specifies a collating sequence name
and a comparison function to implement that collating sequence.  The
comparison function is only used for comparing text values.  The eTextRep
parameter is one of SQLITE_UTF8, SQLITE_UTF16LE, SQLITE_UTF16BE, or
SQLITE_ANY to specify which text representation the comparison function works
with.  Separate comparison functions can exist for the same collating
sequence for each of the UTF-8, UTF-16LE and UTF-16BE text representations.
The sqlite3_create_collation16() works like sqlite3_create_collation() except
that the collation name is specified in UTF-16 host byte order instead of
in UTF-8.



The sqlite3_collation_needed() routine registers a callback which the
database engine will invoke if it encounters an unknown collating sequence.
The callback can lookup an appropriate comparison function and invoke
sqlite_3_create_collation() as needed.  The fourth parameter to the callback
is the name of the collating sequence in UTF-8.  For sqlite3_collation_need16()
the callback sends the collating sequence name in UTF-16 host byte order.


c2: SQLite Older News
c5: oldnews.html
c4:
2015-11-02 - Release 3.9.2
SQLite version 3.9.2 is a patch release fixing two obscure bugs.
(Details: (1),
(2)).
Upgrade only if you are having problems.
2015-10-16 - Release 3.9.1
SQLite version 3.9.1 is a
small patch
to version 3.9.0 that includes
a few simple build script and #ifdef tweaks to make the code easier to 
compile on a wider variety of platform.  There are no functional changes, 
except for a single
minor bug-fix in 
the json1 extension to stop it from recognizing form-feed 
(ASCII 0x0c) as a whitespace character, in conformance with 
RFC7159.
2015-10-14 - Release 3.9.0
SQLite version 3.9.0 is a regularly schedule maintenance release.
Key changes include:

Begin using semantic versioning.
JSON SQL functions
The FTS5 full-text search engine
Support for indexes on expressions
Support for table-valued functions

See the change log for a long and more complete list
of changes.
2015-07-29 - Release 3.8.11.1
SQLite version 3.8.11.1 is a patch release that fixes two arcane
   issues that were reported shortly after 3.8.11 was released.  Upgrade
   from 3.8.11 only in the unlikely event that one of these obscure
   issues affect your code.
2015-07-27 - Release 3.8.11
SQLite version 3.8.11 is a regularly scheduled maintenance release.
   See the change log for details.
2015-05-20 - Release 3.8.10.2
Yikes!  Index corruption after a sequence of valid SQL statements!
It has been many years since anything like 
   this bug has snuck into
   an official SQLite release.  But for the pasts seven months
   (version 3.8.7 through version 3.8.10.1)
   if you do an INSERT into a carefully
   crafted schema in which there are two nested triggers that convert
   an index key value from TEXT to INTEGER and then back
   to TEXT again, the INTEGER value might get inserted as the index
   key instead of the correct TEXT, resulting in index corruption.
   This patch release adds a single line of code to fix the problem.
If you do actually encounter this problem, running REINDEX on the
   damaged indexes will clear it.
2015-05-09 - Release 3.8.10.1
The 3.8.10 release did not add the new SQLITE_ENABLE_DBSTAT_VTAB
   compile-time option to the sqlite3_compileoption_used() interface.
   This patch release fixes that omission.  And while we are at it,
   the associated dbstat virtual table was enhanced slightly and a
   harmless compiler warning was fixed.

   There is no reason to upgrade from version 3.8.10 unless you are
   using the new SQLITE_ENABLE_DBSTAT_VTAB compile-time option.
2015-05-07 - Release 3.8.10
SQLite version 3.8.10 is a regularly scheduled maintenance release.
   This release features performance improvements, fixes to several
   arcane bugs found by the AFL fuzzer, the new "sqldiff.exe" command-line
   utility, improvements to the documentation, and other enhancements.
   See the release notes for 
   additional information.
2015-04-08 - Release 3.8.9
SQLite version 3.8.9 is a regularly scheduled maintenance release.
   New features in this release include the 
   PRAGMA index_xinfo command, the sqlite3_status64() interface,
   and the ".dbinfo" command of the command-line shell.
   See the release notes for
   additional information.
2015-02-25 - Release 3.8.8.3
The 3.8.8.3 patch release fixes an obscure problem in the SQLite code
   generator that can cause incorrect results when the qualifying expression
   of a partial index is used inside the ON clause of a LEFT JOIN.
   This problem has been in the code since support for partial indexes
   was first added in version 3.8.0.  However, it is difficult to imagine
   a valid reason to every put the qualifying constraint inside the ON
   clause of a LEFT JOIN, and so this issue has never come up before.

Any applications that is vulnerable to this bug would have encountered
   problems already.  Hence, upgrading from the previous release is optional.
2015-01-30 - Release 3.8.8.2
The 3.8.8.2 patch release fixes a single minor problem:  It ensures
   that the sqlite3_wal_checkpoint(TRUNCATE) operation will always truncate
   the write-ahead log even if log had already been reset and contained
   no new content.  It is unclear if this is a bug fix or a new feature.

Something like this would normally go into the next regularly scheduled
   release, but a prominent SQLite user needed the change in a hurry so
   we were happy to rush it out via this patch.

There is no reason to upgrade unless you actually need the enhanced
   behavior of sqlite3_wal_checkpoint(TRUNCATE).
2015-01-20 - Release 3.8.8.1
Within hours of releasing version 3.8.8, a bug was reported against
   the 10-month-old 3.8.4 release.  As that bug exists in all subsequent
   releases, the decision was made to issue a small patch to the
   3.8.8 before it came into widespread use.

See ticket
   f97c4637102a3ae72b7911
   for a description of the bug.

The changes between versions 3.8.8 and 3.8.8.1 are minimal.
2015-01-16 - Release 3.8.8
SQLite version 3.8.8 is a regularly schedule maintenance release of
   SQLite.

There are no dramatic new features or performance enhancements in this
   release, merely incremental improvements.  Most of the performance gain
   in this release comes from refactoring the B-Tree rebalancing logic to
   avoid unnecessary memcpy() operations.  New features include the
   PRAGMA data_version statement and the ability to accept a
   VALUES clause with no arbitrary limit on the number of rows.
   Several obscure bugs have been fixed, including some multithreading
   races and a work-around for a compiler bug on some Macs.

See the change log for a longer list of
   enhancements and bug fixes.
2014-12-09 - Release 3.8.7.4
SQLite version 3.8.7.4 an unscheduled bug-fix release.  Changes from
   the previous release and from version 3.8.7 are minimal.

This release fixes adds in a mutex that is required by the changes of
   the 3.8.7.3 patch but was accidentally omitted.  The mutex was not required
   by any of the internal SQLite tests, but Firefox crashes without it.
   Test cases have been added to ensure that mutex is never again missed.
2014-12-06 - Release 3.8.7.3
SQLite version 3.8.7.3 an unscheduled bug-fix release.  Changes from
   the previous release and from version 3.8.7 are minimal.

This release fixes two obscure bugs that can result in incorrect
   query results and/or application crashes, but not (as far as we can
   tell) security vulnerabilities.  Both bugs have been latent in the
   code across multiple prior releases and have never before been encountered,
   so they are unlikely to cause problems.  Nevertheless
   it seems prudent to publish fixes for them both.  See the
   change log for details.
2014-11-19 - Release 3.8.7.2
SQLite version 3.8.7.2 is a patch and bug-fix release.  Changes from
   the previous release are minimal.

The primary reason for this release is to enhance the ROLLBACK command
   so that it allows running queries on the same database connection to 
   continue running as long as the ROLLBACK does not change the schema.
   In all previous versions of SQLite, a ROLLBACK would cause pending
   queries to stop immediately and return SQLITE_ABORT or
   SQLITE_ABORT_ROLLBACK. Pending queries still abort if the ROLLBACK
   changes the database schema, but as of this patch release, the queries
   are allowed to continue running if the schema is unmodified.

In addition to the ROLLBACK enhancement, this patch release also
   includes fixes for three obscure bugs.  See the 
   change log for details.
2014-10-30 - Release 3.8.7.1
SQLite version 3.8.7.1 is a bug-fix release.

The primary reason for this bug-fix release is to address a problem with
   updating the value of fields at the end of a table that were added
   using ALTER TABLE ADD COLUMN.  This problem
   1 first appeared in the
   3.8.7 release.

Another minor annoyance in the 3.8.7 release was the fact that the
   Android build tried to use the strchrnul() function from the standard
   C library but that function is not available on Android.  Android builds
   had to add -DHAVE_STRCHRNUL=0 to work around the problem.  This patch
   fixes that so that Android builds should now work without any changes.

The operation of PRAGMA journal_mode=TRUNCATE has been enhanced so that
   it invokes fsync() after truncating the journal file when
   PRAGMA synchronous=FULL.  This helps to preserve transaction durability
   in the case of a power loss occurring shortly after commit.

Finally, a couple of long-standing and obscure problems associated with run
   UPDATE and DELETE on VIEWs were fixed.

The changes from 3.8.7 are minimal.
2014-10-17 - Release 3.8.7
SQLite version 3.8.7 is a regularly scheduled maintenance release.
   Upgrading from all prior versions is recommended.

Most of the changes from the previous release have been micro-optimizations
   designed to help SQLite run a little faster.  Each individual optimization
   has an unmeasurably small performance impact.  But the improvements add up.
   Measured on a well-defined workload (which the SQLite developers use
   as a proxy for a typical application workload) using cachegrind on Linux
   and compiled with gcc 4.8.1 and -Os on x64 linux, the current release 
   does over 20% more work for the same number of CPU cycles compared to the
   previous release.  Cachegrind is not a real CPU, and the workload
   used for measurement is only a proxy.  So your performance may vary.
   We expect to see about half the measured and reported improvement in
   real-world applications. 10% is less than 20% but it is still pretty
   good, we think.

This release includes a new set of C-language interfaces that have
   unsigned 64-bit instead of signed 32-bit length parameters.  The new
   APIs do not provide any new capabilities.  But they do make it easier
   to write applications that are more resistant to integer overflow
   vulnerabilities.

This release also includes a new sorter that is able to use multiple
   threads to help with large sort operations.  (Sort operations are
   sometimes required to implement ORDER BY and/or GROUP BY clauses and
   are almost always required for CREATE INDEX.)  The multi-threads sorter
   is turned off by default and must be enabled using the
   PRAGMA threads SQL command.  Note that the multi-threaded sorter
   provides faster real-time performance for large sorts, but it also
   uses more CPU cycles and more energy.
2014-08-15 - Release 3.8.6
SQLite version 3.8.6 is a regularly scheduled maintenance release.
   Upgrading from all previous versions is recommended.

This release contains the usual assortment of obscure bug fixes.
   One bug, however, deserves special attention.
   A problem appeared in the CREATE INDEX command beginning with
   version 3.8.2 (2013-12-06) that allowed, under some circumstances,
   a UNIQUE index to be created on a column that was not unique.  Once
   the index was created, no new non-unique entries could be inserted, but
   preexisting non-unique entries would remain.  See ticket
   9a6daf340df99ba93c
   for further information.  In addition to fixing this bug, the
   PRAGMA integrity_check command has been enhanced to detect
   non-uniqueness in UNIQUE indices, so that if this bug did introduce
   any problems in databases, those problems can be easily detected.

Other noteworthy changes include the addition of support for
   hexadecimal integers (ex: 0x1234), and performance enhancements
   to the IN operator which, according to 
   mailing list reports,
   help some queries run up to five times faster.

Version 3.8.6 uses 25% fewer CPU cycles than version 3.8.0 from
   approximately one year ago, according to valgrind
   and the
   test/speedtest1.c
   test program.
   On the other hand,
   the compiled binary for version 3.8.6 is about 5% larger than 3.8.0.
   The size increase is 
   due in part to the addition of new features such as WITHOUT ROWID
   tables and common table expressions.
2014-06-04 - Release 3.8.5
SQLite version 3.8.5 is a regularly scheduled maintenance release.
   Upgrading from the previous version is recommended.

Version 3.8.5 fixes more than a dozen obscure bugs.  None of these
   bugs should be a problem for existing applications.  Nor do any of
   the bugs represent a security vulnerability.  Nevertheless, upgrading
   is recommended to prevent future problems.

In addition to bug fixes, the 3.8.5 release adds improvements to the
   query planner, especially regarding sorting using indices and handling 
   OR terms
   in the WHERE clause for WITHOUT ROWID tables.  The ".system" and
   ".once" dot-commands were added to the command-line interface.  And
   there were enhancements to the FTS4 and RTREE virtual tables.  See
   the change log for details.
2014-04-03 - Release 3.8.4.3
The optimizations added in version 3.8.4 caused some queries that involve
   subqueries in the FROM clause, DISTINCT, and ORDER BY clauses, to give an incorrect
   result.  See
   ticket 98825a79ce145
   for details.
   This release adds a 
   one-character change
   to a single line of code to fix the problem.
2014-03-26 - Release 3.8.4.2
The code changes that resulted in the performance improvements
   in version 3.8.4 missed a single buffer overflow test, which could
   result in a read past the end of a buffer while searching a database
   that is corrupted in a particular way.  Version 3.8.4.2 fixes that
   problem using a
   one-line patch.

We are not aware of any problems in version 3.8.4
   when working with well-formed database files.  The problem fixed by this
   release only comes up when reading corrupt database files.
2014-03-11 - Release 3.8.4.1
SQLite version 3.8.4.1 is a patch against version 3.8.4 that fixes
   two minor issues:

 Work around a C-preprocessor macro conflict that causes compilation
     problems for some configurations of Visual Studio.
 Adjust the cost computation for the skip-scan optimization for
     improved performance.

Both of these issues came to light within minutes of tagging the previous
release.  Neither issue is serious but they can be annoying.  Hence, the
decision was made to do a quick patch release to address both issues.
2014-03-10 - Release 3.8.4
SQLite version 3.8.4 is a maintenance release featuring performance
   enhancements and fixes for a number of obscure bugs.
   There are no significant new features in SQLite version 3.8.4.
   However, the number of CPU cycles (measured by valgrind) needed to 
   do many common operations has be reduced by about 12% relative to the
   previous release, and by about 25% relative to version 3.7.16
   from approximately one year ago.

Version 3.8.4 of SQLite fixes several corner-case bugs that were
   found since the previous release.  These bugs were unlikely to appear
   in practice, and none represent a security vulnerability. 
   Nevertheless, developers are encouraged to upgrade from all prior releases.
2014-02-11 - Release 3.8.3.1
SQLite version 3.8.3.1 fixes a bug present in versions 3.8.1,
   3.8.2 and 3.8.3 that can cause queries to omit valid output rows.
   Upgrading from those versions is recommended.

The problem only comes up if SQLite is compiled with either the
   SQLITE_ENABLE_STAT3 or SQLITE_ENABLE_STAT4 compile-time options.
   In that case, if a query has a WHERE clause that contains expressions
   like this:
   
WHERE (expr1 OR expr2 OR ... OR exprN) AND column IS NOT NULL
   
   Where all of expr1 through exprN are suitable for use by indexes,
   then during query planning SQLite might mistakenly converted
   the "column IS NOT NULL" term into "column>NULL".  But the latter
   term is never true, and so the query would return no rows.

The trouble ticket for this bug is
   &#91;4c86b126f2&#93;.
   It is recommended that all users upgrade to avoid this problem.
2014-02-03 - Release 3.8.3
  SQLite version 3.8.3 is a regularly scheduled maintenance release.
  Upgrading from the previous release is optional.

  The most visible change in version 3.8.3 is the addition of
  support for common table expressions.  It is now possible to write a
  single SELECT statement that will query a tree or graph, using either
  a depth-first or a breadth-first search.  A single SQLite query will 
  even solve Sudoku puzzles or compute the Mandelbrot set.  As part
  of this change, SQLite now accepts a VALUES clause anyplace that
  a SELECT statement is valid.

  This release also includes many small performance enhancements which
  should give a small speed boost to legacy applications.  And there are
  other minor enhancements such as the addition of the printf() SQL
  function.  See the change log for details.
2013-12-06 - Release 3.8.2
  SQLite version 3.8.2 is a regularly scheduled maintenance release.
  Upgrading from the previous release is optional.

  Version 3.8.2 adds support for WITHOUT ROWID tables.  This is a
  significant extension to SQLite.  Database files that contain WITHOUT ROWID
  tables are not readable or writable by prior versions of SQLite, however
  databases that do not use WITHOUT ROWID tables are fully backwards 
  and forwards compatible.

  The 3.8.2 release contains a potentially incompatible change.  In
  all prior versions of SQLite, a cast from a very large positive 
  floating point number into an integer resulted in the most negative integer.
  In other words, CAST(+99.9e99 to INT) would yield -9223372036854775808.
  This behavior came about because it is what x86/x64 hardware does 
  for the equivalent cast in the C language.  But the behavior is
  bizarre.  And so it has been changed effective with this release so that
  a cast from a floating point number into an integer returns the integer
  between the floating point value and zero that is closest to the floating
  point value.  Hence, CAST(+99.9e99 to INT) now returns +9223372036854775807.
  Since routines like sqlite3_column_int64() do an implicit cast if the
  value being accessed is really a floating point number, they are also
  affected by this change.

  Besides the two changes mentioned above, the 3.8.2 release also 
  includes a number of performance enhancements.  The
  skip-scan optimization is now available for databases that have been
  processed by ANALYZE.  Constant SQL functions are now factored out of
  inner loops, which can result in a significant speedup for queries that
  contain WHERE clause terms like "date>datetime('now','-2 days')".  And
  various high-runner internal routines have been refactored for reduced
  CPU load.
2013-10-17 - Release 3.8.1
  SQLite version 3.8.1 is a regularly scheduled maintenance release.
  Upgrading from the previous release is optional, though you should upgrade
  if you are using partial indices as there was a 
  bug related to partial
  indices in the previous release that could result in an incorrect answer
  for count(*) queries.

  The next generation query planner that was premiered in the previous
  release continues to work well.  
  The new query planner has been tweaked slightly
  in the current release to help it make better decisions in some
  cases, but is largely unchanged.  Two new SQL functions, likelihood() and
  unlikely(), have been added to allow developers to give hints to the
  query planner without forcing the query planner into a particular decision.

  Version 3.8.1 is the first SQLite release to take into account the
  estimated size of table and index rows when choosing a query plan.
  Row size estimates are based on the declared datatypes of columns.
  For example, a column of type VARCHAR(1000) is assumed
  to use much more space than a column of type INT.  The datatype-based
  row size estimate can be
  overridden by appending a term of the form "sz=NNN" (where NNN is the
  average row size in bytes) to the end of the sqlite_stat1.stat
  record for a table or index.  Currently, row sizes are only used to help the
  query planner choose between a table or one of its indices when doing a
  table scan or a count(*) operation, though future releases are likely
  to use the estimated row size in other contexts as well.  The new
  PRAGMA stats statement can be used to view row size estimates.

  Version 3.8.1 adds the SQLITE_ENABLE_STAT4 compile-time option.
  STAT4 is very similar to STAT3 in that it uses samples from indices to 
  try to guess how many rows of the index will be satisfy by WHERE clause
  constraints.  The difference is that STAT4 samples all columns of the
  index whereas the older STAT3 only sampled the left-most column.  Users
  of STAT3 are encouraged to upgrade to STAT4.  Application developers should
  use STAT3 and STAT4 with caution since both options, by design, violate 
  the query planner stability guarantee, making it more difficult to ensure
  uniform performance is widely-deployed and mass-produced embedded
  applications.
2013-09-03 - Release 3.8.0.2
  SQLite version 3.8.0.2 contains a one-line fix to a bug in the
  new optimization that tries to omit unused LEFT JOINs from a query.
2013-08-29 - Release 3.8.0.1
  SQLite version 3.8.0.1 fixes some obscure bugs that were uncovered by
  users in the 3.8.0 release.  Changes from 3.8.0 are minimal.
2013-08-26 - Release 3.8.0
  Do not fear the zero!

  SQLite version 3.8.0 might easily have been called "3.7.18" instead.
  However, this release features the cutover of the
  next generation query planner or NGQP, and there is a small chance of
  breaking legacy programs that
  rely on undefined behavior in previous SQLite releases, and so the
  minor version number was incremented for that reason.
  But the risks are low and there is a query planner checklist is 
  available to application developers to aid in avoiding problems.

  SQLite version 3.8.0 is actually one of the most heavily tested
  SQLite releases ever.  Thousands and thousands of beta copies have
  be downloaded, and presumably tested, and there have been no problem
  reports.

  In addition to the next generation query planner, the 3.8.0 release
  adds support for partial indices, as well as several other new features.
  See the change log for further detail.
2013-05-20 - Release 3.7.17
  SQLite version 3.7.17 is a regularly schedule maintenance release.
  Visit the change log for a full explanation of the
  changes in this release.

  There are many bug fixes in version 3.7.17.  But this does not indicate
  that 3.7.16 was a problematic release.  All of the bugs in 3.7.17 are 
  obscure and are unlikely to impact any particular application.  And most 
  of the bugs that are fixed in 3.7.17 predate 3.7.16 and have been in 
  the code for years without ever before being noticed.
  Nevertheless, due to the large number of fixes,
  all users are encouraged to upgrade when possible.
2013-04-12 - Release 3.7.16.2
  SQLite version 3.7.16.2 fixes a long-standing flaw in the Windows
  OS interface that
  can result in database corruption under a rare race condition.
  See http://www.sqlite.org/src/info/7ff3120e4f for a full description
  of the problem.

  As far as we know, this bug has never been seen in the wild.  The
  problem was discovered by the SQLite developers while writing stress tests
  for a separate component of SQLite.  Those stress tests have not yet
  found any problems with the component they were intended to verify, but 
  they did find the bug which is the subject of this patch release.

  Other than updates to version numbers, the only difference between this
  release and 3.7.16.1 is a two-character change in a single identifier,
  which is contained in the windows-specific OS interface logic.  There 
  are no changes in this release (other than version numbers) for platforms
  other than Windows.
2013-03-29 - Release 3.7.16.1
  SQLite version 3.7.16.1 is a bug fix release that fixes a few problems
  that were present in the previous releases.

  The primary motivation for version 3.7.16.1 is to fix a bug in the
  query optimizer that was introduced as part of version 3.7.15.  The
  query optimizer was being a little overzealous in optimizing out some
  ORDER BY clauses, which resulted in sorting being omitted on occasions
  where sorting is required to get the correct answer.  See
  ticket a179fe7465 for
  details.

  In addition to the ORDER BY fix, several other patches to fix obscure
  (and mostly harmless) bugs and to fix spelling errors in source code
  comments are also included in this release.
2013-03-18 - Release 3.7.16
  SQLite version 3.7.16 is a regularly scheduled release of SQLite.
  This release contains several language enhancements and improvements
  to the query optimizer.  A list of the major enhancements and optimizations
  can be see on the change log.

  There was one important bug fix
  (see Ticket fc7bd6358f)
  that addresses an incorrect query result that could have occurred in
  a three-way join where the join constraints compared INTEGER columns
  to TEXT columns.  This issue had been in the code for time out of mind
  and had never before been reported, so we surmise that it is very obscure.
  Nevertheless, all users are advised to upgrade to avoid any future problems
  associated with this issue.
2013-01-09 - Release 3.7.15.2
  SQLite version 3.7.15.2 is a patch release that fixes a single bug
  that was introduced in version version 3.7.15.  The fix is a 4-character
  edit to a single line of code.  Other than this 4-character change and
  the update of the version number, nothing has changed from
  version 3.7.15.1.
2012-12-19 - Release 3.7.15.1
  SQLite version 3.7.15.1 is a patch release that fixes a single bug
  that was introduced in version version 3.7.15.  The fix involved changing
  two lines of code and adding a single assert().  This release also includes
  some new test cases to prevent a regression of the bug, and the version
  number is increased, of course.  But otherwise, nothing has changed from
  version 3.7.15.
2012-12-12 - Release 3.7.15
  SQLite version 3.7.15 is a regularly schedule release of SQLite.  This
  release contains several improvements to the query planner and optimizer
  and one important bug fix.  This is the first release to officially
  support Windows 8 Phone.

  The important bug fix is a problem that can lead to segfaults when using
  shared cache mode on a schema that contains a COLLATE operator within
  a CHECK constraint or within a view.  Collating functions are associated
  with individual database connections.  But a pointer to the collating function
  was also being cached within expressions.  If an expression was part of the
  schema and contained a cached collating function, it would point to the
  collating function in the database connection that originally parsed the
  schema.  If that database connection closed while other database 
  connections using the same shared cache continued to operate, they other
  database connections would try to use the deallocated collating function
  in the database connection that closed.  The fix in version 3.7.15 was to
  not cache collating function pointers in the expression structure but 
  instead look them up each time a new statement is prepared.

  This release also contains some important enhancements to the query planner
  which should (we hope) make some queries run faster.  The enhancements
  include:

  When doing a full-table scan, try to use an index instead of
  the original table, under the theory that indices contain less information
  and are thus smaller and hence require less disk I/O to scan.

  Enhance the IN operator to allow it to make use of 
  indices that have numeric affinity.

  Do a better job of recognizing when an ORDER BY clause can be 
  implemented using indices - especially in cases where the ORDER BY clause
  contains terms from two or more tables in a join.
  
2012-10-04 - Release 3.7.14.1
  SQLite version 3.7.14.1 is a patch release.  Changes from the baseline
  version 3.7.14 are minimal and are restricted to fixing three bugs.

  One of the fixed bugs is a long-standing issue with the TCL interface.
  Another is an external compiler bug that SQLite merely works around and
  that only comes up if you are using the VisualStudio-2012 compiler to
  generate WinRT applications on ARM with optimizations enabled.  The
  third problem is an SQLite core bug, introduced in version 3.7.14, that
  can cause a segfault if a query contains a LEFT JOIN that contains an OR
  in the ON clause.
2012-09-03 - Release 3.7.14
  SQLite version 3.7.14 is a regularly scheduled maintenance release
  of SQLite.  The previous release continues to work well.  Upgrading
  is optional.

  Version 3.7.14 drops native support for OS/2.  We are not aware of any
  active projects that were using SQLite on OS/2 and since the SQLite
  developers had no way of testing on OS/2 it seemed like it was time
  to simply remove the OS/2 code from the SQLite tree.  If there are
  OS/2 projects out there that still need SQLite support, they can
  continue to maintain their own private VFS which can be linked to
  SQLite at start-time using the sqlite3_vfs_register() interface.

  The sqlite3_close_v2() interface has been added.  The sqlite3_close_v2()
  interface differs from sqlite3_close() in that it is designed to work
  better for host language that use a garbage collector.  With the older
  sqlite3_close() interface, the associated prepared statements and
  sqlite3_backup objects must be destroyed before the database connection.
  With the newer sqlite3_close_v2() interface, the objects can be destroyed
  in any order.

  This release also includes performance improvements to the sort algorithm
  that is used to implement ORDER BY and CREATE INDEX.  And the query planner
  has been enhanced to better use covering indices on queries that use OR
  terms in the WHERE clause.
2012-06-11 - Release 3.7.13
  SQLite version 3.7.13 adds support for WinRT and metro style
  applications for Microsoft Windows 8.  The 3.7.13 release is
  coming sooner than is usual after the previous release in order to get
  this new capability into the hands of developers.  To use SQLite in 
  a metro style application, compile with the -DSQLITE_OS_WINRT flag.
  Because of the increased application security and safety requirements
  of WinRT, all database
  filenames should be full pathnames.  Note that SQLite is not capable
  of accessing databases outside the installation directory and application
  data directory.  This restriction is another security and safety feature
  of WinRT. Apart from these restrictions, SQLite should work exactly
  the same on WinRT as it does on every other system.

  Also in this release: when a database is opened using URI filenames
  and the mode=memory query parameter 
  then the database is an in-memory database, just as if it had
  been named ":memory:".  But, if shared cache mode is enabled, then
  all other database connections that specify the same URI filename
  will connect to the same in-memory database.  This allows two or more
  database connections (in the same process) to share the same in-memory
  database.

  This release also includes some corner-case performance optimizations
  that are obscure yet significant to an important subset of SQLite users.
  Getting these performance optimizations into circulation quickly is
  yet another reason for making this release so soon following the previous.

  The next release of SQLite is scheduled to occur after the usual
  2 or 3 month interval.
2012-05-22 - Patch Release 3.7.12.1
  SQLite version 3.7.12.1 is a patch release for version 3.7.12 that
  fixes a bug that was
  introduced in version 3.7.12 and that can
  cause a segfault for certain obscure nested aggregate queries.
  There are very few changes in 3.7.12.1, and upgrading is only needed for
  applications that do nested aggregate queries.
2012-05-14 - Version 3.7.12
  SQLite version 3.7.12 is a regularly scheduled maintenance release.
  This release contains several new optimizations and bug fixes and upgrading
  is recommended.  See the change summary for details.
2012-03-20 - Version 3.7.11
  SQLite version 3.7.11 is a regularly scheduled maintenance release
  which was rushed out early due to a 
  bug in the query optimizer
  introduced in the previous release.  The bug is obscure - it changes
  a LEFT JOIN into an INNER JOIN in some cases when there is a 3-way join
  and OR terms in the WHERE clause.  But it was considered serious enough to
  rush out a fix.  Apart from this one problem, SQLite version 3.7.10 has
  not given any trouble.  Upgrading to version 3.7.11 from versions
  3.7.6.3, 3.7.7, 3.7.7.1, 3.7.8, or 3.7.9 is
  optional.  Upgrading from other releases, including the previous release
  3.7.10, is recommended.

  Other enhancements found in this release are enumerated in the
  change log.
2012-01-16 - Version 3.7.10
  SQLite version 3.7.10 is a regularly scheduled maintenance release.
  Upgrading from version 3.7.6.3, 3.7.7, 3.7.7.1, 3.7.8, or 3.7.9 is
  optional.  Upgrading from other releases is recommended.

  The SQLITE_CONFIG_PCACHE mechanism has been replaced with
  SQLITE_CONFIG_PCACHE2.  If you do not know what this mechanism
  is (it is an extreme corner-case and is seldom used) then this 
  change will not effect you in the least.

  The default schema format number for new database files has changed
  from 1 to 4.  SQLite has been able to generate and read database files
  using schema format 4 for six years.  But up unto now, the default
  schema format has been 1 so that older versions of SQLite could read
  and write databases generated by newer versions of SQLite.  But those
  older versions of SQLite have become so scarce now that it seems 
  reasonable to make the new format the default.

  SQLite is changing some of the assumptions it makes above the behavior
  of disk drives and flash memory devices during a sudden power loss.
  This change is completely transparent to applications.
  Read about the powersafe overwrite property for additional information.

  Lots of new interfaces have been added in this release:
  
   sqlite3_db_release_memory()
   PRAGMA shrink_memory
   sqlite3_db_filename()
   sqlite3_stmt_busy()
   sqlite3_uri_boolean()
   sqlite3_uri_int64()
  

  The PRAGMA cache_size statement has been enhanced.  Formerly, you would
  use this statement to tell SQLite how many pages of the database files it
  should hold in its cache at once.  The total memory requirement would
  depend on the database page size. Now, if you give PRAGMA cache_size
  a negative value -N, it will allocate roughly N 
  kibibytes of memory to cache,
  divided up according to page size.  This enhancement allows programs to
  more easily control their memory usage.

  There have been several obscure bug fixes.  One noteworthy bug,
  ticket ff5be73dee,
  could in theory result in a corrupt database file if a power loss
  occurred at just the wrong moment on an unusually cantankerous disk
  drive.  But that is mostly a theoretical concern and is very unlikely
  to happen in practice.  The bug was found during laboratory testing
  and has never been observed to occur in the wild.
2011-11-01 - Version 3.7.9
  SQLite version 3.7.9 is a regularly scheduled maintenance release.
  Upgrading from version 3.7.6.3, 3.7.7, 3.7.7.1, and 3.7.8 is optional.
  Upgrading from other versions is recommended.

  The SQLITE_ENABLE_STAT2 compile-time option is now a no-op.  The enhanced
  query-planner functionality formerly available using SQLITE_ENABLE_STAT2
  is now available through SQLITE_ENABLE_STAT3.  The enhanced query planning
  is still disabled by default.  However, future releases of SQLite might
  convert STAT3 from an enable-option to a disable-option so that it is 
  available by default and is only omitted upon request.

  The FTS4 full-text search engine has been enhanced such that tokens in
  the search string that begin with "&#94;" must be the first token in their
  respective columns in order to match.   Formerly, "&#94;" characters in the
  search string were simply ignored.  Hence, if a legacy application was
  including "&#94;" characters in FTS4 search strings, thinking that they would
  always be ignored, then those legacy applications might break with this
  update.  The fix is simply remove the "&#94;" characters from the search 
  string.

  See the change summary for additional changes associated
  with this release.  
2011-September-19 - Version 3.7.8
  SQLite version 3.7.8 is a quarterly maintenance release.  Upgrading from
  versions 3.7.6.3, 3.7.7, or 3.7.7.1 is optional.  Upgrading from other
  versions is recommended.

  This release features a new "external merge sort" algorithm used to
  implement ORDER BY and GROUP BY and also to presort the content of an
  index for CREATE INDEX.  The new algorithm does approximately the same
  number of comparisons and I/Os as before, but the I/Os are much more
  sequential and so runtimes are greatly reduced when the size of the
  set being sorted is larger than the filesystem cache.  The performance 
  improvement can be dramatic - orders of magnitude faster 
  for large CREATE INDEX commands.  On the other hand, 
  the code is slightly slower (1% or 2%)
  for a small CREATE INDEX.  Since CREATE INDEX is not an
  operation that commonly occurs on a speed-critical path, we feel that 
  this tradeoff is a good one.  The slight slowdown for small CREATE INDEX
  statements might be recovered in a future release.  ORDER BY and GROUP BY
  operations should now be faster for all cases, large and small.

  The query planner has been enhanced to do a better job of handling
  the DISTINCT keyword on SELECT statements.

  There has been a lot of work on the default VFSes.  The unix VFS has
  been enhanced to include more overrideable system calls - a feature requested
  by Chromium to make it easier to build SQLite into a sandbox.  The
  windows VFS has been enhanced to be more resistant to interference from
  anti-virus software.

  Every version of SQLite is better tested than the previous, and 3.7.8
  is no exception to this rule.  Version 3.7.8 has been used internally by
  the SQLite team for mission critical functions and has performed flawlessly.
  And, of course, it passes our rigorous testing procedures with no
  problems detected.  Version 3.7.8 is recommended for all new development.
2011-06-28 - Version 3.7.7.1
  SQLite version 3.7.7.1 adds a one-line bug fix to 3.7.7 to fix
  a problem
  causing PRAGMA case_sensitive_like statements compiled using the legacy
  sqlite3_prepare() interface to fail with an SQLITE_SCHEMA error. Because
  sqlite3_exec() uses sqlite3_prepare() internally, the problem also affects
  sqlite3_exec().

  Upgrading from 3.7.7 is only required for applications that use "PRAGMA
  case_sensitive_like" and the sqlite3_prepare() (or sqlite3_exec()) interface.
2011-06-24 - Version 3.7.7
  SQLite version 3.7.7 is a regularly scheduled bi-monthly maintenance
  release.  Upgrading from version 3.7.6.3 is optional.  Upgrading from all
  prior releases is recommended.

  This release adds support for naming database files using URI filenames.
  URI filenames are disabled by default (for backwards compatibility) but
  applications are encouraged to enable them since incompatibilities are
  likely to be exceedingly rare and the feature is useful.  See the 
  URI filename documentation for details.

  Most of the other enhancements in this release involve 
  virtual tables.  The virtual table interface has been enhanced to
  support SAVEPOINT and ON CONFLICT clause processing, and the built-in 
  RTREE and FTS3/FTS4 have been augmented to take advantage of 
  the new capability.  This means, for example, that it is now possible
  to use the REPLACE command on FTS3/FTS4 and RTREE tables.

  The FTS4 full-text index extension has been enhanced to support
  the FTS4 prefix option and the FTS4 order option.  These two enhancements
  are provided in support of search-as-you-type interfaces where search
  results begin to appear after the first keystroke in the "search" box
  and are refined with each subsequent keystroke.  The way this is done is
  to do a separate full-text search after each key stroke, and add the
  "*" wildcard at the end of the word currently being typed.  So, for
  example, if the text typed so far is "fast da" and the next character
  typed is "t", then the application does a full-text search of the
  pattern "fast dat*" and displays the results.  Such capability has
  always existed.  What is new is that the FTS4 prefix option allows
  the search to be very fast (a matter of milliseconds) even for difficult
  cases such as "t*" or "th*".

  There has been a fair amount of work done on the FTS4 module for this
  release.  But the core SQLite code has changed little and the previous
  release has not given any problems, so we expect this to be a very
  stable release.
2011-05-19 - Version 3.7.6.3
  SQLite version 3.7.6.3 is a patch release that fixes a 
  single bug
  associated with WAL mode.  The bug has been in SQLite ever since WAL
  was added, but the problem is very obscure and so nobody has noticed
  before now.  Nevertheless, all users are encouraged to upgrade to
  version 3.7.6.3 or later.

  The bug is this:
  If the cache_size is set very small (less than 10) and SQLite comes
  under memory pressure and if a multi-statement transaction is started
  in which the last statement prior to COMMIT is a SELECT statement and if
  a checkpoint occurs right after the transaction commit, then
  it might happen that the transaction will be silently rolled back instead
  of being committed.

  The default setting for cache_size is 2000. So in most situations, this
  bug will never appear.  But sometimes programmers set cache_size to
  very small values on gadgets and other low-memory devices in order to
  save memory space.  Such applications are vulnerable.
  Note that this bug does not cause database corruption.  It is
  as if ROLLBACK were being run instead of COMMIT in some cases.

  Bug Details

  Transactions commit in WAL mode by adding a record onto the end of
  the WAL (the write-ahead log) that contains a "commit" flag.  So to
  commit a transaction, SQLite takes all the pages that have changed
  during that transaction, appends them to the WAL, and sets the commit
  flag on the last page.  Now, if SQLite comes under memory pressure, it
  might try to free up memory space by writing changed pages to the WAL
  prior to the commit.  We call this "spilling" the cache to WAL.  There 
  is nothing wrong with spilling cache to WAL.  But if the
  memory pressure is severe, it might be that by the time COMMIT is run,
  all changed pages for the transaction have already been spilled to WAL
  and there are no pages left to be written to WAL.
  And with no unwritten pages, there was nothing to put the commit flag
  on.  And without a commit flag, the transaction would end up being
  rolled back.

  The fix to this problem was that if all changed pages has already
  been written to the WAL when the commit was started, then page 1 of
  the database will be written to the WAL again, so that there will always
  be a page available on which to set the commit flag.
2011-04-17 - Version 3.7.6.2
  SQLite version 3.7.6.2 adds a one-line bug fix to 3.7.6.1 that enables
  pthreads to work correctly on NetBSD.  The problem was a faulty function
  signature for the open system call.  The problem does not appear to have
  any adverse impact on any system other than NetBSD.

  Upgrading from version 3.7.6.1 is only needed on NetBSD.
2011-04-13 - Version 3.7.6.1
  SQLite version 3.7.6.1 fixes a single bug in 3.7.6 that can cause a
  segfault if SQLITE_FCNTL_SIZE_HINT is used on a unix build that has
  SQLITE_ENABLE_LOCKING_MODE set to 0 and is compiled with
  HAVE_POSIX_FALLOCATE.

  Upgrading from 3.7.6 is only needed for users effected by the 
  configuration-specific bug described above.  There are no other changes
  to the code.
2011-04-12 - Version 3.7.6
  SQLite version 3.7.6 is a regularly scheduled bi-monthly maintenance
  release of SQLite.  Upgrading from version 3.7.5 is optional.  Upgrading
  releases prior to 3.7.5 is recommended.
2011-02-01 - Version 3.7.5
  SQLite version 3.7.5 is a regularly scheduled bi-monthly maintenance
  release of SQLite.  Due to the discovery and fix of 
  an obscure bug
  that could cause database corruption, upgrading from all prior
  releases of SQLite is recommended.  This bug was found during code
  review and has not been observed in the wild.

  This release adds new opcodes for the
  sqlite3_db_status() interface that allow more precise measurement of
  how the lookaside memory allocator is performing, which can be useful
  for tuning in applications with very tight memory constraints.

  The sqlite3_vsnprintf() interface was added.  This routine is simply
  a varargs version of the long-standing sqlite3_snprintf() interface.

  The output from sqlite3_trace() interface has been enhanced to work
  better (and faster) in systems that use recursive extensions such as
  FTS3 or RTREE.

  Testing with Valgrind shows that this release of SQLite is about 1%
  or 2% faster than the previous release for most operations.

  A fork of the popular ADO.NET adaptor for SQLite known as System.Data.SQLite
  is now available on http://System.Data.SQLite.org/.  The originator
  of System.Data.SQLite, Robert Simpson, is aware of this fork, has
  expressed his approval, and has commit privileges on the new Fossil
  repository.  The SQLite development team intends to maintain
  System.Data.SQLite moving forward.
2010-12-08 - Version 3.7.4
  SQLite version 3.7.4 is a regularly scheduled bi-monthly maintenance
  release of SQLite.  Upgrading from version 3.7.2 and version 3.7.3
  is optional.  Upgrading from all other SQLite releases is recommended.

  This release features full-text search enhancements.  The older 
  FTS3 virtual table is still fully supported, and should also run
  faster.  In addition, the new FTS4 virtual table is added.  FTS4
  follows the same syntax as FTS3 but holds additional metadata which
  facilitates some performance improvements and more advanced 
  matchinfo() output.  Look for further full-text search enhancements
  in subsequent releases.

  Also in this release, the EXPLAIN QUERY PLAN output has been enhanced
  and new documentation is provided so that application developers can
  more easily understand how SQLite is performing their queries.

  Thanks to an account from the folks at http://www.devio.us/, OpenBSD
  has been added to the list of platforms upon which we 
  test SQLite prior to every release.  That list of platforms
  now includes:

  
   Linux x86 &amp; x86_64
   MacOS 10.5 &amp; 10.6
   MacOS 10.2 PowerPC
   WinXP and Win7
   Android 2.2
   OpenBSD 4.7
  

  The previous release of SQLite (version 3.7.3) has proven to be very
  robust.  The only serious issue discovered was
  ticket 80ba201079 that
  describes an incorrect query result that can occur under very 
  unusual circumstances.  The ticket description contains details of the
  problem.  Suffice it to say here that the problem is very obscure and
  is unlikely to effect most applications and so upgrading is optional.
  The problem is fixed, of course, in this release.
2010-October-08 - Version 3.7.3
  SQLite version 3.7.3 is a regularly scheduled bi-monthly maintenance
  release of SQLite.  Upgrading from version 3.7.2 is optional.
  Upgrading from all other releases is recommended.

  This release adds two new interfaces (really just variations on existing
  interfaces).  The sqlite3_create_function_v2() interface adds a 
  destructor for the application-data pointer.  The new 
  sqlite3_soft_heap_limit64() interface allows the soft heap limit to
  be set to a value greater than 231. 

  The RTREE extension has been enhanced with the ability to have an
  application-defined query region.  This might
  be used, for example, to locate all objects within 
  the field of view of a camera.

  The 3.7.3 release also includes some performance enhancements, including
  query planner improvements, documentation updates,
  and fixes to some very obscure bugs.
2010-August-24 - Version 3.7.2
  SQLite version 3.7.2 fixes a long-standing bug that can cause the
  database free-page list to go corrupt if incremental_vacuum is used
  multiple times to
  partially reduce the size of a database file that contains many hundreds
  of unused database pages.  The original bug reports together with links
  to the patch that fixes it can be seen 
  here.

  This bug has been in the code for at least a year and possibly longer.
  The bug has nothing to do with the versions 3.7.1 or 3.7.0 or any other
  recent release.  The fact that the bug was discovered (and fixed)
  within hours of the 3.7.1 release is purely a coincidence.

  The bug is impossible to hit without using incremental_vacuum and is
  very difficult to hit even with incremental_vacuum.  And the kind of
  corruption that the bug causes can usually be fixed 
  simply by running VACUUM.  Nevertheless, because the bug can result
  in database corruption, it is recommended that all SQLite users upgrade
  to version 3.7.2 or later.
2010-August-23 - Version 3.7.1
  SQLite version 3.7.1 is a stabilization release for the 3.7.x series.
  Other than the filesize-in-header bug that was fixed in version 3.7.0.1,
  no major problems have been seen in 3.7.0.  Some minor corner-case
  performance regressions have been fixed.  A typo in the OS/2 interface
  has been repaired.

  A biggest part of the 3.7.1 release is a cleanup and refactoring of
  the pager module within SQLite.  This refactoring should have no
  application-visible effects.  The purpose was to reorganize the code
  in ways that make it easier to prove correctness.

  The 3.7.1 release adds new experimental methods for obtained more 
  detailed memory usage information and for controlling database file
  fragmentation.  And the query planner now does a better job of
  optimizing the LIKE and GLOB operators.

  This release increases the maximum size of database pages from 32KiB to
  64KiB.  A database with 64KiB pages will not be readable or writable by
  older versions of SQLite.  Note that further increases in page size
  are not feasible since the file format uses 16-bit offsets to structures
  within each page.
2010-August-04 - Version 3.7.0.1
  SQLite version 3.7.0.1 is a patch release to fix a bug in the new
  filesize-in-header feature of the SQLite file format
  that could cause database corruption if the same database file is
  written alternately with version 3.7.0 and version 3.6.23.1 or earlier.
  A performance regression was also fixed in this release.
2010-07-22 - Version 3.7.0
  SQLite version 3.7.0 is a major release of SQLite that features
  a new transaction control mechanism using a write-ahead log or WAL.
  The traditional rollback-journal is still used as the default so there
  should be no visible change for legacy programs.  But newer programs
  can take advantage of improved performance and concurrency by enabling
  the WAL journaling mode.

  SQLite version 3.7.0 also contains some query planner enhancements and
  a few obscure bug fixes, but the only really big change is the addition
  of WAL mode.
2010-03-30 - Version 3.6.23.1
  SQLite version 3.6.23.1 is a patch release to fix a bug in the
  offsets() function of FTS3 at the request of the Mozilla.  
2010-03-09 - Version 3.6.23
  SQLite version 3.6.23 is a regular bimonthly release of SQLite.
  Upgrading from the prior release is purely optional.

  This release contains new pragmas: the secure_delete pragma, and
  the compile_options pragma.
  There are a new SQL functions: sqlite_compileoption_used()
  and sqlite_compileoption_get().
  New C/C++ interfaces: sqlite3_compileoption_used(),
  sqlite3_compileoption_get(), SQLITE_CONFIG_LOG, and
  sqlite3_log().

  This release also includes several minor bug fixes and performance
  improvements.  Support for SQLITE_OMIT_FLOATING_POINT is enhanced.
  There are on-going improvements to FTS3.

  The ".genfkey" command in the Command Line Interface has been
  removed.  SQLite has supported standard SQL foreign key constraints
  since version 3.6.19 and so the ".genfkey" command was seen as
  an anachronism.
2010-01-06 - Version 3.6.22
  SQLite version 3.6.22 is a bug-fix release.  Two bugs have been fixed
  that might cause incorrect query results.  
  
  Ticket 31338dca7e
  describes a
  problem with queries that have a WHERE clause of the form (x AND y) OR z
  where x and z come from one table of a join and y comes from a different
  table.
   Ticket eb5548a849
  describes
  a problem where the use of the CAST operator in the WHERE clause can lead
  to incorrect results if the column being cast to a new datatype is also
  used in the same WHERE clause without being cast.
  
  Both bugs are obscure,
  but because they could arise in an application after deployment, it is
  recommended that all applications upgrade SQLite to version 3.6.22.

  This release also includes other minor bug fixes and performance
  enhancements, especially in the FTS3 extension.
2009-12-07 - Version 3.6.21
  SQLite version 3.6.21 focuses on performance optimization.  For
  a certain set of traces, this version uses 12% fewer CPU instructions
  than the previous release (as measured by Valgrind).  In addition, the
  FTS3 extension has been through an extensive cleanup and rework and
  the sqlite3_trace() interface has been modified to insert 
  bound parameter values into its output.
2009-11-04 - Version 3.6.20
  SQLite version 3.6.20 is a general maintenance release.  The
  query planner has been enhanced to work better with bound parameters
  in LIKE and GLOB operators and in range constraints and various minor
  bugs have been fixed.  Upgrading from 3.6.19 is optional.
2009-10-14 - Version 3.6.19
  SQLite version 3.6.19 adds native support for
  foreign key constraints, including deferred constraints and
  cascading deletes.  Enforcement of foreign keys is disabled by
  default for backwards compatibility and must be turned on using
  the foreign_keys pragma.

  Version 3.6.19 also adds support for the
  IS and IS NOT operators.  Formerly, SQLite (as most
  other SQL database engines) supported IS NULL and IS NOT NULL.  The
  IS and IS NOT operators are generalizations that allow the right-hand
  side to be an arbitrary expression.  IS and IS NOT work the same as
  == (equals) and != (not equals) except that with IS and IS NOT the
  NULL values compare equal to one another.
2009-09-11 - Version 3.6.18
  Beginning with this release, the SQLite source code is tracked and
  managed using the Fossil
  distributed configuration management system.  SQLite was previously
  versioned using CVS.  The entire CVS history has been imported into
  Fossil.  The older CVS repository remains on the website but is
  read-only.

  There are two major enhancements in SQLite version 3.6.18.  The first
  is a series or refinements to the query planner that help SQLite to
  choose better plans for joins where in the past it was selecting suboptimal
  query plans.  The SQLITE_ENABLE_STAT2 compile-time option has been
  added to cause SQLite to collect histogram data on indices when the
  ANALYZE command is run.  The use of histograms improve the query
  planning performance even more.

  The second major enhancement is that SQLite now support recursive
  triggers.  The older non-recursive behavior of triggers is still the
  default behavior.  Recursive triggers are activated using the
  recursive_triggers pragma.  In addition to allowing triggers to
  call themselves (either directly or indirectly) the new capability
  also fires DELETE triggers on rows that are removed from a table
  as a result of REPLACE conflict resolution processing.

  Non-recursive triggers are still the default behavior since this
  is least likely to cause problems for existing applications.  However,
  we anticipate that triggers will become
  recursive by default beginning with release 3.7.0.  At that point,
  applications that want to continue using the older non-recursive
  trigger behavior will need to use the recursive_triggers pragma
  to disable recursive triggers.

  This version of SQLite also contains bug fixes, though none of the
  bugs are serious and all are obscure, so upgrading is optional.

  The SQLite core continues to have 100% branch test coverage
  and so despite the many changes in this release, the developers 
  believe that this version of SQLite is stable and ready for production
  use.
2009-08-10 - Version 3.6.17
  This is a monthly maintenance release with a focus of bug fixes,
  performance improvements, and increased test coverage.  This is the
  first release of SQLite since 100% branch test coverage
  was achieved on the SQLite core.  

  In addition, a new interface sqlite3_strnicmp() is provided for the
  convenience of extension writers.

  None of the bugs fixed in this release are serious.  All bugs are
  obscure.  Upgrading is optional.
2009-07-25 - 100% Branch Test Coverage
  A subset of the TH3 test suite was measured by gcov to provide
  100% branch test coverage over the SQLite core
  (exclusive of the VFS backend and of extensions such as FTS3 and RTREE)
  when compiled for SuSE 10.1 Linux on x86.  The SQLite developers pledge
  to maintain branch test coverage at 100% in all future releases.
  Ongoing work will strive for 100% branch test coverage on the 
  operating-system backends and extensions as well.
2009-06-27 - Version 3.6.16
  SQLite version 3.6.16 is another general maintenance release containing
  performance and robustness enhancements.  A single notable bug was fixed
  (ticket #3929).  This bug cause cause INSERT or UPDATE statements to fail
  on indexed tables that have AFTER triggers that modify the same table and
  index.
2009-06-15 - Version 3.6.15
  SQLite version 3.6.15 is a general maintenance release containing
  performance and robustness enhancements and fixes for various obscure
  bugs.  
2009-05-25 - Version 3.6.14.2
  SQLite version 3.6.14.2 fixes an obscure bug in the code generator
  (ticket #3879)
  section of SQLite which can potentially cause incorrect query results.
  The changes from the prior release consist of only this one bug fix,
  check-in [6676]
  and a change to the version number text.

  The bug was introduced in version 3.6.14.  It is recommended that
  users of version 3.6.14 and 3.6.14.1 upgrade to this release.  Applications
  are unlikely to hit this bug, but since it is difficult to predict which
  applications might hit it and which might not, we recommend that all
  users of 3.6.14 and 3.5.14.1 upgrade to this release.
2009-05-19 - Version 3.6.14.1
  SQLite version 3.6.14.1 is a patch release to version 3.6.14 with
  minimal changes that fixes three bugs.  Upgrading is only necessary 
  for users who are impacted by one or more of those bugs.
2009-05-07 - Version 3.6.14
  SQLite version 3.6.14 provides new performance enhancements in
  the btree and pager layers and in the query optimizer.  Certain
  workloads can be as much as twice as fast as the previous release,
  though 10% faster is a more typical result.

  Queries against virtual tables that contain OR and IN operators
  in the WHERE clause are now able to use indexing.

  A new optional asynchronous I/O backend is available for
  unix and windows.  The asynchronous backend gives the illusion of faster
  response time by pushing slow write operations into a background thread.
  The tradeoff for faster response time is that more memory is required
  (to hold the content of the pending writes) and if a power failure or
  program crash occurs, some transactions that appeared to have committed
  might end up being rolled back upon restart.

  This release also contains many minor bug fixes, documentation enhancements,
  new test cases, and cleanups and simplifications to the source code.

  There is no compelling reason to upgrade from versions 3.6.12 or
  3.6.13 if those prior versions are working.  Though many users may
  benefit from the improved performance.
2008-12-16 - Version 3.6.7
  SQLite version 3.6.7 contains a major cleanup of the Unix driver,
  and support for the new Proxy Locking mechanism on Mac OS X.  Though
  the Unix driver is reorganized, its functionality is the same and so
  applications should not notice a difference.
2008-11-26 - Version 3.6.6.2
  This release fixes a bug that was introduced into SQLite version 3.6.6
  and which seems like it might be able to cause database corruption.  This
  bug was detected during stress testing.  It has not been seen in the wild.
  An analysis of the problem suggests that the bug might be able to cause
  database corruption, however focused efforts to find a real-world test
  cases that actually causes database corruption have so far been unsuccessful.
  Hence, the likelihood of this bug causing problems is low.  Nevertheless,
  we have decided to do an emergency branch release out of an abundance of
  caution.

  The version 3.6.6.2 release also fixes an obscure memory leak that
  can occur following a disk I/O error.
2008-11-22 - Version 3.6.6.1
  This release fixes a bug that was introduced into SQLite version 3.6.4
  and that can cause database corruption in obscure cases.  This bug has
  never been seen in the wild; it was first detected by internal stress
  tests and required substantial analysis before it could be shown to 
  potentially lead to corruption.  So we feel that SQLite versions 3.6.4,
  3.6.5, and 3.6.6 are safe to use for development work.  But upgrading
  to this patch release or later is recommended prior to deploying
  products that incorporate SQLite.

  We have taken the unusual step of issuing a patch release in order to
  get the fix for this bug into circulation quickly.  SQLite version 3.6.7
  will continue on its normal path of development with an anticipated
  release in mid December.  
2008-11-19 - Version 3.6.6
  SQLite version 3.6.5 is released.  This is a quick turn-around release
  that fixes a bug in virtual tables and FTS3 that snuck into
  version 3.6.5.  This release also adds the new
  application-defined page cache mechanism.
2008-11-12 - Version 3.6.5
  SQLite version 3.6.5 is released.  There are various minor feature
  enhancements and numerous obscure bug fixes. 
  The change log contains the details.  Upgrading is
  optional.
2008-11-01 - Bloomberg Joins SQLite Consortium
  The SQLite developers are honored to announce that
  Bloomberg has joined the
  SQLite Consortium.
2008-10-15 - Version 3.6.4
  SQLite version 3.6.4 adds new features designed to help applications
  detect when indices are not being used on query.  There are also some
  important performance improvements.  Upgrading is optional.
2008-09-22 - Version 3.6.3
  SQLite version 3.6.3 fixes a bug in SELECT DISTINCT that was introduced
  by the previous version.  No new features are added.  Upgrading is
  recommended for all applications that make use of DISTINCT.
2008-08-30 - Version 3.6.2
  SQLite version 3.6.2 contains rewrites of the page-cache subsystem and
  the procedures for matching identifiers to table columns in SQL statements.
  These changes are designed to better modularize the code and make it more
  maintainable and reliable moving forward.  Nearly 5000 non-comment lines 
  of core code (about 11.3%) have changed
  from the previous release.  Nevertheless, there should be no
  application-visible changes, other than bug fixes.
2008-08-06 - Version 3.6.1
  SQLite version 3.6.1 is a stabilization and performance enhancement
  release.  
2008-07-16 - Version 3.6.0 beta
  Version 3.6.0 makes changes to the VFS object in order
  to make SQLite more easily portable to a wider variety of platforms.
  There are potential incompatibilities with some legacy applications.
  See the 35to36.html document for details.

  Many new interfaces are introduced in version 3.6.0.  The code is
  very well tested and is appropriate for use in stable systems.  We
  have attached the "beta" designation only so that we can make tweaks to
  the new interfaces in the next release without having to declare an
  incompatibility.
2008-05-12 - Version 3.5.9
  Version 3.5.9 adds a new experimental PRAGMA: journal_mode.
  Setting the journal mode to PERSIST can provide performance improvement
  on systems where deleting a file is expensive.  The PERSIST journal
  mode is still considered experimental and should be used with caution
  pending further testing.

  Version 3.5.9 is intended to be the last stable release prior to
  version 3.6.0.  Version 3.6.0 will make incompatible changes to the
  sqlite3_vfs VFS layer in order to address deficiencies in the original
  design.  These incompatibilities will only effect programmers who
  write their own custom VFS layers (typically embedded device builders).
  The planned VFS changes will be much smaller
  than the changes that occurred on the 
  3.4.2 to 3.5.0 transaction that occurred last
  September.

  This release of SQLite is considered stable and ready for production use.
2008-04-16 - Version 3.5.8
  Version 3.5.8 includes some important new performance optimizations
  in the virtual machine code generator, including constant subexpression
  factoring and common subexpression elimination.  This release also
  creates new public interfaces:
  sqlite3_randomness() provides access to SQLite's internal
  pseudo-random number generator, sqlite3_limit() allows size
  limits to be set at run-time on a per-connection basis, and
  sqlite3_context_db_handle() is a convenience routine that allows
  an application-defined SQL function implementation to retrieve
  its database connection handle.

  This release of SQLite is considered stable and ready for production use.
2008-03-17 - Version 3.5.7
  Version 3.5.7 fixes several minor and obscure bugs, especially 
  in the autoconf-generated makefile.  Upgrading is optional.
  This release of SQLite is considered stable and ready for production use.
2008-02-06 - Version 3.5.6
  Version 3.5.6 fixes a minor regression in 3.5.5 - a regression that
  had nothing to do with the massive change of the virtual machine
  to a register-based design.
  No problems have been reported with the new virtual machine.  This
  release of SQLite is considered stable and ready for production use.
2008-01-31 - Version 3.5.5
  Version 3.5.5 changes over 8% of the core source code of SQLite in order
  to convert the internal virtual machine from a stack-based design into
  a register-based design.  This change will allow future optimizations
  and will avoid an entire class of stack overflow bugs that have caused
  problems in the past.  Even though this change is large, extensive testing
  has found zero errors in the new virtual machine and so we believe this
  to be a very stable release.
2007-12-14 - Version 3.5.4
  Version 3.5.4 fixes a long-standing but obscure bug in UPDATE and
  DELETE which might cause database corruption.  (See ticket #2832.)
  Upgrading is recommended for all users.

  This release also brings the processing of ORDER BY statements into
  compliance with standard SQL.  This could, in theory, cause problems
  for existing applications that depend on the older, buggy behavior.
  See ticket #2822 for additional information.
2007-12-12 - SQLite Consortium Announced
  The SQLite Consortium was launched
  today with Mozilla and
  Symbian as charter members.
  As noted in the press release,
  the Consortium's goal is to promote the continuing vitality and
  independence of SQLite.  
2007-11-27 - Version 3.5.3
  This is an incremental release that fixes several minor problems.
  Upgrading is optional.  If Version 3.5.2 or 3.5.1 is working fine
  for you, then there is no pressing need to change to 3.5.3.

  The prebuilt binaries and the amalgamation found on the 
  download page include the FTS3 fulltext
  search extension module.  We are doing this on an experimental
  basis and are not promising to provide prebuilt binaries with
  FTS3 in the future.
2007-11-05 - Version 3.5.2
  This is an incremental release that fixes several minor problems,
  adds some obscure features, and provides some performance tweaks.  
  Upgrading is optional.

  The experimental compile-time option
  SQLITE_OMIT_MEMORY_ALLOCATION is no longer supported.  On the other
  hand, it is now possible to compile SQLite so that it uses a static
  array for all its dynamic memory allocation needs and never calls
  malloc.  Expect to see additional radical changes to the memory 
  allocation subsystem in future releases.
2007-10-04 - Version 3.5.1
  Fix a long-standing bug that might cause database corruption if a
  disk-full error occurs in the middle of a transaction and that
  transaction is not rolled back.
  Ticket #2686.

  The new VFS layer is stable.  However, we still reserve the right to
  make tweaks to the interface definition of the VFS if necessary.
2007-09-04 - Version 3.5.0 alpha
  The OS interface layer and the memory allocation subsystems in
  SQLite have been reimplemented.  The published API is largely unchanged
  but the (unpublished) OS interface has been modified extensively.  
  Applications that implement their own OS interface will require
  modification.  See
  34to35.html for details.

  This is a large change.  Approximately 10% of the source code was
  modified.  We are calling this first release "alpha" in order to give
  the user community time to test and evaluate the changes before we
  freeze the new design.
2007-08-13 - Version 3.4.2
  While stress-testing the 
  soft_heap_limit
  feature, a bug that could lead to
  database corruption was discovered and fixed.
  Though the consequences of this bug are severe, the chances of hitting 
  it in a typical application are remote.  Upgrading is recommended
  only if you use the 
  sqlite3_soft_heap_limit
  interface.
2007-07-20 - Version 3.4.1
  This release fixes a bug in VACUUM that
  can lead to database corruption.  The bug was introduced in version 
  3.3.14.
  Upgrading is recommended for all users.  Also included are a slew of
  other more routine
  enhancements and bug fixes.
2007-06-18 - Version 3.4.0
  This release fixes two separate bugs either of which 
  can lead to database corruption.  Upgrading
  is strongly recommended.  If you must continue using an older version
  of SQLite, please at least read about how to avoid these bugs
  at CorruptionFollowingBusyError and ticket #2418
  
  This release also adds explicit limits on the
  sizes and quantities of things SQLite will handle.  The new limits might
  causes compatibility problems for existing applications that
  use excessively large strings, BLOBs, tables, or SQL statements. 
  The new limits can be increased at compile-time to work around any problems
  that arise.  Nevertheless, the version number of this release is
  3.4.0 instead of 3.3.18 in order to call attention to the possible
  incompatibility.
  
  There are also new features, including
  incremental BLOB I/O and
  incremental vacuum.
  See the change log 
  for additional information.
2007-04-25 - Version 3.3.17
  This version fixes a bug in the forwards-compatibility logic of SQLite
  that was causing a database to become unreadable when it should have
  been read-only.  Upgrade from 3.3.16 only if you plan to deploy into
  a product that might need to be upgraded in the future.  For day to day
  use, it probably does not matter.
2007-04-18 - Version 3.3.16
  Performance improvements added in 3.3.14 but mistakenly turned off
  in 3.3.15 have been reinstated.  A bug has been fixed that prevented 
  VACUUM from running if a NULL value was in a UNIQUE column.
2007-04-09 - Version 3.3.15
  An annoying bug introduced in 3.3.14 has been fixed.  There are
  also many enhancements to the test suite.  
2007-04-02 - Version 3.3.14
  This version focuses on performance improvements.  If you recompile
  the amalgamation using GCC option -O3 (the precompiled binaries
  use -O2) you may see performance
  improvements of 35% or more over version 3.3.13 depending on your
  workload.  This version also
  adds support for 
  exclusive access mode.
2007-02-13 - Version 3.3.13
  This version fixes a subtle bug in the ORDER BY optimizer that can 
  occur when using joins.  There are also a few minor enhancements.
  Upgrading is recommended.
2007-01-27 - Version 3.3.12
  The first published build of the previous version used the wrong
  set of source files.  Consequently, many people downloaded a build
  that was labeled as "3.3.11" but was really 3.3.10.  Version 3.3.12
  is released to clear up the ambiguity.  A couple more bugs have
  also been fixed and 
  PRAGMA integrity_check has been enhanced.
2007-01-22 - Version 3.3.11
  Version 3.3.11 fixes for a few more problems in version 3.3.9 that
  version 3.3.10 failed to catch.  Upgrading is recommended.
2007-01-09 - Version 3.3.10
  Version 3.3.10 fixes several bugs that were introduced by the previous
  release.  Upgrading is recommended.
2007-01-04 - Version 3.3.9
  Version 3.3.9 fixes bugs that can lead to database corruption under
  obscure and difficult to reproduce circumstances.  See
  DatabaseCorruption in the
  wiki for details.
  This release also adds the new
  sqlite3_prepare_v2()
  API and includes important bug fixes in the command-line
  shell and enhancements to the query optimizer.  Upgrading is
  recommended.
2006-10-09 - Version 3.3.8
  Version 3.3.8 adds support for full-text search using the 
  FTS1 module.  There are also minor bug fixes.  Upgrade only if
  you want to try out the new full-text search capabilities or if
  you are having problems with 3.3.7.
2006-08-12 - Version 3.3.7
  Version 3.3.7 includes support for loadable extensions and virtual
  tables.  But both features are still considered "beta" and their
  APIs are subject to change in a future release.  This release is
  mostly to make available the minor bug fixes that have accumulated
  since 3.3.6.  Upgrading is not necessary.  Do so only if you encounter
  one of the obscure bugs that have been fixed or if you want to try
  out the new features.
2006-06-19 - New Book About SQLite
  The Definitive Guide to SQLite,
  a new book by
  Mike Owens
  is now available from Apress.
  The books covers the latest SQLite internals as well as
  the native C interface and bindings for PHP, Python,
  Perl, Ruby, Tcl, and Java.  Recommended.
2006-06-6 - Version 3.3.6
  Changes include improved tolerance for Windows virus scanners
  and faster :memory: databases.  There are also fixes for several
  obscure bugs.  Upgrade if you are having problems.
2006-04-5 - Version 3.3.5
  This release fixes many minor bugs and documentation typos and
  provides some minor new features and performance enhancements.
  Upgrade only if you are having problems or need one of the new features.
2006-02-11 - Version 3.3.4
  This release fixes several bugs, including a 
  blunder that might cause a deadlock on multithreaded systems.
  Anyone using SQLite in a multithreaded environment should probably upgrade.
2006-01-31 - Version 3.3.3 stable
  There have been no major problems discovered in version 3.3.2, so
  we hereby declare the new APIs and language features to be stable
  and supported.
2006-01-24 - Version 3.3.2 beta
  More bug fixes and performance improvements as we move closer to
  a production-ready version 3.3.x.
2006-01-16 - Version 3.3.1 alpha
  Many bugs found in last week's alpha release have now been fixed and
  the library is running much faster again.

  Database connections can now be moved between threads as long as the
  connection holds no locks at the time it is moved.  Thus the common
  paradigm of maintaining a pool of database connections and handing
  them off to transient worker threads is now supported.
  Please help test this new feature.
  See the MultiThreading wiki page for additional
  information.
2006-01-10 - Version 3.3.0 alpha
  Version 3.3.0 adds support for CHECK constraints, DESC indices,
  separate REAL and INTEGER column affinities, a new OS interface layer
  design, and many other changes.  The code passed a regression
  test but should still be considered alpha.  Please report any
  problems.

  The file format for version 3.3.0 has changed slightly to support
  descending indices and
  a more efficient encoding of boolean values.  SQLite 3.3.0 will read and
  write legacy databases created with any prior version of SQLite 3.  But
  databases created by version 3.3.0 will not be readable or writable
  by earlier versions of the SQLite.  The older file format can be
  specified at compile-time for those rare cases where it is needed.
2005-12-19 - Versions 3.2.8 and 2.8.17
  These versions contain one-line changes to 3.2.7 and 2.8.16 to fix a bug
  that has been present since March of 2002 and version 2.4.0.
  That bug might possibly cause database corruption if a large INSERT or
  UPDATE statement within a multi-statement transaction fails due to a
  uniqueness constraint but the containing transaction commits.
2005-09-24 - Version 3.2.7
  This version fixes several minor and obscure bugs.
  Upgrade only if you are having problems.
2005-09-16 - Version 3.2.6 - Critical Bug Fix
  This version fixes a bug that can result in database
  corruption if a VACUUM of a 1 gigabyte or larger database fails
  (perhaps do to running out of disk space or an unexpected power loss)
  and is later rolled back.
  
  Also in this release:
  The ORDER BY and GROUP BY processing was rewritten to use less memory.
  Support for COUNT(DISTINCT) was added.  The LIKE operator can now be
  used by the optimizer on columns with COLLATE NOCASE.
2005-08-27 - Version 3.2.5
  This release fixes a few more lingering bugs in the new code.
  We expect that this release will be stable and ready for production use.
2005-08-24 - Version 3.2.4
  This release fixes a bug in the new optimizer that can lead to segfaults
  when parsing very complex WHERE clauses.
2005-08-21 - Version 3.2.3
  This release adds the ANALYZE command,
  the CAST operator, and many
  very substantial improvements to the query optimizer.  See the
  change log for additional
  information.
2005-08-02 - 2005 Open Source Award for SQLite
  SQLite and its primary author D. Richard Hipp have been honored with
  a 2005 Open Source Award
  from Google and O'Reilly.
2005-06-13 - Version 3.2.2
  This release includes numerous minor bug fixes, speed improvements,
  and code size reductions.  There is no reason to upgrade unless you
  are having problems or unless you just want to.
2005-03-29 - Version 3.2.1
  This release fixes a memory allocation problem in the new
  ALTER TABLE ADD COLUMN
  command.
2005-03-21 - Version 3.2.0
  The primary purpose for version 3.2.0 is to add support for
  ALTER TABLE ADD COLUMN.
  The new ADD COLUMN capability is made
  possible by AOL developers supporting and embracing great
  open-source software.  Thanks, AOL!

  Version 3.2.0 also fixes an obscure but serious bug that was discovered
  just prior to release.  If you have a multi-statement transaction and
  within that transaction an UPDATE or INSERT statement fails due to a
  constraint, then you try to rollback the whole transaction, the rollback
  might not work correctly.  See
  Ticket #1171 for details.  Upgrading is recommended for all users.
2005-03-16 - Version 3.1.6
  Version 3.1.6 fixes a critical bug that can cause database corruption
  when inserting rows into tables with around 125 columns. This bug was
  introduced in version 3.0.0.  See
  Ticket #1163 for additional information.
2005-03-11 - Versions 3.1.4 and 3.1.5 Released
  Version 3.1.4 fixes a critical bug that could cause database corruption
  if the autovacuum mode of version 3.1.0 is turned on (it is off by
  default) and a CREATE UNIQUE INDEX is executed within a transaction but
  fails because the indexed columns are not unique.  Anyone using the
  autovacuum feature and unique indices should upgrade.

  Version 3.1.5 adds the ability to disable
  the F_FULLFSYNC ioctl() in OS-X by setting "PRAGMA synchronous=on" instead
  of the default "PRAGMA synchronous=full".  There was an attempt to add
  this capability in 3.1.4 but it did not work due to a spelling error.
2005-02-19 - Version 3.1.3 Released
  Version 3.1.3 cleans up some minor issues discovered in version 3.1.2.
2005-02-15 - Versions 2.8.16 and 3.1.2 Released
  A critical bug in the VACUUM command that can lead to database
  corruption has been fixed in both the 2.x branch and the main
  3.x line.  This bug has existed in all prior versions of SQLite.
  Even though it is unlikely you will ever encounter this bug,
  it is suggested that all users upgrade.  See
  ticket #1116 for additional information.

  Version 3.1.2 is also the first stable release of the 3.1
  series.  SQLite 3.1 features added support for correlated
  subqueries, autovacuum, autoincrement, ALTER TABLE, and
  other enhancements.  See the 
  release notes
  for version 3.1.0 for a detailed description of the
  changes available in the 3.1 series.
2005-02-01 - Version 3.1.1 (beta) Released
  Version 3.1.1 (beta) is now available on the
  website.  Version 3.1.1 is fully backwards compatible with the 3.0 series
  and features many new features including Autovacuum and correlated
  subqueries.  The
  release notes
  From version 3.1.0 apply equally to this release beta.  A stable release
  is expected within a couple of weeks.
2005-01-21 - Version 3.1.0 (alpha) Released
  Version 3.1.0 (alpha) is now available on the
  website.  Version 3.1.0 is fully backwards compatible with the 3.0 series
  and features many new features including Autovacuum and correlated
  subqueries.  See the
  release notes
  for details.

  This is an alpha release.  A beta release is expected in about a week
  with the first stable release to follow after two more weeks.
2004-11-09 - SQLite at the 2004 International PHP Conference
  There was a talk on the architecture of SQLite and how to optimize
  SQLite queries at the 2004 International PHP Conference in Frankfurt,
  Germany.
  
  Slides from that talk are available.
2004-10-11 - Version 3.0.8
  Version 3.0.8 of SQLite contains several code optimizations and minor
  bug fixes and adds support for DEFERRED, IMMEDIATE, and EXCLUSIVE
  transactions.  This is an incremental release.  There is no reason
  to upgrade from version 3.0.7 if that version is working for you.
2004-10-10 - SQLite at the 11th
Annual Tcl/Tk Conference
  There will be a talk on the use of SQLite in Tcl/Tk at the
  11th Tcl/Tk Conference this week in
  New Orleans.  Visit http://www.tcl-lang.org/community/tcl2004/
  for details.
  
  Slides from the talk are available.
2004-09-18 - Version 3.0.7
  Version 3.0 has now been in use by multiple projects for several
  months with no major difficulties.   We consider it stable and
  ready for production use. 
2004-09-02 - Version 3.0.6 (beta)
  Because of some important changes to sqlite3_step(),
  we have decided to
  do an additional beta release prior to the first "stable" release.
  If no serious problems are discovered in this version, we will
  release version 3.0 "stable" in about a week.
2004-08-29 - Version 3.0.5 (beta)
  The fourth beta release of SQLite version 3.0 is now available.
  The next release is expected to be called "stable".
2004-08-08 - Version 3.0.4 (beta)
  The third beta release of SQLite version 3.0 is now available.
  This new beta fixes several bugs including a database corruption
  problem that can occur when doing a DELETE while a SELECT is pending.
  Expect at least one more beta before version 3.0 goes final.
2004-07-22 - Version 3.0.3 (beta)
  The second beta release of SQLite version 3.0 is now available.
  This new beta fixes many bugs and adds support for databases with
  varying page sizes.  The next 3.0 release will probably be called
  a final or stable release.

  Version 3.0 adds support for internationalization and a new
  more compact file format. 
  Details.
  The API and file format have been fixed since 3.0.2.  All
  regression tests pass (over 100000 tests) and the test suite
  exercises over 95% of the code.

  SQLite version 3.0 is made possible in part by AOL
  developers supporting and embracing great Open-Source Software.
2004-07-22 - Version 2.8.15
  SQLite version 2.8.15 is a maintenance release for the version 2.8
  series.  Version 2.8 continues to be maintained with bug fixes, but
  no new features will be added to version 2.8.  All the changes in
  this release are minor.  If you are not having problems, there is
  there is no reason to upgrade.
2004-06-30 - Version 3.0.2 (beta) Released
  The first beta release of SQLite version 3.0 is now available.
  Version 3.0 adds support for internationalization and a new
  more compact file format. 
  Details.
  As of this release, the API and file format are frozen.  All
  regression tests pass (over 100000 tests) and the test suite
  exercises over 95% of the code.

  SQLite version 3.0 is made possible in part by AOL
  developers supporting and embracing great Open-Source Software.
2004-06-25 - Website hacked
  The www.sqlite.org website was hacked sometime around 2004-06-22
  because the lead SQLite developer failed to properly patch CVS.
  Evidence suggests that the attacker was unable to elevate privileges
  above user "cvs".  Nevertheless, as a precaution the entire website
  has been reconstructed from scratch on a fresh machine.  All services
  should be back to normal as of 2004-06-28.
2004-06-18 - Version 3.0.0 (alpha) Released
  The first alpha release of SQLite version 3.0 is available for
  public review and comment.  Version 3.0 enhances internationalization support
  through the use of UTF-16 and user-defined text collating sequences.
  BLOBs can now be stored directly, without encoding.
  A new file format results in databases that are 25% smaller (depending
  on content).  The code is also a little faster.  In spite of the many
  new features, the library footprint is still less than 240KB
  (x86, gcc -O1).
  Additional information.

  Our intent is to freeze the file format and API on 2004-07-01.
  Users are encouraged to review and evaluate this alpha release carefully 
  and submit any feedback prior to that date.

  The 2.8 series of SQLite will continue to be supported with bug
  fixes for the foreseeable future.
2004-06-09 - Version 2.8.14 Released
  SQLite version 2.8.14 is a patch release to the stable 2.8 series.
  There is no reason to upgrade if 2.8.13 is working ok for you.
  This is only a bug-fix release.  Most development effort is
  going into version 3.0.0 which is due out soon.
2004-05-31 - CVS Access Temporarily Disabled
  Anonymous access to the CVS repository will be suspended
  for 2 weeks beginning on 2004-06-04.  Everyone will still
  be able to download
  prepackaged source bundles, create or modify trouble tickets, or view
  change logs during the CVS service interruption. Full open access to the
  CVS repository will be restored on 2004-06-18.
2004-04-23 - Work Begins On SQLite Version 3
  Work has begun on version 3 of SQLite.  Version 3 is a major
  changes to both the C-language API and the underlying file format
  that will enable SQLite to better support internationalization.
  The first beta is schedule for release on 2004-07-01.

  Plans are to continue to support SQLite version 2.8 with
  bug fixes.  But all new development will occur in version 3.0.


c2: WAL-mode File Format
c5: walformat.html
c4:
This document describes low-level details on how WAL mode is
implemented on unix and windows.

The separate file format description provides details on the
structure of a database file and of the write-head log file used in
WAL mode.  But details of the locking protocol and of the format
of the WAL-index are deliberately omitted since those details
are left to discretion of individual VFS implementations.  This
document fills in those missing details for the unix and windows VFSes.

For completeness, some of the higher level formatting information
contains in the file format document and elsewhere is replicated here,
when it pertains to WAL mode processing.

1. Files On Disk

When in active use, the state of a WAL mode database is described
by three separate files:


 The main database file with an arbitrary name "X".
 The write-ahead log file, usually named "X-wal".
 The wal-index file, usually named "X-shm".


1.1. The Main Database File

The format of the main database file is as described in the
file format document.  The file format version numbers at offsets
18 and 19 into the main database must both be 2 to indicate that the
database is in WAL mode.  The main database may have an arbitrary
name allowed by the underlying filesystem.  No special file suffixes
are required, though ".db", ".sqlite", and ".sqlite3" seem to be
popular choices.

1.2. The Write-Ahead-Log or "-wal" File
The write-ahead log or "wal" file is a roll-forward journal
that records transactions that have been committed but not yet applied
to the main database.  Details on the format of the wal file are
describe in the WAL format subsection of the main file format
document.  The wal file is named by appending the four characters
"-wal" to the end of the name of the main database file.  Except
on 8+3 filesystems, such names are not allowed, and in that case
the file suffix is changed to ".WAL".  But as 8+3 filesystems are
increasingly rare, that exceptional case can usually be ignored.



1.3. The Wal-Index or "-shm" file
The wal-index file or "shm" file is not actually used as a file.
Rather, individual database clients mmap the shm file and use it
as shared memory for coordinating access to the database and as a cache
for quickly locating frame within the wal file.  The name
of the shm file is the main database file name with the four characters
"-shm" appended.  Or, for 8+3 filesystems, the shm file is the main
database file with the suffix changed to ".SHM".

The shm does not contain any database content and is not required
to recover the database following a crash.  For that reason, the first
client to connect to a quiescent database will normally truncate the
shm file if it exists.  Since the content of the shm file does not need
to be preserved across a crash, the shm file is never fsync()-ed to disk.
In fact, if there were a mechanism by which SQLite could tell the 
operating system to never persist the shm file to disk but always hold
it in cache memory, SQLite would use that mechanism to avoid any
unnecessary disk I/O associated with the shm file.  However, no such
mechanism exists in standard posix.

Because the shm is only used to coordinate access between concurrent
clients, the shm file is omitted if exclusive locking mode
is set, as an optimization.  When exclusive locking mode is set,
SQLite uses heap memory in place of the memory-mapped shm file.

1.4. File Lifecycles

When a WAL mode database is in active use, all three of the above
files usually exist.  Except, the Wal-Index file is omitted if
exclusive locking mode is set.

If the last client using the database shuts down cleanly by
calling sqlite3_close(), then a checkpoint is run automatically
in order to transfer all information from the wal file
over into the main database, and both the shm file
and the wal file are unlinked.  Thus, when the database is not in
use by any client, it is usually the case that only the main
database file exists on disk.
However, if the last client did not call sqlite3_close() before it
shut down, or if the last client to disconnect was a read-only client,
then the final cleanup operation does not occur and the
shm and wal files may still exist on disk even when the database is
not in use.

1.5. Variations

When PRAGMA locking_mode=EXCLUSIVE (exclusive locking mode) is set,
only a single client is allowed to have the database open at one time.  Since
only a single client can use the database, the shm file is omitted.
The single client uses a buffer in heap memory as a substitute for the
memory-mapped shm file.

If a read/write client invokes
sqlite3_file_control(SQLITE_FCNTL_PERSIST_WAL) prior to shutdown,
then at shutdown a checkpoint is still run, but the shm file and wal
file are not deleted.
This allows subsequent read-only clients to connect to and read the
database.



2. The WAL-Index File Format


The WAL-index or "shm" file is used to coordinate access to the database
by multiple clients, and as a cache to help clients quickly locate frames
within the wal file.


Because the shm file is not involved in recovery, the shm file does not
need to be machine byte-order independent.  Hence, numeric values in
the shm file are written in the native byte order of the host computer,
rather than being converted into a specific cross-platform byte order as
is done with the main database file and the wal file.


The shm file consists of one or more hash tables, where each hash table
is 32768 bytes in size.  Except, a 136-byte header is carved out of the
front of the very first hash table, so the first hash table is only
32632 bytes in size.  The total size of the shm file is always a multiple
of 32768.  In most cases, the total size of the shm file is exactly 32768
bytes.  The shm file only needs to grow beyond a single hash table if
when the wal file grows very large (more than 4079 frames).  Since the
default automatic checkpoint threshold is
1000, WAL files rare reach the 4079 threshold needed to make the shm file
grow.

2.1. The WAL-Index Header

The first 136 bytes of the shm file are a header.  The shm header has
three main divisions as follows:


WAL-Index Header Divisions

BytesDescription
0..47First copy of the WAL Index Information
48..95Second copy of the WAL Index Information
96..135Checkpoint Information and Locks



Individual fields of the shm header, except for the
salt values copied from the WAL header, are unsigned integers
in the native byte-order of the host machine.  The salt values
are exact copies from the WAL header and are in whatever byte
order is used by the WAL file.
The size of integers may be 8, 16, 32, or 64 bits.
A detailed breakout of the individual fields of the shm header
follows:


WAL-Index Header Details

BytesNameMeaning

0..3iVersion
The WAL-index format version number.  Always 3007000.


4..7&nbsp;
Unused padding space.  Must be zero.


8..11iChange
Unsigned integer counter, incremented with each transaction


12isInit
The "isInit" flag.  1 when the shm file has been initialized.


13bigEndCksum
True if the WAL file uses big-ending checksums.  0 if the WAL
    uses little-endian checksums.


14..15szPage
The database page size in bytes, or 1 if the page size is 65536.


16..19mxFrame
Number of valid and committed frames in the WAL file.


20..23nPage
Size of the database file in pages.


24..31aFrameCksum
Checksum of the last frame in the WAL file.


32..39aSalt
The two salt value copied from the WAL file header.
These values are in the byte-order of the WAL file, which
might be different from the native byte-order of the
machine.


40..47aCksum
A checksum over bytes 0 through 39 of this header.


48..95&nbsp;
A copy of bytes 0 through 47 of this header.


96..99nBackfill
Number of WAL frames that have already been backfilled into the database
    by prior checkpoints


100..119read-mark&#91;0..4&#93;
Five "read marks".  Each read mark is a 32-bit unsigned integer (4 bytes).


120..127&nbsp;
Unused space set aside for 8 file locks.


128..132nBackfillAttempted
Number of WAL frames that have attempted to be backfilled but which might
    not have been backfilled successfully.


132..136&nbsp;
Unused space reserved for further expansion.






2.1.1. The mxFrame field

The 32-bit unsigned integer at offset 16 (and repeated at offset 64)
is the number of valid frames in the WAL.  Because WAL frame are numbered
starting with 1, mxFrame is also the index of the last valid commit frame
in the WAL. A commit frame is a frame that has a non-zero "size of database"
value in bytes 4 through 7 of the frame header, and that indicates the end
of a transaction.

When mxFrame field is zero, it indicates that the WAL is empty and that
all content should be obtained directly from the database file.

When mxFrame is equal to nBackfill, that indicates that all content
in the WAL has been written back into the database.
In that case, all content can be read directly from the database.
Furthermore, the next writer is free to reset the WAL if no other
connections hold locks on WAL_READ_LOCK(N) for N>0.

The mxFrame value is always greater than or equal to both
nBackfill and nBackfillAttempted.




2.1.2. The nBackfill field

The 32-bit unsigned integer at offset 128 in the WAL-index header
is called the "nBackfill".
this field holds the number of frames in the WAL file which
have been copied back into the main database.

The nBackfill number is never greater than mxFrame.
When nBackfill equals mxFrame, that means that the WAL content
has been completely written back into the database and it is
ok to reset the WAL if there are no locks held on any of WAL_READ_LOCK(N)
for N>0.

The nBackfill can only be increased while holding the
WAL_CKPT_LOCK.  However, nBackfill is changed to zero
during a WAL reset, and this happens while holding
the WAL_WRITE_LOCK.



2.1.3. WAL Locks

Eight bytes of space are set aside in the header to support
file locking using the xShmLock() method in the sqlite3_io_methods
object.  These eight bytes are never read nor written by SQLite since
some VFSes (ex: Windows) might implement locks using mandatory file locks.

These are the eight locks supported:


WAL-Index Locks Controlled By xShmLock()

NameOffset
xShmLockFile

WAL_WRITE_LOCK
0
120



WAL_CKPT_LOCK
1
121



WAL_RECOVER_LOCK
2
122



WAL_READ_LOCK(0)
3
123



WAL_READ_LOCK(1)
4
124



WAL_READ_LOCK(2)
5
125



WAL_READ_LOCK(3)
6
126



WAL_READ_LOCK(4)
7
127






TBD:  More information about the header

2.2. WAL-Index Hash Tables

The hash tables in the shm file are designed to answer the
following question quickly:


FindFrame(P,M):
Given a page number P and a maximum WAL frame index M,
return the largest WAL frame index for page P that does not exceed M, 
or return NULL if there are no frames for page P that do not exceed M.



Let the datatypes "u8", "u16", and "u32" mean unsigned integers of
length 8, 16, and 32 bits, respectively.  Then, the first 32768-byte unit
of the shm file is organized as follows:


u8 aWalIndexHeader[136];
u32 aPgno[4062];
u16 aHash[8192];


The second and all subsequent 32768-byte units of the shm file are
like this:


u32 aPgno[4096];
u16 aHash[8192];


Collectively, the aPgno entries record the database page number stored
in all frames of the WAL file.  The aPgno[0] entry on the first hash table
records the database page number stored in the very first frame in the WAL 
file. The aPgno[i] entry from the first hash table is the database page number
for the i-th frame in the WAL file.  The aPgno[k] entry for the second
hash table is the database page number for the (k+4062)-th frame in the
WAL file.  The aPgno[k] entry for the n-th 32768-byte hash table in the
shm file (for n>1) holds the database page number stored in the
(k+4062+4096*(n-2))-th frame of the WAL file.

Here is a slightly different way to describe the aPgno values:
If you think of all aPgno values as a contiguous array, then
the database page number stored in the i-th frame of the WAL file
is stored in aPgno[i].  Of course, aPgno is not a contiguous array.
The first 4062 entries are on the first 32768-byte unit of the shm
file and subsequent values are in 4096 entry chunks in later units
of the shm file.

One way to compute FindFrame(P,M) would be to scan the aPgno
array starting with the M-th entry and working backwards towards
the beginning and return J where aPgno[J]==P.  Such an algorithm would
work, and it would be faster than searching the whole WAL file for
the latest frame with page number P.  But the search can be made
much faster still by using the aHash structure.

A database page number P is mapped into a hash value
using the following hash function:


h = (P * 383)%8192


This function maps every page number into an integer between 0
and 8191 inclusive.  The aHash field of each 32768-byte shm file unit
maps P values into indexes of the aPgno field of the same unit as
follows:


 Compute the hash value: h = P * 383
 Let X be the largest set of consecutive integers {h, h+1, h+2, ..., h+N}
     such that for every j in X, aPgno&#91;j%8192]!=0.  The X set will be empty
     if aPgno&#91;h%8192]==0.  The X set is easily computed by starting with
     the value h%8192, and adding h%8192 to X and incrementing h until
     encountering the first aPgno&#91;h%8192] entry that is zero.
 The set X contains the index in aPgno of every entry in the current
     32768-byte unit of the shm file that might possible be a solution
     to the FindFrame(P,M) function.  Each of these entries must be
     checked separately to ensure that the aPgno value is P and that the
     frame number does not exceed M.  The largest frame number that passes
     those two tests is the answer.


Each entry in the aPgno array has a single corresponding entry
in the aHash array.  There are more available slots in aHash than
there are in aPgno.  The unused slots in aHash are filled with zero.
And since there are guaranteed to be unused slots in aHash, that means
the loop that computes X is guaranteed to terminate.  The expected size
of X is less than 2.  The worst case is that X will be the same as the
number of entries in aPgno, in which case the algorithm runs at about
the same speed as a linear scan of aPgno.  But that worst case performance
is exceedingly rare.  Usually, the size of X will be small and the use
of the aHash array allows one to compute FindFrame(P,M) much faster.

Here is an alternative way of describing the hash look-up algorithm:
Start with h = (P * 383)%8192 and look at aHash[h] and subsequent entries,
wrapping around to zero when h reaches 8192, until finding an entry with
aHash[h]==0.  All aPgno entries having a page number of P will have an
index that is one of the aHash[h] values thusly computed.
But not all the computed aHash[h] values will
meet the matching criteria, so you must check them independently.  The
speed advantage comes about because normally this set of h values is
very small.

Note that each 32768-byte unit of the shm file has its own aHash and
aPgno arrays.  The aHash array for a single unit is only helpful in finding
aPgno entries in that same unit.  The overall FindFrame(P,M) function
needs to do hash lookups beginning with the latest unit and working
backwards to the oldest unit until it finds an answer.

2.3. Locking Matrix

Access is coordinated in WAL mode using both the legacy DELETE-mode
locks controlled by the xLock and xUnlock methods of the sqlite3_io_methods
object and the WAL locks controlled by the xShmLock method of the
sqlite3_io_methods object.

Conceptually, there is just a single DELETE-mode lock.  The DELETE-mode
lock for a single database connection can be in exactly one of the
following states:

 SQLITE_LOCK_NONE (unlocked)
 SQLITE_LOCK_SHARED (reading)
 SQLITE_LOCK_RESERVED (reading, waiting to write)
 SQLITE_LOCK_PENDING (new readers blocked, waiting to write)
 SQLITE_LOCK_EXCLUSIVE (writing)

The DELETE-mode locks are stored on the lock-byte page of the
main database file.
Only SQLITE_LOCK_SHARED and SQLITE_LOCK_EXCLUSIVE are factors for WAL-mode
databases.
The other locking states are used in rollback-mode, but not in WAL-mode.

The WAL-mode locks are described above.

2.3.1. How the various locks are used

The following rules show how each of the locks is used.


SQLITE_LOCK_SHARED

All connections hold SQLITE_LOCK_SHARED continuously while attached
to a WAL-mode database.  This is true for both read/write connections
and read-only connections.
The SQLITE_LOCK_SHARED lock is held even by connections that are
not within transaction.
This is different from rollback mode, where the SQLITE_LOCK_SHARED is
released at the end of each transaction.

SQLITE_LOCK_EXCLUSIVE

Connections hold an exclusive lock when change in between WAL mode
and any of the various rollback-modes.  Connections might also attempt to
obtain an EXCLUSIVE lock when they disconnect from WAL mode.  If
a connection is able to obtain an EXCLUSIVE lock, that means it is the
only connection to the database and so it may attempt to checkpoint
and then delete the WAL-index and WAL files.

When a connection is holding a SHARED lock on the main database,
that will prevent any other connection from acquiring the EXCLUSIVE
lock, which in turn prevents the WAL-index and WAL files from being
deleted out from under other users, and prevents a transition out of
WAL-mode while other users are accessing the database in WAL-mode.


WAL_WRITE_LOCK

The WAL_WRITE_LOCK is only locked exclusively.  There is never a shared
lock taken on WAL_WRITE_LOCK.

An EXCLUSIVE WAL_WRITE_LOCK is held by any connection that is appending
content to the end of the WAL.  Hence, only a single process at a time
can append content to the WAL.  If a WAL reset occurs as a consequence of
a write, then the nBackfill field of the WAL-index header is reset to
zero while holding this lock.
An EXCLUSIVE is also held WAL_WRITE_LOCK, and on several other locking
bytes, when a connection is running recovery on the shared WAL-index.

WAL_CKPT_LOCK

The WAL_CKPT_LOCK is only locked exclusively.  There is never a shared
lock taken on WAL_CKPT_LOCK.

An EXCLUSIVE WAL_CKPT_LOCK is held by any connection that is running
a checkpoint.  The nBackfill field of the WAL-index header may be
increased while holding this exclusive lock, but it may not be decreased.
An EXCLUSIVE is also held WAL_CKPT_LOCK, and on several other locking
bytes, when a connection is running recovery on the shared WAL-index.

WAL_RECOVER_LOCK

The WAL_RECOVER_LOCK is only locked exclusively.  There is never a shared
lock taken on WAL_RECOVER_LOCK.

An EXCLUSIVE WAL_RECOVER_LOCK is held by any connection that is running
recovery to reconstruct the shared WAL-index.

A read-only connection that is rebuilding its private heap-memory WAL-index
does not hold this lock.  (It cannot, since read-only connections are not
allowed to hold any exclusive locks.)  This lock is only held when rebuilding
the global shared WAL-index contained in the memory-mapped SHM file.

In addition to locking this byte, a connection running recovery also
gets an exclusive lock on all other WAL locks except for WAL_READ_LOCK(0).

WAL_READ_LOCK(N)

There are five separate read locks, numbers 0 through 4.
Read locks may be either SHARED or EXCLUSIVE.
Connections obtain a shared lock on one of the read locks bytes while
they are within a transaction.
Connections also obtain an exclusive lock on read locks, one at a time,
for the brief moment while they are updating the values of the corresponding
read-marks.
Read locks 1 through 4 are held exclusively when running recovery.

Each read lock byte corresponds to one of the five 32-bit read-mark
integers located in bytes 100 through 119 of the WAL-index header, as
follows:


Lock NameLock offsetRead-mark nameRead-mark offset
WAL_READ_LOCK(0)123read-mark[0]100..103
WAL_READ_LOCK(1)124read-mark[1]104..107
WAL_READ_LOCK(2)125read-mark[2]108..111
WAL_READ_LOCK(3)126read-mark[3]112..115
WAL_READ_LOCK(4)127read-mark[4]116..119



When a connection holds a shared lock on WAL_READ_LOCK(N), that is a
promise by the connection that it will use the WAL and not the database
file for any database pages that are modified by the first
read-mark[N] entries in the WAL.
The read-mark[0] is always zero.  If a connection holds a shared lock
on WAL_READ_LOCK(0), that means the connection expects to be able to ignore
the WAL and read any content it wants from the main database.
If N>0 then the connection is free to use more of the WAL file beyond
read-mark[N] if it wants to, up to the first mxFrame frames.
But when a connection holds a shared lock on WAL_READ_LOCK(0), that is a
promise that it will never read content from the WAL
and will acquire all content directly from the main database.

When a checkpoint runs, if it sees a lock on WAL_READ_LOCK(N), then it
must not move WAL content into the main database for more than the first
read-mark[N] frames.  Were it to do so, it would overwrite content that
the process holding the lock was expecting to be able to read out of the
main database file.  A consequence of if this is that f the WAL file
contains more than read-mark[N] frames (if mxFrame>read-mark[N]
for any read-mark for which WAL_READ_LOCK(N) is held by another process),
then the checkpoint cannot run to completion.

When a writer wants to reset the WAL, it must ensure that there are
no locks on WAL_READ_LOCK(N) for N>0 because such locks indicate
that some other connection is still using the current WAL file and 
a WAL reset would delete content out from those other connections.  It is
ok for a WAL reset to occur if other connections are holding WAL_READ_LOCK(0)
because by holding WAL_READ_LOCK(0), those other connections are promising
not to use any content from the WAL.


2.3.2. Operations that require locks and which locks those operations use


Transition into and out of WAL-mode

The SQLITE_LOCK_EXCLUSIVE lock must be held by a connection that wants
to transition into our out of WAL mode.
Transitioning into WAL mode is, therefore, just like any other write
transaction, since every write transaction in rollback mode requires
the SQLITE_LOCK_EXCLUSIVE lock.
If the database file is already in WAL mode (hence if the desire it to change
it back into rollback mode) and if there are two
or more connections to the database, then each of these connections will
be holding an SQLITE_LOCK_SHARED lock.  That means that the
SQLITE_LOCK_EXCLUSIVE cannot be obtained, and the transition out of
WAL mode will not be allowed.  This prevents one connection from deleting
WAL mode out from under another.  It also means that the only way to move
a database from WAL mode into rollback mode is to close all but one
connection to the database.


Close a connection to a WAL mode database

When a database connection closes (via sqlite3_close() or
sqlite3_close_v2()), an attempt is made to acquire
SQLITE_LOCK_EXCLUSIVE.  If this attempt is successful, that means
the connection that is closing is the last connection to the database.
In that case, it is desirable to clean up the WAL and WAL-index files,
so the closing connection runs a checkpoint (while holding
SQLITE_LOCK_EXCLUSIVE) and the deletes both the WAL and WAL-index files.
The SQLITE_LOCK_EXCLUSIVE is not released until after both the
WAL and WAL-index files have been deleted.

If the application invokes
sqlite3_file_control(SQLITE_FCNTL_PERSIST_WAL) on the database
connection prior to closing, then the final checkpoint is still
run but the WAL and WAL-index files are
not deleted as they normally would be.
This leaves the database in a state that allows other processes
without write permission on the database, WAL, or WAL-index files
to open the database read-only.
If the WAL and WAL-index files are missing, then a process that
lacks permission to create and initialize those files will not be
able to open the database, unless the database is designated
as immutable using the immutable query parameter.

Reconstruct the global shared WAL-index during recovery

All of the WAL-index locks, except for WAL_READ_LOCK(0),
are held exclusively while reconstructing the global shared WAL-index
during recovery.  

Append a new transaction to the end of the WAL

An exclusive lock is held on WAL_WRITE_LOCK while adding new
frame onto the end of a WAL file.  

Read content from the database and WAL as part of a transaction

Run a checkpoint

Reset the WAL file

A WAL reset means to rewind the WAL and start adding new frames
at the beginning.  This occurs while appending new frames to a WAL
that has mxFrame equal to nBackfill and which has no locks on
WAL_READ_LOCK(1) through WAL_READ_LOCK(4).  The WAL_WRITE_LOCK is
held.





3. Recovery


Recovery is the process of rebuilding the WAL-index so that it is
synchronized with the WAL.


Recovery is run by the first thread to connect to a WAL-mode database.
Recovery restores the WAL-index so that it accurately describes the
WAL file.  If there is no WAL file present when the first thread connects
to the database, there is nothing to recover, but the recovery process
still runs to initialize the WAL-index.


If the WAL-index is implemented as a memory-mapped file and that file is
read-only to the first thread to connect, then that thread creates an
private heap-memory ersazt WAL-index and runs the recovery routine to
populate that private WAL-index.  The same data results, but it is held
privately rather that being written into the public shared memory area.


Recovery works by doing a single pass over the WAL, from beginning to end.
The checksums are verified on each frame of the WAL as it is read.  The
scan stops at the end of the file or at the first invalid checksum.
The mxFrame field is set to the index of the last valid commit frame
in WAL.  Since WAL frame numbers are indexed starting with 1, mxFrame is
also the number of valid frames in the WAL.  A "commit frame" is a frame
that has a non-zero value in bytes 4 through 7 of the frame header.
Since the recovery procedure has no way of knowing how many frames of the
WAL might have previously been copied back into the database, it initializes
the nBackfill value to zero.


During recovery of the global shared-memory WAL-index, exclusive locks are
held on WAL_WRITE_LOCK, WAL_CKPT_LOCK, WAL_RECOVER_LOCK, and WAL_READ_LOCK(1) through WAL_READ_LOCK(4).  In other words, all locks associated with the
WAL-index except for WAL_READ_LOCK(0) are held exclusively.  This prevents
any other thread from writing the database and from reading any transactions
that are held in the WAL, until the recovery is complete.


c2: High Reliability
c5: hirely.html
c4:
SQLite is a high-reliability storage solution.  It does not give
problems.  It just works.


The high-reliability of SQLite is proven in practice.
SQLite has been used without problems in multiple billions of smart-phones, 
IoT devices, and desktop applications, around the world, and for almost two
decades.


SQLite responds gracefully to system errors.  Obscure out-of-memory and
I/O errors are reported back up to the application.  These error reporting
paths are all carefully tested to ensure they will always work.


SQLite is resilient in the face of corrupt inputs, including maliciously
designed database files and SQL strings.  Extensive fuzz-testing ensures
that corrupt inputs will not lead to crashes or undefined behavior, but
will instead cause sensible errors to be reported back to the application.
(More...)


SQLite is built using a 
DO-178B-inspired process.
The testing standards for SQLite are among the highest for
commercial software.


SQLite is open-source but it is not open-contribution.
All the code in SQLite is written by a small team of experts.
The project does not accept "pull requests" or patches from 
anonymous passers-by on the internet.


The developers of SQLite intend to support the product through
the year 2050.  To this end, the source code is carefully documented
to promote long-term maintainability.  We prefer mature and stable
over trendy and cutting-edge.


All of these factors combine to make SQLite a very trouble-free
software library.


c2: The Carray() Table-Valued Function
c5: carray.html
c4:
1. Overview

Carray($PTR,$N) is a table-valued function with a single column (named
"value") and zero or more rows.
The "value" of each row in the carray() is taken from a C-language array
that is $N elements long.  $PTR is a pointer to the beginning of the array.
In this way, the carray() function provides a convenient mechanism to
bind C-language arrays to SQL queries.

2. Availability

The carray() function is not compiled into SQLite by default.
It is available as a loadable extension in the
ext/misc/carray.c
source file.

3. Details

The carray() function takes two or three arguments.
The first argument is a pointer to an array.  Since pointer values cannot
be specified directly in SQL, the first argument must be a parameter that
is bound to a pointer value using the sqlite3_bind_pointer() interface
using a pointer-type of "carray".
The second argument is the number of elements in the array.  The optional
third argument is a string that determines the datatype of the elements
in the C-language array.  Allowed values for the third argument are:


 'int32'
 'int64'
 'double'
 'char*'


The default datatype is 'int32'.

The carray() function can be used in the FROM clause of a query.
For example, to query two entries from the OBJ table using rowids
taken from a C-language array at address $PTR.

SELECT obj.* FROM obj, carray($PTR, 10) AS x
 WHERE obj.rowid=x.value;


This query gives the same result:

SELECT * FROM obj WHERE rowid IN carray($PTR, 10);


c2: Full-Featured SQL
c5: fullsql.html
c4:
Do not be misled by the "Lite" in the name.  SQLite has a full-featured
SQL implementation, including:


Tables, indexes,
    triggers, and views
    in unlimited quantity
Up to 32K columns in a table and unlimited rows
Multi-column indexes
Indexes can use DESC and COLLATE
Partial indexes
Indexes On Expressions
Clustered indexes
Covering indexes
CHECK, UNIQUE, NOT NULL, and FOREIGN KEY constraints.
ACID transactions using BEGIN, COMMIT, and ROLLBACK
Nested transactions using SAVEPOINT, RELEASE, and 
    ROLLBACK TO
Subqueries, including correlated subqueries
Up to 64-way joins
LEFT JOIN
DISTINCT, ORDER BY, GROUP BY, HAVING, LIMIT, and OFFSET
UNION, UNION ALL, INTERSECT, and EXCEPT
A rich library of standard SQL functions
Aggregate functions including DISTINCT aggregates
Window functions
UPDATE, DELETE, and INSERT (of course)
Common table expressions including
    recursive common table expressions
Row values
UPSERT
An advanced query planner
Full-text search
R-tree indexes
JSON support
The IS operator
Table-valued functions
REPLACE INTO
VACUUM
REINDEX
The GLOB operator
Hexadecimal integer literals
The ON CONFLICT clause
The INDEXED BY clause
Virtual tables
Multiple databases on the same database connection using
    ATTACH DATABASE
The ability to add application-defined SQL functions, including
    aggregate and table-valued functions.
Application-defined collating functions



There are many more features not listed above.
SQLite may be small in size and have "Lite" in its name, but it is
not lacking in capability.


c2: SQLite Autoincrement
c5: autoinc.html
c4:
1. Summary



  The AUTOINCREMENT keyword imposes extra CPU, memory, disk space,
  and disk I/O overhead and should be avoided if not strictly needed.
  It is usually not needed.

  In SQLite, a column with type INTEGER PRIMARY KEY is an alias for the ROWID
  (except in WITHOUT ROWID tables) which is always a 64-bit signed integer.

  On an INSERT, if the ROWID or INTEGER PRIMARY KEY column is not 
  explicitly given a value, then it
  will be filled automatically with an unused integer, usually
  one more than the largest ROWID currently in use.
  This is true regardless of whether or not the AUTOINCREMENT keyword is used.

  If the AUTOINCREMENT keyword appears after INTEGER PRIMARY KEY, that
  changes the automatic ROWID assignment algorithm to prevent
  the reuse of ROWIDs over the lifetime of the database.  In other words,
  the purpose of AUTOINCREMENT is to prevent the reuse of ROWIDs from
  previously deleted rows.


2. Background


In SQLite, table rows normally have a 64-bit signed integer ROWID
which is unique among all rows in the same table.
(WITHOUT ROWID tables are the exception.)



You can access the ROWID of an SQLite table using one of the special column
names ROWID, _ROWID_, or OID.
Except if you declare an ordinary table column to use one of those special
names, then the use of that name will refer to the declared column not
to the internal ROWID.



If a table contains a column of type INTEGER PRIMARY KEY, then that
column becomes an alias for the ROWID.  You can then access the ROWID
using any of four different names, the original three names described above
or the name given to the INTEGER PRIMARY KEY column.  All these names are
aliases for one another and work equally well in any context.



When a new row is inserted into an SQLite table, the ROWID can either
be specified as part of the INSERT statement or it can be assigned
automatically by the database engine.  To specify a ROWID manually,
just include it in the list of values to be inserted.  For example:


CREATE TABLE test1(a INT, b TEXT);
INSERT INTO test1(rowid, a, b) VALUES(123, 5, 'hello');



If no ROWID is specified on the insert, or if the specified ROWID has a value
of NULL, then an appropriate ROWID is created
automatically.  The usual algorithm is to give the newly created row
a ROWID that is one larger than the largest ROWID in the table prior
to the insert.  If the table is initially empty, then a ROWID of 1 is
used.  If the largest ROWID is equal to the largest possible integer
(9223372036854775807) then the database
engine starts picking positive candidate ROWIDs at random until it finds one
that is not previously used.
If no unused ROWID can be found after a reasonable number of attempts,
the insert operation fails with an SQLITE_FULL error.
If no negative ROWID values are inserted explicitly, then automatically 
generated ROWID values will always be greater than zero.



The normal ROWID selection algorithm described above
will generate monotonically increasing
unique ROWIDs as long as you never use the maximum ROWID value and you never
delete the entry in the table with the largest ROWID. 
If you ever delete rows or if you ever create a row with the maximum possible
ROWID, then ROWIDs from previously deleted rows might be reused when creating
new rows and newly created ROWIDs might not be in strictly ascending order.



3. The AUTOINCREMENT Keyword


If a column has the type INTEGER PRIMARY KEY AUTOINCREMENT then a slightly
different ROWID selection algorithm is used.  
The ROWID chosen for the new row is at least one larger than the largest ROWID
that has ever before existed in that same table.  If the table has never
before contained any data, then a ROWID of 1 is used.  If the largest possible
ROWID has previously been inserted, then
new INSERTs are not allowed and any attempt to insert a new row will
fail with an SQLITE_FULL error.  
Only ROWID values from previous transactions that
were committed are considered.  ROWID values that were rolled back
are ignored and can be reused.



SQLite keeps track of the largest ROWID
using an internal table named "sqlite_sequence".
The sqlite_sequence table is created
and initialized automatically whenever a normal table that contains an
AUTOINCREMENT column is created.  The content of the sqlite_sequence table
can be modified using ordinary UPDATE, INSERT, and DELETE statements.
But making modifications to this table will likely perturb the AUTOINCREMENT
key generation algorithm.  Make sure you know what you are doing before
you undertake such changes.
The sqlite_sequence table does not track ROWID changes associated with
UPDATE statement, only INSERT statements.



The behavior implemented by the AUTOINCREMENT keyword is subtly different
from the default behavior.  With AUTOINCREMENT, rows with automatically
selected ROWIDs are guaranteed to have ROWIDs that have never been used
before by the same table in the same database.  And the automatically generated
ROWIDs are guaranteed to be monotonically increasing.  These are important
properties in certain applications.  But if your application does not
need these properties, you should probably stay with the default behavior
since the use of AUTOINCREMENT requires additional work to be done
as each row is inserted and thus causes INSERTs to run a little slower.


Note that "monotonically increasing" does not imply that the ROWID always
increases by exactly one.  One is the usual increment.  However, if an
insert fails due to (for example) a uniqueness constraint, the ROWID of
the failed insertion attempt might not be reused on subsequent inserts,
resulting in gaps in the ROWID sequence.  AUTOINCREMENT guarantees that
automatically chosen ROWIDs will be increasing but not that they will be
sequential.

Because AUTOINCREMENT keyword changes the behavior of the ROWID selection
algorithm, AUTOINCREMENT is not allowed on WITHOUT ROWID tables or on any
table column other than INTEGER PRIMARY KEY.  Any attempt to use 
AUTOINCREMENT on a WITHOUT ROWID table or on a column other than the
INTEGER PRIMARY KEY column results in an error.


c2: The CSV Virtual Table
c5: csv.html
c4:
1. Overview


The CSV virtual table reads
RFC 4180 formatted comma-separated
values, and returns that content as if it were rows and columns of an SQL
table.



The CSV virtual table is useful to applications that need to bulk-load
large amounts of comma-separated value content.
The CSV virtual table is also useful as a template source file for
implementing other virtual tables.




The CSV virtual table is not built into the SQLite amalgamation.
It is available as a
separate source file
that can be compiled into a loadable extension.
Typical usage of the CSV virtual table from the
command-line shell would be something like this:

.load ./csv
CREATE VIRTUAL TABLE temp.t1 USING csv(filename='thefile.csv');
SELECT * FROM t1;



The first line of the script above causes the command-line shell to
read and activate the run-time loadable extension for CSV.  For an
application, the equivalent C-language API is
sqlite3_load_extension().
Observe that the filename extension (ex: ".dll" or ".so" or ".dylib") is
omitted from the extension filename.  Omitting the filename extension is
not required, but it helps in making the script cross-platform.  SQLite
will automatically append the appropriate extension.


The second line above creates a virtual table named "t1" that reads
the content of the file named in the argument.  The number and names of
the columns is determined automatically by reading the first line of
content.  Other options to the CSV virtual table provide the ability to
take the CSV content from a string rather than a separate file, and give 
the programmer more control over the number and names of the columns.
The options are detailed below.  The CSV virtual table is usually
created as a TEMP table so that it exists only for the current database
connection and does not become a permanent part of the database schema.
Note that there is no "CREATE TEMP VIRTUAL TABLE" command in SQLite.
Instead, prepend the "temp." schema prefix to the name of the virtual
table.


The third line of the example shows the virtual table being used, to read
all content of the CSV file.  This is perhaps the simplest possible use
of the virtual table.  The CSV virtual table can be used anywhere an ordinary
virtual table can be used.  One can use the CSV virtual table inside subqueries,
or common table expressions or add WHERE, GROUP BY, HAVING, ORDER BY,
and LIMIT clauses as required.

2. Arguments


The example above showed a single filename='thefile.csv' argument
for the CSV virtual table.  But other arguments are also possible.


filename=FILENAME
The filename= argument specifies an external file from which
CSV content is read.  Every CSV virtual table must have either a 
filename= argument or a data= argument and not both.

data=TEXT
The data= argument specifies that TEXT is the literal
content of the CSV file.

schema=SCHEMA
 The schema= argument specifies a CREATE TABLE statement that
the CSV virtual table passes to the sqlite3_declare_vtab() interface in
order to define the names of the columns in the virtual table.

columns=N
The columns=N argument specifies the number of columns
in the CSV file.
If the input data contains more columns than this,
then the excess columns are ignored.  If the input data contains fewer columns,
then extra columns are filled with NULL.
If the columns=N argument is omitted, the first line of the
CSV file is read to determine the number of columns.

header=BOOLEAN
or just
header
If the header argument is true then the first row of the CSV file
to be treated as a header rather than as data.  The second line of the CSV
file becomes the first row of content.
If the schema= options is omitted, then the first line of the CSV
file determines the names of the columns.


3. Column Names


The column names of the virtual table are determined primarily by the
schema= argument.
If the schema= argument is omitted, but header is true, then
the values found in the first line of the CSV file become the column names.
If the schema= argument is omitted and header is false, then
the columns are named "c0", "c1", "c2", and so forth.


c2: SQLite Version 3 Overview
c5: version3.html
c4:
Editorial Note:
This document was written in 2004 as a guide to programmers who were
transitioning from SQLite2 to SQLite3.
It is retained as part of the historical record of SQLite.
Modern programmers should refer to
more up-to-date documentation on SQLite available elsewhere
on this website.


SQLite Version 3 Overview


SQLite version 3.0 introduces important changes to the library, including:



A more compact format for database files.
Manifest typing and BLOB support.
Support for both UTF-8 and UTF-16 text.
User-defined text collating sequences.
64-bit ROWIDs.
Improved Concurrency.



This document is a quick introduction to the changes for SQLite 3.0
for users who are already familiar with SQLite version 2.8.


Naming Changes


SQLite version 2.8 will continue to be supported with bug fixes
for the foreseeable future.  In order to allow SQLite version 2.8
and SQLite version 3.0 to peacefully coexist, the names of key files
and APIs in SQLite version 3.0 have been changed to include the
character "3".  For example, the include file used by C programs
has been changed from "sqlite.h" to "sqlite3.h".  And the name of
the shell program used to interact with databases has been changed
from "sqlite.exe" to "sqlite3.exe".  With these changes, it is possible
to have both SQLite 2.8 and SQLite 3.0 installed on the same system at
the same time.  And it is possible for the same C program to link
against both SQLite 2.8 and SQLite 3.0 at the same time and to use
both libraries at the same time.


New File Format


The format used by SQLite database files has been completely revised.
The old version 2.1 format and the new 3.0 format are incompatible with
one another.  Version 2.8 of SQLite will not read a version 3.0 database
files and version 3.0 of SQLite will not read a version 2.8 database file.



To convert an SQLite 2.8 database into an SQLite 3.0 database, have
ready the command-line shells for both version 2.8 and 3.0.  Then
enter a command like the following:



sqlite OLD.DB .dump | sqlite3 NEW.DB



The new database file format uses B+trees for tables.  In a B+tree, all
data is stored in the leaves of the tree instead of in both the leaves and
the intermediate branch nodes.  The use of B+trees for tables allows for
better scalability and the storage of larger data fields without the use of
overflow pages.  Traditional B-trees are still used for indices.


The new file format also supports variable pages sizes between 512 and
65536 bytes.  The size of a page is stored in the file header so the
same library can read databases with different pages sizes, in theory,
though this feature has not yet been implemented in practice.



The new file format omits unused fields from its disk images.  For example,
indices use only the key part of a B-tree record and not the data.  So
for indices, the field that records the length of the data is omitted.
Integer values such as the length of key and data are stored using
a variable-length encoding so that only one or two bytes are required to
store the most common cases but up to 64-bits of information can be encoded
if needed. 
Integer and floating point data is stored on the disk in binary rather
than being converted into ASCII as in SQLite version 2.8.
These changes taken together result in database files that are typically
25% to 35% smaller than the equivalent files in SQLite version 2.8.



Details of the low-level B-tree format used in SQLite version 3.0 can
be found in header comments to the 
btreeInt.h
source file and in the file format documentation.


Manifest Typing and BLOB Support


SQLite version 2.8 will deal with data in various formats internally,
but when writing to the disk or interacting through its API, SQLite 2.8
always converts data into ASCII text.  SQLite 3.0, in contrast, exposes 
its internal data representations to the user and stores binary representations
to disk when appropriate.  The exposing of non-ASCII representations was
added in order to support BLOBs.



SQLite version 2.8 had the feature that any type of data could be stored
in any table column regardless of the declared type of that column.  This
feature is retained in version 3.0, though in a slightly modified form.
Each table column will store any type of data, though columns have an
affinity for the format of data defined by their declared datatype.
When data is inserted into a column, that column will make an attempt
to convert the data format into the column's declared type.   All SQL
database engines do this.  The difference is that SQLite 3.0 will 
still store the data even if a format conversion is not possible.



For example, if you have a table column declared to be of type "INTEGER"
and you try to insert a string, the column will look at the text string
and see if it looks like a number.  If the string does look like a number
it is converted into a number and into an integer if the number does not
have a fractional part, and stored that way.  But if the string is not
a well-formed number it is still stored as a string.  A column with a
type of "TEXT" tries to convert numbers into an ASCII-Text representation
before storing them.  But BLOBs are stored in TEXT columns as BLOBs because
you cannot in general convert a BLOB into text.



In most other SQL database engines the datatype is associated with
the table column that holds the data - with the data container.
In SQLite 3.0, the datatype is associated with the data itself, not
with its container.
Paul Graham in his book 
ANSI Common Lisp
calls this property "Manifest Typing".
Other writers have other definitions for the term "manifest typing",
so beware of confusion.  But by whatever name, that is the datatype
model supported by SQLite 3.0.



Additional information about datatypes in SQLite version 3.0 is
available
separately.


Support for UTF-8 and UTF-16


The new API for SQLite 3.0 contains routines that accept text as
both UTF-8 and UTF-16 in the native byte order of the host machine.
Each database file manages text as either UTF-8, UTF-16BE (big-endian),
or UTF-16LE (little-endian).  Internally and in the disk file, the
same text representation is used everywhere.  If the text representation
specified by the database file (in the file header) does not match
the text representation required by the interface routines, then text
is converted on-the-fly.
Constantly converting text from one representation to another can be
computationally expensive, so it is suggested that programmers choose a
single representation and stick with it throughout their application.



In the current implementation of SQLite, the SQL parser only works
with UTF-8 text.  So if you supply UTF-16 text it will be converted.
This is just an implementation issue and there is nothing to prevent
future versions of SQLite from parsing UTF-16 encoded SQL natively.



When creating new user-defined SQL functions and collating sequences,
each function or collating sequence can specify if it works with
UTF-8, UTF-16be, or UTF-16le.  Separate implementations can be registered
for each encoding.   If an SQL function or collating sequence is required
but a version for the current text encoding is not available, then 
the text is automatically converted.  As before, this conversion takes
computation time, so programmers are advised to pick a single
encoding and stick with it in order to minimize the amount of unnecessary
format juggling.



SQLite is not particular about the text it receives and is more than
happy to process text strings that are not normalized or even
well-formed UTF-8 or UTF-16.  Thus, programmers who want to store
ISO8859 data can do so using the UTF-8 interfaces.  As long as no
attempts are made to use a UTF-16 collating sequence or SQL function,
the byte sequence of the text will not be modified in any way.


User-defined Collating Sequences


A collating sequence is just a defined order for text.  When SQLite 3.0
sorts (or uses a comparison operator like "&lt;" or "&gt;=") the sort
order is first determined by the data type.



NULLs sort first
Numeric values sort next in numerical order
Text values come after numerics
BLOBs sort last



Collating sequences are used for comparing two text strings.
The collating sequence does not change the ordering of NULLs, numbers,
or BLOBs, only text.



A collating sequence is implemented as a function that takes the
two strings being compared as inputs and returns negative, zero, or
positive if the first string is less than, equal to, or greater than
the second.
SQLite 3.0 comes with a single built-in collating sequence named "BINARY"
which is implemented using the memcmp() routine from the standard C library.
The BINARY collating sequence works well for English text.  For other
languages or locales, alternative collating sequences may be preferred.



The decision of which collating sequence to use is controlled by the
COLLATE clause in SQL.  A COLLATE clause can occur on a table definition,
to define a default collating sequence to a table column, or on field
of an index, or in the ORDER BY clause of a SELECT statement.
Planned enhancements to SQLite are to include standard CAST() syntax
to allow the collating sequence of an expression to be defined.


64-bit ROWIDs


Every row of a table has a unique rowid.
If the table defines a column with the type "INTEGER PRIMARY KEY" then that
column becomes an alias for the rowid.  But with or without an INTEGER PRIMARY
KEY column, every row still has a rowid.



In SQLite version 3.0, the rowid is a 64-bit signed integer.
This is an expansion of SQLite version 2.8 which only permitted
rowids of 32-bits.



To minimize storage space, the 64-bit rowid is stored as a variable length
integer.  Rowids between 0 and 127 use only a single byte.  
Rowids between 0 and 16383 use just 2 bytes.  Up to 2097152 uses three
bytes.  And so forth.  Negative rowids are allowed but they always use
nine bytes of storage and so their use is discouraged.  When rowids
are generated automatically by SQLite, they will always be non-negative.


Improved Concurrency


SQLite version 2.8 allowed multiple simultaneous readers or a single
writer but not both.  SQLite version 3.0 allows one process to begin
writing the database while other processes continue to read.  The
writer must still obtain an exclusive lock on the database for a brief
interval in order to commit its changes, but the exclusive lock is no
longer required for the entire write operation.
A more detailed report on the locking
behavior of SQLite version 3.0 is available separately.



A limited form of table-level locking is now also available in SQLite.
If each table is stored in a separate database file, those separate
files can be attached to the main database (using the ATTACH command)
and the combined databases will function as one.  But locks will only
be acquired on individual files as needed.  So if you redefine "database"
to mean two or more database files, then it is entirely possible for
two processes to be writing to the same database at the same time.
To further support this capability, commits of transactions involving
two or more ATTACHed database are now atomic.


Credits


SQLite version 3.0 is made possible in part by AOL developers
supporting and embracing great Open-Source Software.


c2: The SQLite Amalgamation
c5: amalgamation.html
c4:
1. Executive Summary

Over 100 separate source files are concatenated into a
single large files of C-code named "sqlite3.c" and
called "the amalgamation". The amalgamation
contains everything an application needs to embed SQLite.
The amalgamation file is more than 220,000 lines long and over 7.5
megabytes in size (as of 2018-11-24).

Combining all the code for SQLite into one big file makes SQLite
easier to deploy &mdash; there is just one file to keep track of.  
And because all code is in
a single translation unit, compilers can do better inter-procedure
optimization resulting in machine code that is between 5% and 10% faster.

2. The SQLite Amalgamation

The SQLite library consists of 102 files of C code
(as of Version 3.9.0 - 2015-10-14)
in the core with 32 additional files that
implement the FTS3, FTS5, RTREE, DBSTAT, JSON1, and
RBU extensions.
Of the 102 
main source files, about 75% are C code and about 25% are C header files.
Most of these are "source" files in the sense that they are stored 
in the SQLite version control system
and are edited manually in an ordinary text editor.
But some of the C-language files are generated using scripts
or auxiliary programs.  For example, the
parse.y
file contains an LALR(1) grammar of the SQL language which is compiled
down into are parser in files "parse.c" and "parse.h" by the
Lemon parser generator.


The makefiles for SQLite have an "sqlite3.c" target for building the
file we call "the amalgamation".
The amalgamation is a single C code file, named "sqlite3.c",
that contains all C code 
for the core SQLite library and the FTS3, FTS5, RTREE,
DBSTAT, JSON1, and RBU extensions.
This file contains about 184K lines of code 
(113K if you omit blank lines and comments) and is over 6.4 megabytes
in size.
Though the various extensions are included in the
"sqlite3.c" amalgamation file, they are disabled using #ifdef statements.
Activate the extensions using compile-time options like:


 -DSQLITE_ENABLE_FTS3
 -DSQLITE_ENABLE_FTS5
 -DSQLITE_ENABLE_RTREE
 -DSQLITE_ENABLE_DBSTAT_VTAB
 -DSQLITE_ENABLE_JSON1
 -DSQLITE_ENABLE_RBU


The amalgamation contains everything you need to integrate SQLite 
into a larger project.  Just copy the amalgamation into your source 
directory and compile it along with the other C code files in your project.
(A more detailed discussion of the compilation process is
available.)
You may also want to make use of the "sqlite3.h" header file that 
defines the programming API for SQLite. 
The sqlite3.h header file is available separately. 
The sqlite3.h file is also contained within the amalgamation, in
the first few thousand lines. So if you have a copy of 
sqlite3.c but cannot seem to locate sqlite3.h, you can always 
regenerate the sqlite3.h by copying and pasting from the amalgamation.

In addition to making SQLite easier to incorporate into other
projects, the amalgamation also makes it run faster. Many
compilers are able to do additional optimizations on code when 
it is contained with in a single translation unit such as it 
is in the amalgamation. We have measured performance improvements
of between 5 and 10% when we use the amalgamation to compile 
SQLite rather than individual source files.  The downside of this
is that the additional optimizations often take the form of 
function inlining which tends to make the size of the resulting
binary image larger.



3. The Split Amalgamation

Developers sometimes experience trouble debugging the
185,000-line-long amalgamation source file because some debuggers
are only able to handle source code line numbers less than 32,768.
The amalgamation source code runs fine.  One just cannot single-step
through it in a debugger.

To circumvent this limitation, the amalgamation is also available in
a split form, consisting of files "sqlite3-1.c", "sqlite3-2.c", and
so forth, where each file is less than 32,768 lines in length and
where the concatenation of the files contain all of the code for the
complete amalgamation.  Then there is a separate source file named
"sqlite3-all.c" which basically consists of code like this:

#include "sqlite3-1.c"
#include "sqlite3-2.c"
#include "sqlite3-3.c"
#include "sqlite3-4.c"
#include "sqlite3-5.c"
#include "sqlite3-6.c"
#include "sqlite3-7.c"


Applications using the split amalgamation simply compile against
"sqlite3-all.c" instead of "sqlite3.c".  The two files work exactly
the same.  But with "sqlite3-all.c", no single source file contains more
than 32,767 lines of code, and so it is more convenient to use some
debuggers.  The downside of the split amalgamation is that it consists
of 6 C source code files instead of just 1.

4. Download Copies Of The Amalgamation

The amalgamation and
the sqlite3.h header file are available on
the download page as a file 
named sqlite-amalgamation-X.zip
where the X is replaced by the appropriate version number.



5. Building The Amalgamation From Canonical Source Code

To build the amalgamation (either the full amalgamation or the
split amalgamation), first
get the canonical source code from one of the three servers.
Then, on both unix-like systems and on Windows systems that have the
free MinGW development environment
installed, the amalgamation can be built using the
following commands:

sh configure
make sqlite3.c


To build using Microsoft Visual C++, run this command:

nmake /f makefile.msc sqlite3.c


In both cases, the split amalgamation can be obtained by
substituting "sqlite3-all.c" for "sqlite3.c" as the make target.

5.1. Dependencies

The build process makes extensive use of the 
Tcl scripting language.  You will need to have a
copy of TCL installed in order for the make targets above to work.
Easy-to-use installers can be obtained from http://www.tcl-lang.org/.
Many unix workstations have Tcl installed by default.

5.2. See Also

Additional notes on compiling SQLite can be found on the
How To Compile SQLite page.


c2: The Tcl interface to the SQLite library
c5: tclsqlite.html
c4:
The Tcl interface to the SQLite library

The SQLite library is designed to be very easy to use from
a Tcl or Tcl/Tk script.  SQLite
began as a Tcl extension
and the primary test suite for SQLite is written in TCL.  SQLite
can be used with any programming language, but its connections to
TCL run deep.

This document gives an overview of the Tcl
programming interface for SQLite.

The API

The interface to the SQLite library consists of single
tcl command named sqlite3
Because there is only this
one command, the interface is not placed in a separate
namespace.

The sqlite3 command is used as follows:


sqlite3&nbsp;&nbsp;dbcmd&nbsp;&nbsp;database-name&nbsp;&nbsp;?options?



The sqlite3 command opens the database named in the second
argument.  If the database does not already exist, the default behavior
is for it to be created automatically (though this can be changed by
using the "-create false" option).
The sqlite3 command always creates a new Tcl
command to control the database.  The name of the new Tcl command
is given by the first argument.  This approach is similar to the
way widgets are created in Tk.



The name of the database is usually just the name of a disk file in which
the database is stored.  If the name of the database is 
the special name ":memory:" then a new database is created
in memory.  If the name of the database is an empty string, then
the database is created in an empty file that is automatically deleted
when the database connection closes.  URI filenames can be used if
the "-uri yes" option is supplied on the sqlite3 command.



Options understood by the sqlite3 command include:





-create BOOLEAN

If true, then a new database is created if one does not already exist.
If false, then an attempt to open a database file that does not previously
exist raises an error.  The default behavior is "true".


-nomutex BOOLEAN

If true, then all mutexes for the database connection are disabled.
This provides a small performance boost in single-threaded applications.


-readonly BOOLEAN

If true, then open the database file read-only.  If false, then the
database is opened for both reading and writing if filesystem permissions
allow, or for reading only if filesystem write permission is denied
by the operating system.  The default setting is "false".  Note that
if the previous process to have the database did not exit cleanly
and left behind a hot journal, then the write permission is required
to recover the database after opening, and the database cannot be
opened read-only.


-uri BOOLEAN

If true, then interpret the filename argument as a URI filename.  If
false, then the argument is a literal filename.  The default value is
"false".


-vfs VFSNAME

Use an alternative VFS named by the argument.








Once an SQLite database is open, it can be controlled using 
methods of the dbcmd.
There are currently 40 methods
defined.




authorizer
backup
bind_fallback
busy
cache
changes
close
collate
collation_needed
commit_hook
complete
config
copy
deserialize


enable_load_extension
errorcode
eval
exists
function
incrblob
interrupt
last_insert_rowid
nullvalue
onecolumn
preupdate
profile
progress
restore


rollback_hook
serialize
status
timeout
total_changes
trace
trace_v2
transaction
unlock_notify
update_hook
version
wal_hook





The use of each of these methods will be explained in the sequel, though
not in the order shown above.


The "eval" method


The most useful dbcmd method is "eval".  The eval method is used
to execute SQL on the database.  The syntax of the eval method looks
like this:


dbcmd&nbsp;&nbsp;eval&nbsp;&nbsp;?options?&nbsp;&nbsp;sql
&nbsp;&nbsp;?array-name?&nbsp;&nbsp;?script?



The job of the eval method is to execute the SQL statement or statements
given in the second argument.  For example, to create a new table in
a database, you can do this:


sqlite3 db1 ./testdb
db1 eval {CREATE TABLE t1(a int, b text)}


The above code creates a new table named t1 with columns
a and b.  What could be simpler?

Query results are returned as a list of column values.  If a
query requests 2 columns and there are 3 rows matching the query,
then the returned list will contain 6 elements.  For example:


db1 eval {INSERT INTO t1 VALUES(1,'hello')}
db1 eval {INSERT INTO t1 VALUES(2,'goodbye')}
db1 eval {INSERT INTO t1 VALUES(3,'howdy!')}
set x &#91;db1 eval {SELECT * FROM t1 ORDER BY a}&#93;


The variable $x is set by the above code to


1 hello 2 goodbye 3 howdy!


You can also process the results of a query one row at a time
by specifying the name of an array variable and a script following
the SQL code.  For each row of the query result, the values of all
columns will be inserted into the array variable and the script will
be executed.  For instance:


db1 eval {SELECT * FROM t1 ORDER BY a} values {
&nbsp;&nbsp;&nbsp;&nbsp;parray values
&nbsp;&nbsp;&nbsp;&nbsp;puts ""
}


This last code will give the following output:


values(*) = a b
values(a) = 1
values(b) = hello

values(*) = a b
values(a) = 2
values(b) = goodbye

values(*) = a b
values(a) = 3
values(b) = howdy!



For each column in a row of the result, the name of that column
is used as an index in to array and the value of the column is stored
in the corresponding array entry.  (Caution:  If two or more columns
in the result set of a query have the same name, then the last column
with that name will overwrite prior values and earlier columns with the
same name will be inaccessible.) The special array index * is
used to store a list of column names in the order that they appear.



Normally, NULL SQL results are stored in the array using the
nullvalue setting.  However, if
the -withoutnulls option is used, then NULL SQL values
cause the corresponding array element to be unset instead.



If the array variable name is omitted or is the empty string, then the value of
each column is stored in a variable with the same name as the column
itself.  For example:



db1 eval {SELECT * FROM t1 ORDER BY a} {
&nbsp;&nbsp;&nbsp;&nbsp;puts "a=$a b=$b"
}



From this we get the following output



a=1 b=hello
a=2 b=goodbye
a=3 b=howdy!





Tcl variable names can appear in the SQL statement of the second argument
in any position where it is legal to put a string or number literal.  The
value of the variable is substituted for the variable name.  If the
variable does not exist a NULL values is used.  For example:



db1 eval {INSERT INTO t1 VALUES(5,$bigstring)}



Note that it is not necessary to quote the $bigstring value.  That happens
automatically.  If $bigstring is a large string or binary object, this
technique is not only easier to write, it is also much more efficient
since it avoids making a copy of the content of $bigstring.



If the $bigstring variable has both a string and a "bytearray" representation,
then TCL inserts the value as a string.  If it has only a "bytearray"
representation, then the value is inserted as a BLOB.  To force a
value to be inserted as a BLOB even if it also has a text representation,
use a "@" character to in place of the "$".  Like this:



db1 eval {INSERT INTO t1 VALUES(5,@bigstring)}



If the variable does not have a bytearray representation, then "@" works
just like "$".  Note that ":" works like "$" in all cases so the following
is another way to express the same statement:



db1 eval {INSERT INTO t1 VALUES(5,:bigstring)}


The use of ":" instead of "$" before the name of a variable can 
sometimes be useful if the SQL text is enclosed in double-quotes "..."
instead of curly-braces {...}.  
When the SQL is contained within double-quotes "..." then TCL will do
the substitution of $-variables, which can lead to SQL injection if
extreme care is not used.  But TCL will never substitute a :-variable
regardless of whether double-quotes "..." or curly-braces {...} are 
used to enclose the SQL, so the use of :-variables adds an extra 
measure of defense against SQL
injection.



The "close" method



As its name suggests, the "close" method to an SQLite database just
closes the database.  This has the side-effect of deleting the
dbcmd Tcl command.  Here is an example of opening and then
immediately closing a database:



sqlite3 db1 ./testdb
db1 close



If you delete the dbcmd directly, that has the same effect
as invoking the "close" method.  So the following code is equivalent
to the previous:


sqlite3 db1 ./testdb
rename db1 {}


The "transaction" method



The "transaction" method is used to execute a TCL script inside an SQLite
database transaction.  The transaction is committed when the script completes,
or it rolls back if the script fails.  If the transaction occurs within
another transaction (even one that is started manually using BEGIN) it
is a no-op.



The transaction command can be used to group together several SQLite
commands in a safe way.  You can always start transactions manually using
BEGIN, of
course.  But if an error occurs so that the COMMIT or ROLLBACK are never
run, then the database will remain locked indefinitely.  Also, BEGIN
does not nest, so you have to make sure no other transactions are active
before starting a new one.  The "transaction" method takes care of
all of these details automatically.



The syntax looks like this:



dbcmd&nbsp;&nbsp;transaction&nbsp;&nbsp;?transaction-type?
&nbsp;&nbsp;script




The transaction-type can be one of deferred,
exclusive or immediate.  The default is deferred.


The "cache" method



The "eval" method described above keeps a cache of
prepared statements
for recently evaluated SQL commands.  
The "cache" method is used to control this cache.
The first form of this command is:


dbcmd&nbsp;&nbsp;cache size&nbsp;&nbsp;N


This sets the maximum number of statements that can be cached.
The upper limit is 100.  The default is 10.  If you set the cache size
to 0, no caching is done.

The second form of the command is this:



dbcmd&nbsp;&nbsp;cache flush


The cache-flush method 
finalizes
all prepared statements currently
in the cache.


The "complete" method



The "complete" method takes a string of supposed SQL as its only argument.
It returns TRUE if the string is a complete statement of SQL and FALSE if
there is more to be entered.

The "complete" method is useful when building interactive applications
in order to know when the user has finished entering a line of SQL code.
This is really just an interface to the 
sqlite3_complete() C
function.

The "config" method



The "config" method queries or changes certain configuration settings for
the database connection using the sqlite3_db_config() interface.
Run this method with no arguments to get a TCL list of available
configuration settings and their current values:


dbcmd&nbsp;&nbsp;config


The above will return something like this:


defensive 0 dqs_ddl 1 dqs_dml 1 enable_fkey 0 enable_qpsg 0 enable_trigger 1 enable_view 1 fts3_tokenizer 1 legacy_alter_table 0 legacy_file_format 0 load_extension 0 no_ckpt_on_close 0 reset_database 0 trigger_eqp 0 trusted_schema 1 writable_schema 0


Add the name of an individual configuration setting to query the current
value of that setting.  Optionally add a boolean value to change a setting.


The following four configuration changes are recommended for maximum
application security.  Turning off the trust_schema setting prevents
virtual tables and dodgy SQL functions from being used inside of triggers,
views, CHECK constraints, generated columns, and expression indexes.
Turning off the dqs_dml and dqs_ddl settings prevents the use of 
double-quoted strings.  Turning on defensive prevents direct writes
to shadow tables.


db config trusted_schema 0
db config defensive 1
db config dqs_dml 0
db config dqs_ddl 0


The "copy" method



The "copy" method copies data from a file into a table.
It returns the number of rows processed successfully from the file.
The syntax of the copy method looks like this:


dbcmd&nbsp;&nbsp;copy&nbsp;&nbsp;conflict-algorithm
&nbsp;&nbsp;table-name&nbsp;&nbsp;&nbsp;file-name&nbsp;
&nbsp;&nbsp;&nbsp;&nbsp;?column-separator?
&nbsp;&nbsp;?null-indicator?


Conflict-algorithm must be one of the SQLite conflict algorithms for
the INSERT statement: rollback, abort,
fail,ignore, or replace. See the SQLite Language
section for ON CONFLICT for more information.
The conflict-algorithm must be specified in lower case.


Table-name must already exists as a table.  File-name must exist, and
each row must contain the same number of columns as defined in the table.
If a line in the file contains more or less than the number of columns defined,
the copy method rollbacks any inserts, and returns an error.

Column-separator is an optional column separator string.  The default is
the ASCII tab character \t. 

Null-indicator is an optional string that indicates a column value is null.
The default is an empty string.  Note that column-separator and
null-indicator are optional positional arguments; if null-indicator
is specified, a column-separator argument must be specified and
precede the null-indicator argument.

The copy method implements similar functionality to the .import
SQLite shell command. 

The "timeout" method


The "timeout" method is used to control how long the SQLite library
will wait for locks to clear before giving up on a database transaction.
The default timeout is 0 millisecond.  (In other words, the default behavior
is not to wait at all.)

The SQLite database allows multiple simultaneous
readers or a single writer but not both.  If any process is writing to
the database no other process is allows to read or write.  If any process
is reading the database other processes are allowed to read but not write.
The entire database shared a single lock.

When SQLite tries to open a database and finds that it is locked, it
can optionally delay for a short while and try to open the file again.
This process repeats until the query times out and SQLite returns a
failure.  The timeout is adjustable.  It is set to 0 by default so that
if the database is locked, the SQL statement fails immediately.  But you
can use the "timeout" method to change the timeout value to a positive
number.  For example:

db1 timeout 2000

The argument to the timeout method is the maximum number of milliseconds
to wait for the lock to clear.  So in the example above, the maximum delay
would be 2 seconds.

The "busy" method


The "busy" method, like "timeout", only comes into play when the
database is locked.  But the "busy" method gives the programmer much more
control over what action to take.  The "busy" method specifies a callback
Tcl procedure that is invoked whenever SQLite tries to open a locked
database.  A single integer argument is appended to the callback before
it is invoke.  The argument is the number of prior calls to the busy
callback for the current locking event.  It is intended that
the callback will do some other useful work for a short while 
(such as service GUI events) then return
so that the lock can be tried again.  The callback procedure should
return "0" if it wants SQLite to try again to open the database and
should return "1" if it wants SQLite to abandon the current operation.

If the busy method is invoked without an argument, the name of the callback
procedure last set by the busy method is returned.  If no callback procedure
has been set, an empty string is returned.

The "enable_load_extension" method


The extension loading mechanism of SQLite (accessed using the
load_extension() SQL function) is turned off by default.  This is
a security precaution.  If an application wants to make use of the
load_extension() function it must first turn the capability on using
this method.

This method takes a single boolean argument which will turn the
extension loading functionality on or off.

For best security, do not use this method unless truly needed, 
and run PRAGMA trusted_schema=OFF or the
"db config trusted_schema 0" method before
invoking this method.

This method maps to the sqlite3_enable_load_extension() C/C++
interface.

The "exists" method


The "exists" method is similar to "onecolumn" and "eval" in that
it executes SQL statements.  The difference is that the "exists" method
always returns a boolean value which is TRUE if a query in the SQL
statement it executes returns one or more rows and FALSE if the SQL
returns an empty set.

The "exists" method is often used to test for the existence of
rows in a table.  For example:


if {&#91;db exists {SELECT 1 FROM table1 WHERE user=$user}&#93;} {
&nbsp;&nbsp;&nbsp;# Processing if $user exists
} else {
&nbsp;&nbsp;&nbsp;# Processing if $user does not exist
}


The "last_insert_rowid" method


The "last_insert_rowid" method returns an integer which is the ROWID
of the most recently inserted database row.

The "function" method


The "function" method registers new SQL functions with the SQLite engine.
The arguments are the name of the new SQL function and a TCL command that
implements that function.  Arguments to the function are appended to the
TCL command before it is invoked.


For security reasons, it is recommended that applications first set
PRAGMA trusted_schema=OFF or run the 
"db config trusted_schema 0"
method before using this method.


The following example creates a new SQL function named "hex" that converts
its numeric argument in to a hexadecimal encoded string:



db function hex {format 0x%X}


The "function" method accepts the following options:



-argcount INTEGER

Specify the number of arguments that the SQL function accepts.  The default
value of -1 means any number of arguments.

-deterministic

This option indicates that the function will always return the same
answer given the same argument values.  The SQLite query optimizer
uses this information to cache answers from function calls with 
constant inputs and reuse the result rather than invoke the function
repeatedly.


-directonly

This option restricts the function to only be usable by direct
top-level SQL statement.  The function will not be accessible to
triggers, views, CHECK constraints, generated columns, or index
expressions.  This option is recommended for all application-defined
SQL functions, and is highly recommended for any SQL
function that has side effects or that reveals internal state of
the application.

Security Warning:
Without this switch, an attacker might be able to change the
schema of a database file to include the new function inside a trigger
or view or CHECK constraint and thereby trick the application into
running the function with parameters of the attacker's choosing.
Hence, if the new function has side effects or reveals internal
state about the application and the -directonly option is not
used, that is a potential security vulnerability.


-innocuous

This option indicates that the function has no side effects and
does not leak any information that cannot be computed directly
from its input parameters.  When this option is specified, the
function may be used in triggers, views, CHECK constraints,
generated columns, and/or index expressions even if
PRAGMA trusted_schema=OFF.  The use of this option is discouraged
unless it is truly needed.


-returntype integer|real|text|blob|any

This option is used to configure the type of the result returned by the
function.  If this option is set to "any" (the default), SQLite attempts to
determine the type of each value returned by the function implementation based
on the Tcl value's internal type. Or, if it is set to "text" or "blob", the
returned value is always a text or blob value, respectively. If this option is
set to "integer", SQLite attempts to coerce the value returned by the function
to an integer. If this is not possible without data loss, it attempts to coerce
it to a real value, and finally falls back to text. If this option is set to
"real", an attempt is made to return a real value, falling back to text if this
is not possible.







The "nullvalue" method



The "nullvalue" method changes the representation for NULL returned
as result of the "eval" method.


db1 nullvalue NULL


The "nullvalue" method is useful to differ between NULL and empty
column values as Tcl lacks a NULL representation.  The default
representation for NULL values is an empty string.

The "onecolumn" method


The "onecolumn" method works like 
"eval" in that it evaluates the
SQL query statement given as its argument.  The difference is that
"onecolumn" returns a single element which is the first column of the
first row of the query result.

This is a convenience method.  It saves the user from having to
do a "&#91;lindex&nbsp;...&nbsp;0&#93;" on the results of an "eval"
in order to extract a single column result.

The "changes" method


The "changes" method returns an integer which is the number of rows
in the database that were inserted, deleted, and/or modified by the most
recent "eval" method.

The "total_changes" method


The "total_changes" method returns an integer which is the number of rows
in the database that were inserted, deleted, and/or modified since the
current database connection was first opened.

The "authorizer" method


The "authorizer" method provides access to the 
sqlite3_set_authorizer
C/C++ interface.  The argument to authorizer is the name of a procedure that
is called when SQL statements are being compiled in order to authorize
certain operations.  The callback procedure takes 5 arguments which describe
the operation being coded.  If the callback returns the text string
"SQLITE_OK", then the operation is allowed.  If it returns "SQLITE_IGNORE",
then the operation is silently disabled.  If the return is "SQLITE_DENY"
then the compilation fails with an error.


If the argument is an empty string then the authorizer is disabled.
If the argument is omitted, then the current authorizer is returned.

The "bind_fallback" method


The "bind_fallback" method gives the application control over how to
handle parameter binding when no TCL variable matches the parameter name.

When the eval method sees a named SQL
parameter such as "$abc" or ":def" or "@ghi" in an SQL statement, it tries
to look up a TCL variable with the same name, and it binds the value
of that TCL variable to the SQL parameter.  If no such TCL variable exists,
the default behavior is to bind an SQL NULL value to the parameter.  However,
if a bind_fallback proc is specified, then that proc is invoked with the
name of the SQL parameter and the return value from the proc is bound to
the SQL parameter.  Or if the proc returns an error, then the SQL statement
aborts with that error.  If the proc returns with some code other than
TCL_OK or TCL_ERROR, then the SQL parameter is bound to NULL, as it would
be by default.

The "bind_fallback" method has a single optional argument.  If the argument
is an empty string, then the bind_fallback is cancelled and the default behavior
is restored.  If the argument is a non-empty string, then the argument is a
TCL command (usually the name of a proc) to invoke whenever an SQL parameter 
is seen that does not match any TCL variable.  If the "bind_fallback" method 
is given no arguments, then the current bind_fallback command is returned.

As an example, the following setup causes TCL to throw an error if
an SQL statement contains an parameter that does not match any global
TCL variable:


proc bind_error {nm} {
  error "no such variable: $nm"
}
db bind_fallback bind_error


The "progress" method


This method registers a callback that is invoked periodically during
query processing.  There are two arguments: the number of SQLite virtual
machine opcodes between invocations, and the TCL command to invoke.
Setting the progress callback to an empty string disables it.

The progress callback can be used to display the status of a lengthy
query or to process GUI events during a lengthy query.

The "collate" method


This method registers new text collating sequences.  There are
two arguments: the name of the collating sequence and the name of a
TCL procedure that implements a comparison function for the collating
sequence.


For example, the following code implements a collating sequence called
"NOCASE" that sorts in text order without regard to case:



proc nocase_compare {a b} {
  return &#91;string compare &#91;string tolower $a] &#91;string tolower $b]]
}
db collate NOCASE nocase_compare


The "collation_needed" method


This method registers a callback routine that is invoked when the SQLite
engine needs a particular collating sequence but does not have that
collating sequence registered.  The callback can register the collating
sequence.  The callback is invoked with a single parameter which is the
name of the needed collating sequence.

The "commit_hook" method


This method registers a callback routine that is invoked just before
SQLite tries to commit changes to a database.  If the callback throws
an exception or returns a non-zero result, then the transaction rolls back
rather than commit.

The "rollback_hook" method


This method registers a callback routine that is invoked just before
SQLite tries to do a rollback.  The script argument is run without change.

The "status" method

This method returns status information from the most recently evaluated
SQL statement.  The status method takes a single argument which should be
either "steps" or "sorts".  If the argument is "steps", then the method
returns the number of full table scan steps that the previous SQL statement
evaluated.  If the argument is "sorts", the method returns the number of
sort operations.  This information can be used to detect queries that are
not using indices to speed search or sorting.

The status method is basically a wrapper on the
sqlite3_stmt_status() C-language interface.

The "update_hook" method


This method registers a callback routine that is invoked just before
each row is modified by an UPDATE, INSERT, or DELETE statement.  Four
arguments are appended to the callback before it is invoked:


The keyword "INSERT", "UPDATE", or "DELETE", as appropriate
The name of the database which is being changed
The table that is being changed
The rowid of the row in the table being changed


The "wal_hook" method


This method registers a callback routine that is invoked after transaction
commit when the database is in WAL mode.  Two arguments are appended to the
callback command before it is invoked:


The name of the database on which the transaction was committed
The number of entries in the write-ahead log (WAL) file for that database


This method might decide to run a checkpoint either itself or as a
subsequent idle callback.  Note that SQLite only allows a single WAL hook.
By default this single WAL hook is used for the auto-checkpointing.  If you
set up an explicit WAL hook, then that one WAL hook must ensure that checkpoints
are occurring since the auto-checkpointing mechanism will be disabled.

This method should return an integer value that is equivalent to an 
SQLite error code (usually 0 for SQLITE_OK in the case of success or 1 for
SQLITE_ERROR if some error occurs). As in sqlite3_wal_hook(), the results of
returning an integer that does not correspond to an SQLite error code are
undefined. If the value returned by the script cannot be interpreted as an
integer value, or if the script throws a Tcl exception, no error is returned to
SQLite but a Tcl background-error is raised.

The "incrblob" method


This method opens a TCL channel that can be used to read or write
into a preexisting BLOB in the database.  The syntax is like this:


dbcmd&nbsp;&nbsp;incrblob&nbsp;&nbsp;?-readonly?
&nbsp;&nbsp;?DB?&nbsp;&nbsp;TABLE&nbsp;&nbsp;COLUMN&nbsp;&nbsp;ROWID



The command returns a new TCL channel for reading or writing to the BLOB.
The channel is opened using the underlying 
sqlite3_blob_open() C-language
interface.  Close the channel using the close command of TCL.


The "errorcode" method


This method returns the numeric error code that resulted from the most
recent SQLite operation.

The "trace" method


The "trace" method registers a callback that is invoked as each SQL
statement is compiled.  The text of the SQL is appended as a single string
to the command before it is invoked.  This can be used (for example) to
keep a log of all SQL operations that an application performs.


The "trace_v2" method


The "trace_v2" method registers a callback that is invoked as each SQL
statement is compiled. The syntax is as follows:



dbcmd&nbsp;&nbsp;trace_v2&nbsp;&nbsp;?callback?&nbsp;&nbsp;?mask?


This command causes the "callback" script to be invoked whenever
certain conditions occurs.  The conditions are determined by the mask
argument, which should be a TCL-list of zero or more of the following
keywords:


 statement
 profile
 row
 close


Traces for statement invoke the callback with two arguments
whenever a new SQL statement is run.
The first argument is an integer which is the value of the pointer
to the underlying sqlite3_stmt object.  This integer can be used
to correlate SQL statement text with the result of a profile
or row callback. The second argument is the
unexpanded text of the SQL statement being run.  By "unexpanded", we
mean that variable substitutions in the text are not expanded into the
variable values.  This is different from the behavior of the "trace"
method which does expand variable substitutions.

Traces for profile invoke the callback with two arguments
as each SQL statement finishes.  The first argument is an integer which
is the value of the underlying sqlite3_stmt object.  The second
argument is the approximate run-time for the statement in nanoseconds.
The run-time is the best estimate available depending on the capabilities
of the platform on which the application is running.

Traces for row invoke the callback with a single argument
whenever a new result row is available from an SQL statement.
The argument is an integer which is the value of the underlying
sqlite3_stmt object pointer.

Traces for close invoke the callback with a single argument
as the database connection is closing.  The argument is an integer which
is the value of a pointer to the underlying sqlite3 object that is
closing.

There can only be a single trace callback registered on a database
connection.  Each use of "trace" or "trace_v2" cancels all prior
trace callback.

The "backup" method


The "backup" method makes a backup copy of a live database.  The
command syntax is like this:


dbcmd&nbsp;&nbsp;backup&nbsp;&nbsp;?source-database?&nbsp;&nbsp;backup-filename


The optional source-database argument tells which database in
the current connection should be backed up.  The default value is main
(or, in other words, the primary database file).  To back up TEMP tables
use temp.  To backup an auxiliary database added to the connection
using the ATTACH command, use the name of that database as it was assigned
in the ATTACH command.

The backup-filename is the name of a file into which the backup is
written.  Backup-filename does not have to exist ahead of time, but if
it does, it must be a well-formed SQLite database.

The "restore" method


The "restore" method copies the content from a separate database file 
into the current database connection, overwriting any preexisting content.
The command syntax is like this:


dbcmd&nbsp;&nbsp;restore&nbsp;&nbsp;?target-database?&nbsp;&nbsp;source-filename


The optional target-database argument tells which database in
the current connection should be overwritten with new content.  
The default value is main 
(or, in other words, the primary database file).  To repopulate the TEMP tables
use temp.  To overwrite an auxiliary database added to the connection
using the ATTACH command, use the name of that database as it was assigned
in the ATTACH command.

The source-filename is the name of an existing well-formed SQLite
database file from which the content is extracted.

The "serialize" method


The "serialize" method creates a BLOB which is a complete copy of an
underlying database.  The syntax is like this:


dbcmd&nbsp;&nbsp;serialize&nbsp;&nbsp;?database?


The optional argument is the name of the schema or database to be serialized.
The default value is "main".

This routine returns a TCL byte-array that is the complete content of
the identified database.  This byte-array can be written into a file and
then used as an ordinary SQLite database, or it can be sent over a TCP/IP
connection to some other application, or passed to the "deserialize" method
of another database connection.

This method only functions if SQLite is compiled with -DSQLITE_ENABLE_DESERIALIZE

The "deserialize" method

The "deserialize" method takes a TCL byte-array that contains an SQLite
database file and adds it to the database connection.  The syntax is:


dbcmd&nbsp;&nbsp;deserialize&nbsp;&nbsp;?database?&nbsp;&nbsp;value


The option database argument identifies which attached database
should receive the deserialization.  The default is "main".

This command causes SQLite to disconnect from the previous database and
reattach to an in-memory database with the content in value.  If value
is not a byte-array containing a well-defined SQLite database, then subsequent
commands will likely return SQLITE_CORRUPT errors.

This method only functions if SQLite is compiled with -DSQLITE_ENABLE_DESERIALIZE

The "interrupt" method

The "interrupt" method invokes the sqlite3_interrupt() interface, causing
any pending queries to halt.

The "version" method

  Return the current library version. For example, "3.23.0".

The "profile" method

This method is used to profile the execution of SQL statements run by
   the application. The syntax is as follows:



dbcmd&nbsp;&nbsp;profile&nbsp;&nbsp;?script?


Unless script is an empty string, this method arranges for the
script to be evaluated after the execution of each SQL statement.
Two arguments are appended to script before it is invoked: the
text of the SQL statement executed and the time elapsed while executing
the statement, in nanoseconds.

A database handle may only have a single profile script registered at
any time. If there is already a script registered when the profile method
is invoked, the previous profile script is replaced by the new one. If the 
script argument is an empty string, any previously registered 
profile callback is canceled but no new profile script is registered.


The "unlock_notify" method

The unlock_notify method is used access the sqlite3_unlock_notify()
   interface to the SQLite core library for testing purposes.  The use of
   this method by applications is discouraged.


c2: List Of Virtual Tables
c5: vtablist.html
c4:
1. Introduction

A virtual table is an object that presents an SQL table
interface but which is not stored in the database file, at least
not directly.  The virtual table mechanism is a feature
of SQLite that allows SQLite to access and manipulate resources
other than bits in the database file using the powerful SQL
query language.


The table below lists a few of the virtual tables implementations
available for SQLite.  Developers can deploy these virtual tables
in their own applications, or use the implementations shown below
as templates for writing their own virtual tables.


The list below is not exhaustive.
Other virtual table implementation exist in the SQLite source tree
and elsewhere.
The list below tries to capture the more interesting virtual table
implementations.



2. Virtual Tables


NameDescription
approximate_match
A demonstration of how to use a virtual table to implement
    approximate string matching.
carray
A table-valued function that allows a C-language array of integers,
    doubles, or strings to be used as table in a query.
closure
Compute the transitive closure of a set.
completion
Suggests completions for partially-entered words during interactive
    SQL input.  Used by the CLI to help implement tab-completion.
csv
A virtual table that represents a comma-separated-value or CSV file
    (RFC 4180) as a read-only
    table so that it can be used as part of a larger query.
dbstat
Provides information about the purpose and use of each page in a
    database file.  Used in the implementation of the
    sqlite3_analyzer utility program.
files_of_checkin
Provides information about all files in a single check-in in the
    Fossil version control system.  This
    virtual table is not part of the SQLite project but is included because
    it provides an example of how to use virtual tables and because it is
    used to help version control the SQLite sources.
fsdir
A table-valued function returning one row for each file in a
    selected file hierarchy of the host computer.  Used by the
    CLI to help implement the .archive command.
FTS3
A high-performance full-text search index.
FTS5
A higher-performance full-text search index
generate_series
A table-valued function returning a sequence of increasing
    integers, modeled after the table-valued function by the same
    name in PostgreSQL.
json_each
A table-valued function for decomposing a JSON string.
json_tree
A table-valued function for decomposing a JSON string.
OsQuery
Hundreds of virtual tables that publish various aspects of the
    host computer, such as the process table, user lists, active network
    connections, and so forth.
    OsQuery is a separate project, started by Facebook, hosted on
    GitHub, and intended for
    security analysis and intrusion detection
    OsQuery is not a part of the SQLite project, but is included in this
    list because it demonstrates how the SQL language and the
    SQLite virtual table mechanism
    can be leveraged to provide elegant solutions
    to important real-world problems.
pragma
Built-in table-valued functions that return the results of PRAGMA
    statements for use within ordinary SQL queries.
RTree
An implementation of the Guttmann R*Tree spatial index idea.
spellfix1
A virtual table that implements a spelling correction engine.
sqlite_btreeinfo
This experimental table-valued function provides information about
    a single B-tree in a database file, such as the depth, and estimated
    number of pages and number of entries, and so forth.
sqlite_dbpage
Key/value store for the raw database file content.  The key is the
    page number and the value is binary page content.
sqlite_memstat
Provides SQL access to the sqlite3_status64() and
    sqlite3_db_status() interfaces.
sqlite_stmt
A table-valued function containing one row for each
    prepared statement associated with an open database connection.
swarmvtab
An experimental module providing on-demand read-only access to
    multiple tables spread across multiple databases, via a single
    virtual table abstraction.
tclvar
Represents the global variables of a 
    TCL Interpreter as an SQL
    table. Used as part of the SQLite test suite.
templatevtab
A template virtual table implementation useful as a starting point
    for developers who want to write their own virtual tables
unionvtab
An experimental module providing on-demand read-only access to
    multiple tables spread across multiple databases, via a single
    virtual table abstraction.
vfsstat
A table-valued function which, in combination with a
    co-packaged VFS shim provides information on the number of
    system calls performed by SQLite.
vtablog
A virtual table that prints diagnostic information
    on stdout when its key methods are invoked.  Intended for
    interactive analysis and debugging of virtual table interfaces.
wholenumber
A virtual table returns all integers between 1 and 4294967295.
zipfile
Represent a
    ZIP Archive
    as an SQL table.  Works for both reading and writing.  Used by
    the CLI to implement the ability to read and write ZIP Archives.


c2: The C language interface to SQLite Version 2
c5: c_interface.html
c4:
Editorial Note:
This document describes SQLite version 2, which was deprecated and
replaced by SQLite3 in 2004.
This document is retained as part of the historical record of SQLite.
Modern programmers should refer to
more up-to-date documentation on SQLite is available elsewhere
on this website.

The C language interface to SQLite Version 2

The SQLite library is designed to be very easy to use from
a C or C++ program.  This document gives an overview of the C/C++
programming interface.

1.0 The Core API

The interface to the SQLite library consists of three core functions,
one opaque data structure, and some constants used as return values.
The core interface is as follows:


typedef struct sqlite sqlite;
#define SQLITE_OK           0   /* Successful result */

sqlite *sqlite_open(const char *dbname, int mode, char **errmsg);

void sqlite_close(sqlite *db);

int sqlite_exec(
  sqlite *db,
  char *sql,
  int (*xCallback)(void*,int,char**,char**),
  void *pArg,
  char **errmsg
);



The above is all you really need to know in order to use SQLite
in your C or C++ programs.  There are other interface functions
available (and described below) but we will begin by describing
the core functions shown above.



1.1 Opening a database

Use the sqlite_open function to open an existing SQLite
database or to create a new SQLite database.  The first argument
is the database name.  The second argument is intended to signal
whether the database is going to be used for reading and writing
or just for reading.  But in the current implementation, the
second argument to sqlite_open is ignored.
The third argument is a pointer to a string pointer.
If the third argument is not NULL and an error occurs
while trying to open the database, then an error message will be
written to memory obtained from malloc() and *errmsg will be made
to point to this error message.  The calling function is responsible
for freeing the memory when it has finished with it.

The name of an SQLite database is the name of a file that will
contain the database.  If the file does not exist, SQLite attempts
to create and initialize it.  If the file is read-only (due to
permission bits or because it is located on read-only media like
a CD-ROM) then SQLite opens the database for reading only.  The
entire SQL database is stored in a single file on the disk.  But
additional temporary files may be created during the execution of
an SQL command in order to store the database rollback journal or
temporary and intermediate results of a query.

The return value of the sqlite_open function is a
pointer to an opaque sqlite structure.  This pointer will
be the first argument to all subsequent SQLite function calls that
deal with the same database.  NULL is returned if the open fails
for any reason.


1.2 Closing the database

To close an SQLite database, call the sqlite_close
function passing it the sqlite structure pointer that was obtained
from a prior call to sqlite_open.
If a transaction is active when the database is closed, the transaction
is rolled back.


1.3 Executing SQL statements

The sqlite_exec function is used to process SQL statements
and queries.  This function requires 5 parameters as follows:


A pointer to the sqlite structure obtained from a prior call
       to sqlite_open.
A zero-terminated string containing the text of one or more
       SQL statements and/or queries to be processed.
A pointer to a callback function which is invoked once for each
       row in the result of a query.  This argument may be NULL, in which
       case no callbacks will ever be invoked.
A pointer that is forwarded to become the first argument
       to the callback function.
A pointer to an error string.  Error messages are written to space
       obtained from malloc() and the error string is made to point to
       the malloced space.  The calling function is responsible for freeing
       this space when it has finished with it.
       This argument may be NULL, in which case error messages are not
       reported back to the calling function.



The callback function is used to receive the results of a query.  A
prototype for the callback function is as follows:


int Callback(void *pArg, int argc, char **argv, char **columnNames){
  return 0;
}



The first argument to the callback is just a copy of the fourth argument
to sqlite_exec  This parameter can be used to pass arbitrary
information through to the callback function from client code.
The second argument is the number of columns in the query result.
The third argument is an array of pointers to strings where each string
is a single column of the result for that record.  Note that the
callback function reports a NULL value in the database as a NULL pointer,
which is very different from an empty string.  If the i-th parameter
is an empty string, we will get:

argv&#91;i]&#91;0] == 0

But if the i-th parameter is NULL we will get:

argv&#91;i] == 0


The names of the columns are contained in first argc
entries of the fourth argument.
If the SHOW_DATATYPES pragma
is on (it is off by default) then
the second argc entries in the 4th argument are the datatypes
for the corresponding columns.


If the 
EMPTY_RESULT_CALLBACKS pragma is set to ON and the result of
a query is an empty set, then the callback is invoked once with the
third parameter (argv) set to 0.  In other words

argv == 0

The second parameter (argc)
and the fourth parameter (columnNames) are still valid
and can be used to determine the number and names of the result
columns if there had been a result.
The default behavior is not to invoke the callback at all if the
result set is empty.


The callback function should normally return 0.  If the callback
function returns non-zero, the query is immediately aborted and 
sqlite_exec will return SQLITE_ABORT.

1.4 Error Codes


The sqlite_exec function normally returns SQLITE_OK.  But
if something goes wrong it can return a different value to indicate
the type of error.  Here is a complete list of the return codes:



#define SQLITE_OK           0   /* Successful result */
#define SQLITE_ERROR        1   /* SQL error or missing database */
#define SQLITE_INTERNAL     2   /* An internal logic error in SQLite */
#define SQLITE_PERM         3   /* Access permission denied */
#define SQLITE_ABORT        4   /* Callback routine requested an abort */
#define SQLITE_BUSY         5   /* The database file is locked */
#define SQLITE_LOCKED       6   /* A table in the database is locked */
#define SQLITE_NOMEM        7   /* A malloc() failed */
#define SQLITE_READONLY     8   /* Attempt to write a readonly database */
#define SQLITE_INTERRUPT    9   /* Operation terminated by sqlite_interrupt() */
#define SQLITE_IOERR       10   /* Some kind of disk I/O error occurred */
#define SQLITE_CORRUPT     11   /* The database disk image is malformed */
#define SQLITE_NOTFOUND    12   /* (Internal Only) Table or record not found */
#define SQLITE_FULL        13   /* Insertion failed because database is full */
#define SQLITE_CANTOPEN    14   /* Unable to open the database file */
#define SQLITE_PROTOCOL    15   /* Database lock protocol error */
#define SQLITE_EMPTY       16   /* (Internal Only) Database table is empty */
#define SQLITE_SCHEMA      17   /* The database schema changed */
#define SQLITE_TOOBIG      18   /* Too much data for one row of a table */
#define SQLITE_CONSTRAINT  19   /* Abort due to constraint violation */
#define SQLITE_MISMATCH    20   /* Data type mismatch */
#define SQLITE_MISUSE      21   /* Library used incorrectly */
#define SQLITE_NOLFS       22   /* Uses OS features not supported on host */
#define SQLITE_AUTH        23   /* Authorization denied */
#define SQLITE_ROW         100  /* sqlite_step() has another row ready */
#define SQLITE_DONE        101  /* sqlite_step() has finished executing */



The meanings of these various return values are as follows:




SQLITE_OK
This value is returned if everything worked and there were no errors.

SQLITE_INTERNAL
This value indicates that an internal consistency check within
the SQLite library failed.  This can only happen if there is a bug in
the SQLite library.  If you ever get an SQLITE_INTERNAL reply from
an sqlite_exec call, please report the problem on the SQLite
mailing list.

SQLITE_ERROR
This return value indicates that there was an error in the SQL
that was passed into the sqlite_exec.

SQLITE_PERM
This return value says that the access permissions on the database
file are such that the file cannot be opened.

SQLITE_ABORT
This value is returned if the callback function returns non-zero.

SQLITE_BUSY
This return code indicates that another program or thread has
the database locked.  SQLite allows two or more threads to read the
database at the same time, but only one thread can have the database
open for writing at the same time.  Locking in SQLite is on the
entire database.

SQLITE_LOCKED
This return code is similar to SQLITE_BUSY in that it indicates
that the database is locked.  But the source of the lock is a recursive
call to sqlite_exec.  This return can only occur if you attempt
to invoke sqlite_exec from within a callback routine of a query
from a prior invocation of sqlite_exec.  Recursive calls to
sqlite_exec are allowed as long as they do
not attempt to write the same table.

SQLITE_NOMEM
This value is returned if a call to malloc fails.

SQLITE_READONLY
This return code indicates that an attempt was made to write to
a database file that is opened for reading only.

SQLITE_INTERRUPT
This value is returned if a call to sqlite_interrupt
interrupts a database operation in progress.

SQLITE_IOERR
This value is returned if the operating system informs SQLite
that it is unable to perform some disk I/O operation.  This could mean
that there is no more space left on the disk.

SQLITE_CORRUPT
This value is returned if SQLite detects that the database it is
working on has become corrupted.  Corruption might occur due to a rogue
process writing to the database file or it might happen due to a
previously undetected logic error in of SQLite. This value is also
returned if a disk I/O error occurs in such a way that SQLite is forced
to leave the database file in a corrupted state.  The latter should only
happen due to a hardware or operating system malfunction.

SQLITE_FULL
This value is returned if an insertion failed because there is
no space left on the disk, or the database is too big to hold any
more information.  The latter case should only occur for databases
that are larger than 2GB in size.

SQLITE_CANTOPEN
This value is returned if the database file could not be opened
for some reason.

SQLITE_PROTOCOL
This value is returned if some other process is messing with
file locks and has violated the file locking protocol that SQLite uses
on its rollback journal files.

SQLITE_SCHEMA
When the database first opened, SQLite reads the database schema
into memory and uses that schema to parse new SQL statements.  If another
process changes the schema, the command currently being processed will
abort because the virtual machine code generated assumed the old
schema.  This is the return code for such cases.  Retrying the
command usually will clear the problem.

SQLITE_TOOBIG
SQLite will not store more than about 1 megabyte of data in a single
row of a single table.  If you attempt to store more than 1 megabyte
in a single row, this is the return code you get.

SQLITE_CONSTRAINT
This constant is returned if the SQL statement would have violated
a database constraint.

SQLITE_MISMATCH
This error occurs when there is an attempt to insert non-integer
data into a column labeled INTEGER PRIMARY KEY.  For most columns, SQLite
ignores the data type and allows any kind of data to be stored.  But
an INTEGER PRIMARY KEY column is only allowed to store integer data.

SQLITE_MISUSE
This error might occur if one or more of the SQLite API routines
is used incorrectly.  Examples of incorrect usage include calling
sqlite_exec after the database has been closed using
sqlite_close or 
calling sqlite_exec with the same
database pointer simultaneously from two separate threads.

SQLITE_NOLFS
This error means that you have attempts to create or access a file
database file that is larger that 2GB on a legacy Unix machine that
lacks large file support.

SQLITE_AUTH
This error indicates that the authorizer callback
has disallowed the SQL you are attempting to execute.

SQLITE_ROW
This is one of the return codes from the
sqlite_step routine which is part of the non-callback API.
It indicates that another row of result data is available.

SQLITE_DONE
This is one of the return codes from the
sqlite_step routine which is part of the non-callback API.
It indicates that the SQL statement has been completely executed and
the sqlite_finalize routine is ready to be called.




2.0 Accessing Data Without Using A Callback Function


The sqlite_exec routine described above used to be the only
way to retrieve data from an SQLite database.  But many programmers found
it inconvenient to use a callback function to obtain results.  So beginning
with SQLite version 2.7.7, a second access interface is available that
does not use callbacks.



The new interface uses three separate functions to replace the single
sqlite_exec function.



typedef struct sqlite_vm sqlite_vm;

int sqlite_compile(
  sqlite *db,              /* The open database */
  const char *zSql,        /* SQL statement to be compiled */
  const char **pzTail,     /* OUT: uncompiled tail of zSql */
  sqlite_vm **ppVm,        /* OUT: the virtual machine to execute zSql */
  char **pzErrmsg          /* OUT: Error message. */
);

int sqlite_step(
  sqlite_vm *pVm,          /* The virtual machine to execute */
  int *pN,                 /* OUT: Number of columns in result */
  const char ***pazValue,  /* OUT: Column data */
  const char ***pazColName /* OUT: Column names and datatypes */
);

int sqlite_finalize(
  sqlite_vm *pVm,          /* The virtual machine to be finalized */
  char **pzErrMsg          /* OUT: Error message */
);



The strategy is to compile a single SQL statement using
sqlite_compile then invoke sqlite_step multiple times,
once for each row of output, and finally call sqlite_finalize
to clean up after the SQL has finished execution.


2.1 Compiling An SQL Statement Into A Virtual Machine


The sqlite_compile "compiles" a single SQL statement (specified
by the second parameter) and generates a virtual machine that is able
to execute that statement.  
As with must interface routines, the first parameter must be a pointer
to an sqlite structure that was obtained from a prior call to
sqlite_open.


A pointer to the virtual machine is stored in a pointer which is passed
in as the 4th parameter.
Space to hold the virtual machine is dynamically allocated.  To avoid
a memory leak, the calling function must invoke
sqlite_finalize on the virtual machine after it has finished
with it.
The 4th parameter may be set to NULL if an error is encountered during
compilation.



If any errors are encountered during compilation, an error message is
written into memory obtained from malloc and the 5th parameter
is made to point to that memory.  If the 5th parameter is NULL, then
no error message is generated.  If the 5th parameter is not NULL, then
the calling function should dispose of the memory containing the error
message by calling sqlite_freemem.



If the 2nd parameter actually contains two or more statements of SQL,
only the first statement is compiled.  (This is different from the
behavior of sqlite_exec which executes all SQL statements
in its input string.)  The 3rd parameter to sqlite_compile
is made to point to the first character beyond the end of the first
statement of SQL in the input.  If the 2nd parameter contains only
a single SQL statement, then the 3rd parameter will be made to point
to the '\000' terminator at the end of the 2nd parameter.



On success, sqlite_compile returns SQLITE_OK.
Otherwise and error code is returned.


2.2 Step-By-Step Execution Of An SQL Statement


After a virtual machine has been generated using sqlite_compile
it is executed by one or more calls to sqlite_step.  Each
invocation of sqlite_step, except the last one,
returns a single row of the result.
The number of columns in  the result is stored in the integer that
the 2nd parameter points to.
The pointer specified by the 3rd parameter is made to point
to an array of pointers to column values.
The pointer in the 4th parameter is made to point to an array
of pointers to column names and datatypes.
The 2nd through 4th parameters to sqlite_step convey the
same information as the 2nd through 4th parameters of the
callback routine when using
the sqlite_exec interface.  Except, with sqlite_step
the column datatype information is always included in the in the
4th parameter regardless of whether or not the
SHOW_DATATYPES pragma
is on or off.



Each invocation of sqlite_step returns an integer code that
indicates what happened during that step.  This code may be
SQLITE_BUSY, SQLITE_ROW, SQLITE_DONE, SQLITE_ERROR, or
SQLITE_MISUSE.



If the virtual machine is unable to open the database file because
it is locked by another thread or process, sqlite_step
will return SQLITE_BUSY.  The calling function should do some other
activity, or sleep, for a short amount of time to give the lock a
chance to clear, then invoke sqlite_step again.  This can
be repeated as many times as desired.



Whenever another row of result data is available,
sqlite_step will return SQLITE_ROW.  The row data is
stored in an array of pointers to strings and the 2nd parameter
is made to point to this array.



When all processing is complete, sqlite_step will return
either SQLITE_DONE or SQLITE_ERROR.  SQLITE_DONE indicates that the
statement completed successfully and SQLITE_ERROR indicates that there
was a run-time error.  (The details of the error are obtained from
sqlite_finalize.)  It is a misuse of the library to attempt
to call sqlite_step again after it has returned SQLITE_DONE
or SQLITE_ERROR.



When sqlite_step returns SQLITE_DONE or SQLITE_ERROR,
the *pN and *pazColName values are set to the number of columns
in the result set and to the names of the columns, just as they
are for an SQLITE_ROW return.  This allows the calling code to
find the number of result columns and the column names and datatypes
even if the result set is empty.  The *pazValue parameter is always
set to NULL when the return codes is SQLITE_DONE or SQLITE_ERROR.
If the SQL being executed is a statement that does not
return a result (such as an INSERT or an UPDATE) then *pN will
be set to zero and *pazColName will be set to NULL.



If you abuse the library by trying to call sqlite_step
inappropriately it will attempt return SQLITE_MISUSE.
This can happen if you call sqlite_step() on the same virtual machine
at the same
time from two or more threads or if you call sqlite_step()
again after it returned SQLITE_DONE or SQLITE_ERROR or if you
pass in an invalid virtual machine pointer to sqlite_step().
You should not depend on the SQLITE_MISUSE return code to indicate
an error.  It is possible that a misuse of the interface will go
undetected and result in a program crash.  The SQLITE_MISUSE is
intended as a debugging aid only - to help you detect incorrect
usage prior to a mishap.  The misuse detection logic is not guaranteed
to work in every case.


2.3 Deleting A Virtual Machine


Every virtual machine that sqlite_compile creates should
eventually be handed to sqlite_finalize.  The sqlite_finalize()
procedure deallocates the memory and other resources that the virtual
machine uses.  Failure to call sqlite_finalize() will result in 
resource leaks in your program.



The sqlite_finalize routine also returns the result code
that indicates success or failure of the SQL operation that the
virtual machine carried out.
The value returned by sqlite_finalize() will be the same as would
have been returned had the same SQL been executed by sqlite_exec.
The error message returned will also be the same.



It is acceptable to call sqlite_finalize on a virtual machine
before sqlite_step has returned SQLITE_DONE.  Doing so has
the effect of interrupting the operation in progress.  Partially completed
changes will be rolled back and the database will be restored to its
original state (unless an alternative recovery algorithm is selected using
an ON CONFLICT clause in the SQL being executed.)  The effect is the
same as if a callback function of sqlite_exec had returned
non-zero.



It is also acceptable to call sqlite_finalize on a virtual machine
that has never been passed to sqlite_step even once.


3.0 The Extended API

Only the three core routines described in section 1.0 are required to use
SQLite.  But there are many other functions that provide 
useful interfaces.  These extended routines are as follows:



int sqlite_last_insert_rowid(sqlite*);

int sqlite_changes(sqlite*);

int sqlite_get_table(
  sqlite*,
  char *sql,
  char ***result,
  int *nrow,
  int *ncolumn,
  char **errmsg
);

void sqlite_free_table(char**);

void sqlite_interrupt(sqlite*);

int sqlite_complete(const char *sql);

void sqlite_busy_handler(sqlite*, int (*)(void*,const char*,int), void*);

void sqlite_busy_timeout(sqlite*, int ms);

const char sqlite_version&#91;];

const char sqlite_encoding&#91;];

int sqlite_exec_printf(
  sqlite*,
  char *sql,
  int (*)(void*,int,char**,char**),
  void*,
  char **errmsg,
  ...
);

int sqlite_exec_vprintf(
  sqlite*,
  char *sql,
  int (*)(void*,int,char**,char**),
  void*,
  char **errmsg,
  va_list
);

int sqlite_get_table_printf(
  sqlite*,
  char *sql,
  char ***result,
  int *nrow,
  int *ncolumn,
  char **errmsg,
  ...
);

int sqlite_get_table_vprintf(
  sqlite*,
  char *sql,
  char ***result,
  int *nrow,
  int *ncolumn,
  char **errmsg,
  va_list
);

char *sqlite_mprintf(const char *zFormat, ...);

char *sqlite_vmprintf(const char *zFormat, va_list);

void sqlite_freemem(char*);

void sqlite_progress_handler(sqlite*, int, int (*)(void*), void*);



All of the above definitions are included in the "sqlite.h"
header file that comes in the source tree.

3.1 The ROWID of the most recent insert

Every row of an SQLite table has a unique integer key.  If the
table has a column labeled INTEGER PRIMARY KEY, then that column
serves as the key.  If there is no INTEGER PRIMARY KEY column then
the key is a unique integer.  The key for a row can be accessed in
a SELECT statement or used in a WHERE or ORDER BY clause using any
of the names "ROWID", "OID", or "_ROWID_".

When you do an insert into a table that does not have an INTEGER PRIMARY
KEY column, or if the table does have an INTEGER PRIMARY KEY but the value
for that column is not specified in the VALUES clause of the insert, then
the key is automatically generated.  You can find the value of the key
for the most recent INSERT statement using the
sqlite_last_insert_rowid API function.

3.2 The number of rows that changed

The sqlite_changes API function returns the number of rows
that have been inserted, deleted, or modified since the database was
last quiescent.  A "quiescent" database is one in which there are
no outstanding calls to sqlite_exec and no VMs created by
sqlite_compile that have not been finalized by sqlite_finalize.
In common usage, sqlite_changes returns the number
of rows inserted, deleted, or modified by the most recent sqlite_exec
call or since the most recent sqlite_compile.  But if you have
nested calls to sqlite_exec (that is, if the callback routine
of one sqlite_exec invokes another sqlite_exec) or if
you invoke sqlite_compile to create a new VM while there is
still another VM in existence, then
the meaning of the number returned by sqlite_changes is more
complex.
The number reported includes any changes
that were later undone by a ROLLBACK or ABORT.  But rows that are
deleted because of a DROP TABLE are not counted.

SQLite implements the command "DELETE FROM table" (without
a WHERE clause) by dropping the table then recreating it.  
This is much faster than deleting the elements of the table individually.
But it also means that the value returned from sqlite_changes
will be zero regardless of the number of elements that were originally
in the table.  If an accurate count of the number of elements deleted
is necessary, use "DELETE FROM table WHERE 1" instead.

3.3 Querying into memory obtained from malloc()

The sqlite_get_table function is a wrapper around
sqlite_exec that collects all the information from successive
callbacks and writes it into memory obtained from malloc().  This
is a convenience function that allows the application to get the
entire result of a database query with a single function call.

The main result from sqlite_get_table is an array of pointers
to strings.  There is one element in this array for each column of
each row in the result.  NULL results are represented by a NULL
pointer. In addition to the regular data, there is an added row at the 
beginning of the array that contains the name of each column of the
result.

As an example, consider the following query:


SELECT employee_name, login, host FROM users WHERE login LIKE 'd%';


This query will return the name, login and host computer name
for every employee whose login begins with the letter "d".  If this
query is submitted to sqlite_get_table the result might
look like this:


nrow = 2
ncolumn = 3
result&#91;0] = "employee_name"
result&#91;1] = "login"
result&#91;2] = "host"
result&#91;3] = "dummy"
result&#91;4] = "No such user"
result&#91;5] = 0
result&#91;6] = "D. Richard Hipp"
result&#91;7] = "drh"
result&#91;8] = "zadok"


Notice that the "host" value for the "dummy" record is NULL so
the result&#91;] array contains a NULL pointer at that slot.

If the result set of a query is empty, then by default
sqlite_get_table will set nrow to 0 and leave its
result parameter is set to NULL.  But if the EMPTY_RESULT_CALLBACKS
pragma is ON then the result parameter is initialized to the names
of the columns only.  For example, consider this query which has
an empty result set:


SELECT employee_name, login, host FROM users WHERE employee_name IS NULL;



The default behavior gives this results:



nrow = 0
ncolumn = 0
result = 0



But if the EMPTY_RESULT_CALLBACKS pragma is ON, then the following
is returned:



nrow = 0
ncolumn = 3
result&#91;0] = "employee_name"
result&#91;1] = "login"
result&#91;2] = "host"


Memory to hold the information returned by sqlite_get_table
is obtained from malloc().  But the calling function should not try
to free this information directly.  Instead, pass the complete table
to sqlite_free_table when the table is no longer needed.
It is safe to call sqlite_free_table with a NULL pointer such
as would be returned if the result set is empty.

The sqlite_get_table routine returns the same integer
result code as sqlite_exec.

3.4 Interrupting an SQLite operation

The sqlite_interrupt function can be called from a
different thread or from a signal handler to cause the current database
operation to exit at its first opportunity.  When this happens,
the sqlite_exec routine (or the equivalent) that started
the database operation will return SQLITE_INTERRUPT.

3.5 Testing for a complete SQL statement

The next interface routine to SQLite is a convenience function used
to test whether or not a string forms a complete SQL statement.
If the sqlite_complete function returns true when its input
is a string, then the argument forms a complete SQL statement.
There are no guarantees that the syntax of that statement is correct,
but we at least know the statement is complete.  If sqlite_complete
returns false, then more text is required to complete the SQL statement.

For the purpose of the sqlite_complete function, an SQL
statement is complete if it ends in a semicolon.

The sqlite command-line utility uses the sqlite_complete
function to know when it needs to call sqlite_exec.  After each
line of input is received, sqlite calls sqlite_complete
on all input in its buffer.  If sqlite_complete returns true, 
then sqlite_exec is called and the input buffer is reset.  If
sqlite_complete returns false, then the prompt is changed to
the continuation prompt and another line of text is read and added to
the input buffer.

3.6 Library version string

The SQLite library exports the string constant named
sqlite_version which contains the version number of the
library.  The header file contains a macro SQLITE_VERSION
with the same information.  If desired, a program can compare
the SQLITE_VERSION macro against the sqlite_version
string constant to verify that the version number of the
header file and the library match. 

3.7 Library character encoding

By default, SQLite assumes that all data uses a fixed-size
8-bit character (iso8859).  But if you give the --enable-utf8 option
to the configure script, then the library assumes UTF-8 variable
sized characters.  This makes a difference for the LIKE and GLOB
operators and the LENGTH() and SUBSTR() functions.  The static
string sqlite_encoding will be set to either "UTF-8" or
"iso8859" to indicate how the library was compiled.  In addition,
the sqlite.h header file will define one of the
macros SQLITE_UTF8 or SQLITE_ISO8859, as appropriate.

Note that the character encoding mechanism used by SQLite cannot
be changed at run-time.  This is a compile-time option only.  The
sqlite_encoding character string just tells you how the library
was compiled.

3.8 Changing the library's response to locked files

The sqlite_busy_handler procedure can be used to register
a busy callback with an open SQLite database.  The busy callback will
be invoked whenever SQLite tries to access a database that is locked.
The callback will typically do some other useful work, or perhaps sleep,
in order to give the lock a chance to clear.  If the callback returns
non-zero, then SQLite tries again to access the database and the cycle
repeats.  If the callback returns zero, then SQLite aborts the current
operation and returns SQLITE_BUSY.

The arguments to sqlite_busy_handler are the opaque
structure returned from sqlite_open, a pointer to the busy
callback function, and a generic pointer that will be passed as
the first argument to the busy callback.  When SQLite invokes the
busy callback, it sends it three arguments:  the generic pointer
that was passed in as the third argument to sqlite_busy_handler,
the name of the database table or index that the library is trying
to access, and the number of times that the library has attempted to
access the database table or index.

For the common case where we want the busy callback to sleep,
the SQLite library provides a convenience routine sqlite_busy_timeout.
The first argument to sqlite_busy_timeout is a pointer to
an open SQLite database and the second argument is a number of milliseconds.
After sqlite_busy_timeout has been executed, the SQLite library
will wait for the lock to clear for at least the number of milliseconds 
specified before it returns SQLITE_BUSY.  Specifying zero milliseconds for
the timeout restores the default behavior.

3.9 Using the _printf() wrapper functions

The four utility functions



sqlite_exec_printf()
sqlite_exec_vprintf()
sqlite_get_table_printf()
sqlite_get_table_vprintf()



implement the same query functionality as sqlite_exec
and sqlite_get_table.  But instead of taking a complete
SQL statement as their second argument, the four _printf
routines take a printf-style format string.  The SQL statement to
be executed is generated from this format string and from whatever
additional arguments are attached to the end of the function call.

There are two advantages to using the SQLite printf
functions instead of sprintf.  First of all, with the
SQLite printf routines, there is never a danger of overflowing a
static buffer as there is with sprintf.  The SQLite
printf routines automatically allocate (and later frees)
as much memory as is 
necessary to hold the SQL statements generated.

The second advantage the SQLite printf routines have over
sprintf are two new formatting options specifically designed
to support string literals in SQL.  Within the format string,
the %q formatting option works very much like %s in that it
reads a null-terminated string from the argument list and inserts
it into the result.  But %q translates the inserted string by
making two copies of every single-quote (') character in the
substituted string.  This has the effect of escaping the end-of-string
meaning of single-quote within a string literal. The %Q formatting
option works similar; it translates the single-quotes like %q and
additionally encloses the resulting string in single-quotes.
If the argument for the %Q formatting options is a NULL pointer,
the resulting string is NULL without single quotes.


Consider an example.  Suppose you are trying to insert a string
value into a database table where the string value was obtained from
user input.  Suppose the string to be inserted is stored in a variable
named zString.  The code to do the insertion might look like this:


sqlite_exec_printf(db,
  "INSERT INTO table1 VALUES('%s')",
  0, 0, 0, zString);


If the zString variable holds text like "Hello", then this statement
will work just fine.  But suppose the user enters a string like 
"Hi y'all!".  The SQL statement generated reads as follows:


INSERT INTO table1 VALUES('Hi y'all')


This is not valid SQL because of the apostrophe in the word "y'all".
But if the %q formatting option is used instead of %s, like this:


sqlite_exec_printf(db,
  "INSERT INTO table1 VALUES('%q')",
  0, 0, 0, zString);


Then the generated SQL will look like the following:


INSERT INTO table1 VALUES('Hi y''all')


Here the apostrophe has been escaped and the SQL statement is well-formed.
When generating SQL on-the-fly from data that might contain a
single-quote character ('), it is always a good idea to use the
SQLite printf routines and the %q formatting option instead of sprintf.


If the %Q formatting option is used instead of %q, like this:


sqlite_exec_printf(db,
  "INSERT INTO table1 VALUES(%Q)",
  0, 0, 0, zString);


Then the generated SQL will look like the following:


INSERT INTO table1 VALUES('Hi y''all')


If the value of the zString variable is NULL, the generated SQL
will look like the following:


INSERT INTO table1 VALUES(NULL)


All of the _printf() routines above are built around the following
two functions:


char *sqlite_mprintf(const char *zFormat, ...);
char *sqlite_vmprintf(const char *zFormat, va_list);


The sqlite_mprintf() routine works like the standard library
sprintf() except that it writes its results into memory obtained
from malloc() and returns a pointer to the malloced buffer.  
sqlite_mprintf() also understands the %q and %Q extensions described
above.  The sqlite_vmprintf() is a varargs version of the same
routine.  The string pointer that these routines return should be freed
by passing it to sqlite_freemem().


3.10 Performing background jobs during large queries

The sqlite_progress_handler() routine can be used to register a
callback routine with an SQLite database to be invoked periodically during long
running calls to sqlite_exec(), sqlite_step() and the various
wrapper functions.


The callback is invoked every N virtual machine operations, where N is
supplied as the second argument to sqlite_progress_handler(). The third
and fourth arguments to sqlite_progress_handler() are a pointer to the
routine to be invoked and a void pointer to be passed as the first argument to
it.


The time taken to execute each virtual machine operation can vary based on
many factors.  A typical value for a 1 GHz PC is between half and three million
per second but may be much higher or lower, depending on the query.  As such it
is difficult to schedule background operations based on virtual machine
operations. Instead, it is recommended that a callback be scheduled relatively
frequently (say every 1000 instructions) and external timer routines used to
determine whether or not background jobs need to be run.  



4.0 Adding New SQL Functions

Beginning with version 2.4.0, SQLite allows the SQL language to be
extended with new functions implemented as C code.  The following interface
is used:



typedef struct sqlite_func sqlite_func;

int sqlite_create_function(
  sqlite *db,
  const char *zName,
  int nArg,
  void (*xFunc)(sqlite_func*,int,const char**),
  void *pUserData
);
int sqlite_create_aggregate(
  sqlite *db,
  const char *zName,
  int nArg,
  void (*xStep)(sqlite_func*,int,const char**),
  void (*xFinalize)(sqlite_func*),
  void *pUserData
);

char *sqlite_set_result_string(sqlite_func*,const char*,int);
void sqlite_set_result_int(sqlite_func*,int);
void sqlite_set_result_double(sqlite_func*,double);
void sqlite_set_result_error(sqlite_func*,const char*,int);

void *sqlite_user_data(sqlite_func*);
void *sqlite_aggregate_context(sqlite_func*, int nBytes);
int sqlite_aggregate_count(sqlite_func*);



The sqlite_create_function() interface is used to create 
regular functions and sqlite_create_aggregate() is used to
create new aggregate functions.  In both cases, the db
parameter is an open SQLite database on which the functions should
be registered, zName is the name of the new function,
nArg is the number of arguments, and pUserData is
a pointer which is passed through unchanged to the C implementation
of the function.  Both routines return 0 on success and non-zero
if there are any errors.



The length of a function name may not exceed 255 characters.
Any attempt to create a function whose name exceeds 255 characters
in length will result in an error.



For regular functions, the xFunc callback is invoked once
for each function call.  The implementation of xFunc should call
one of the sqlite_set_result_... interfaces to return its
result.  The sqlite_user_data() routine can be used to
retrieve the pUserData pointer that was passed in when the
function was registered.



For aggregate functions, the xStep callback is invoked once
for each row in the result and then xFinalize is invoked at the
end to compute a final answer.  The xStep routine can use the
sqlite_aggregate_context() interface to allocate memory that
will be unique to that particular instance of the SQL function.
This memory will be automatically deleted after xFinalize is called.
The sqlite_aggregate_count() routine can be used to find out
how many rows of data were passed to the aggregate.  The xFinalize
callback should invoke one of the sqlite_set_result_...
interfaces to set the final result of the aggregate.



SQLite now implements all of its built-in functions using this
interface.  For additional information and examples on how to create
new SQL functions, review the SQLite source code in the file
func.c.


5.0 Multi-Threading And SQLite


If SQLite is compiled with the THREADSAFE preprocessor macro set to 1,
then it is safe to use SQLite from two or more threads of the same process
at the same time.  But each thread should have its own sqlite*
pointer returned from sqlite_open.  It is never safe for two
or more threads to access the same sqlite* pointer at the same time.



In precompiled SQLite libraries available on the website, the Unix
versions are compiled with THREADSAFE turned off but the Windows
versions are compiled with THREADSAFE turned on.  If you need something
different that this you will have to recompile.



Under Unix, an sqlite* pointer should not be carried across a
fork() system call into the child process.  The child process
should open its own copy of the database after the fork().


6.0 Usage Examples

For examples of how the SQLite C/C++ interface can be used,
refer to the source code for the sqlite program in the
file src/shell.c
of the source tree.
Additional information about sqlite is available at
cli.html.
See also the sources to the Tcl interface for SQLite in
the source file 
src/tclsqlite.c.


c2: Powersafe Overwrite
c5: psow.html
c4:
Powersafe Overwrite

"Powersafe overwrite" is a term used by the SQLite team to describe
a behavior of some filesystems and disk-controllers related to
data preservation during a power loss.  Powersafe overwrite
is a boolean property: either the storage system has it or it does not.

We say that a system has the powersafe overwrite property if the following
statement is true:


  When an application writes a range of bytes in a file, no
  bytes outside of that range will change, even if the write occurs
  just before a crash or power failure.


The powersafe overwrite property says nothing about the state of the
bytes that were written.  Those bytes might contain their old values,
their new values, random values, or some combination of these.  The powersafe
overwrite property merely states that writes cannot change bytes outside
of the range of bytes written.

In other words, powersafe overwrite means that there is no "collateral
damage" when a power loss occurs while writing.  Only those bytes actually
being written might be damaged.

In practical terms, what the powersafe write property means is that when
the disk controller detects an impending power loss, it finishes writing
whatever sector it is working on prior to parking the heads.  It means that
individual sector writes will complete once started, even if
there is a power loss.

Consider what would happen if disk sector writes are interrupted
by a power loss.  If an application writes two or three bytes in the middle
of some file, the operating system will implement this by first reading
the entire sector containing those bytes, making the change to the
sector in memory, then writing the entire sector back to the disk.  If a power
loss occurs during the writeback and the sector was not completely written,
then on the next read after reboot, error correcting codes
in the sector will probably detect irreparable damage and the disk 
controller will read out the sector as all zeros or all ones.  Thus
values will have changed outside of the range of the two or three bytes 
that were written at the application level - a violation of the powersafe
overwrite property.

SQLite Assumptions About Powersafe Overwrite

All versions of SQLite up to and including version 3.7.9
(2011-11-01) assume that
the filesystem does not provide powersafe overwrite.  SQLite 
has traditionally assumed that when any one byte of a file changes, all
other bytes within the same sector of that byte have the potential of
being corrupted on a power loss.  When writing, SQLite has made sure
to journal all bytes in the same sector of any modifications
and it pads journal files out to the next sector boundary so that
subsequent appends to that journal cannot damage prior records.
SQLite understands the sector size to be the value returned by the
xSectorSize method in the VFS.  The SQLite team has often referred
to the value returned by xSectorSize as the "blast radius" of a write,
since it expresses the range of bytes that might be damaged if a power
loss occurs during the write.
The default VFSes for unix and windows have always returned 512 as 
the sector size (or blast radius) for all versions of SQLite up to
and including version 3.7.9.

Newer disk drives have begun using 4096 byte sectors however.  Beginning
with SQLite version 3.7.10 (2012-01-16), 
the SQLite development team experimented with 
changes xSectorSize to report 4096 bytes as the blast radius.
This had the effect of increasing write overhead on
many databases.  For a database with a PRAGMA page_size of 1024
(a very common choice) making a change to a single page in the database
now requires SQLite to backup three other adjacent pages to the rollback
journal, whereas formerly it only had to backup the one page that was
changing.  In WAL mode, each transaction had to be padded out to the
next 4096-byte boundary in the WAL file, rather than the next 512-byte
boundary, resulting in thousands of extra bytes being written
per transaction.

The extra write overhead prompted a reexamination of assumptions about
powersafe overwrite.  With modern disk drives, the capacity has become
so large and the data density so great that a single sector is very
small and writing a single sector takes very little time.  We know that
disk drives can detect an impending power loss and continue
to operate for some small amount of time on residual energy because those
drives are able to park their heads before spinning down.  And
so if an impending power loss is detectable by the disk controller, it
seems reasonable that the controller will finish writing
whatever sector it is current working on when the imminent power loss 
is first detected, prior to parking the heads, as long as doing so
does not take too long, which it should not with
small and dense sectors.  Hence it seems reasonable
to assume powersafe overwrite for modern disks.  Indeed, BerkeleyDB has
made this assumption for decades, we are told.  Caution is advised
though. As Roger Binns noted on the SQLite developers mailing list:
"'poorly written' should be the main assumption about drive firmware."



Torn Pages

A torn page occurs when a database page is larger than a disk sector,
the database page is written to disk, but a power loss occurs prior to
all sectors of the database page being written.  Then, upon recovery, part of
the database page will have the old content while some other parts of the
page will have the new content.  Some database engines assume that 
page writes are atomic and hence a torn page is an unrecoverable error.


SQLite never assumes that database page writes are atomic,
regardless of the PSOW setting.(1)
And hence SQLite is always able to automatically recover from torn pages
induced by a crash.  Enabling PSOW does not decrease SQLite's ability
to recover from a torn page.

Changes In SQLite Version 3.7.10

The VFS for SQLite version 3.7.10 (2012-01-16)
adds a new device characteristic 
named SQLITE_IOCAP_POWERSAFE_OVERWRITE.  Database files that report this
characteristic are assumed to reside on storage systems that have the
powersafe overwrite property.
The default unix and windows VFSes now report
SQLITE_IOCAP_POWERSAFE_OVERWRITE if SQLite is compiled with
-DSQLITE_POWERSAFE_OVERWRITE=1 or they
make the legacy assumption that storage does not have the powersafe
overwrite property if compiled with
-DSQLITE_POWERSAFE_OVERWRITE=0.
For now, the default is for powersafe overwrite to be turned on, though
we may revisit this in the future and default it off.

The powersafe overwrite property for individual databases can be
specified as the database is opened using the "psow" query parameter
with a URI filename.  For example, to always assume powersafe
overwrite for a file (perhaps to ensure maximum write performance), 
open it as


   file:somefile.db?psow=1


Or to be extra safe with a database and to force SQLite to assume the
database lacks powersafe overwrite, open it using


   file:somefile.db?psow=0


There is also a new SQLITE_FCNTL_POWERSAFE_OVERWRITE opcode for
the sqlite3_file_control() that allows
an application to query the powersafe overwrite property for a database
file.


Notes:

SQLite never assumes atomic page writes in its default configurations.
But a custom VFS can set one of the 
SQLITE_IOCAP_ATOMIC bits in the result of the xDeviceCharacteristic()
method and then SQLite will assume that page writes are atomic.  The
application must supply a custom VFS to accomplish this, however, since
none of the standard VFSes will ever set any of the atomic bits in the
xDeviceCharacteristics() vector.


