# page_content

c2: Write-Ahead Logging
c3: 7. Implementation Of Shared-Memory For The WAL-Index
c5: wal.html#implementation_of_shared_memory_for_the_wal_index
c4:
The wal-index is implemented using an ordinary file that is
mmapped for robustness.  Early (pre-release) implementations of WAL mode
stored the wal-index in volatile shared-memory, such as files created in
/dev/shm on Linux or /tmp on other unix systems.  The problem
with that approach is that processes with a different root directory
(changed via chroot)
will see different files and hence use different shared memory areas,
leading to database corruption.  Other methods for creating nameless
shared memory blocks are not portable across the various flavors of
unix.  And we could not find any method to create nameless shared
memory blocks on windows.  The only way we have found to guarantee
that all processes accessing the same database file use the same shared
memory is to create the shared memory by mmapping a file in the same
directory as the database itself.

Using an ordinary disk file to provide shared memory has the 
disadvantage that it might actually do unnecessary disk I/O by
writing the shared memory to disk.  However, the developers do not
think this is a major concern since the wal-index rarely exceeds
32 KiB in size and is never synced.  Furthermore, the wal-index 
backing file is deleted when the last database connection disconnects,
which often prevents any real disk I/O from ever happening.

Specialized applications for which the default implementation of
shared memory is unacceptable can devise alternative methods via a
custom VFS.  
For example, if it is known that a particular database
will only be accessed by threads within a single process, the wal-index
can be implemented using heap memory instead of true shared memory.


c2: Write-Ahead Logging
c3: 8. Use of WAL Without Shared-Memory
c5: wal.html#use_of_wal_without_shared_memory
c4:
Beginning in SQLite version 3.7.4 (2010-12-07), 
WAL databases can be created, read, and
written even if shared memory is unavailable as long as the
locking_mode is set to EXCLUSIVE before the first attempted access.
In other words, a process can interact with
a WAL database without using shared memory if that
process is guaranteed to be the only process accessing the database.
This feature allows WAL databases to be created, read, and written
by legacy VFSes that lack the "version 2" shared-memory
methods xShmMap, xShmLock, xShmBarrier, and xShmUnmap on the
sqlite3_io_methods object.

If EXCLUSIVE locking mode
is set prior to the first WAL-mode 
database access, then SQLite never attempts to call any of the
shared-memory methods and hence no shared-memory
wal-index is ever created.
In that case, the database connection remains in EXCLUSIVE mode
as long as the journal mode is WAL; attempts to change the locking
mode using "PRAGMA locking_mode=NORMAL;" are no-ops.
The only way to change out of EXCLUSIVE locking mode is to first
change out of WAL journal mode.

If NORMAL locking mode is in effect for the first WAL-mode database
access, then the shared-memory wal-index is created.  This means that the
underlying VFS must support the "version 2" shared-memory.
If the VFS does not support shared-memory methods, then the attempt to
open a database that is already in WAL mode, or the attempt convert a
database into WAL mode, will fail.
As long as exactly one connection is using a shared-memory wal-index, 
the locking mode can be changed freely between NORMAL and EXCLUSIVE.  
It is only when the shared-memory wal-index is omitted, when the locking 
mode is EXCLUSIVE prior to the first WAL-mode database access, that the 
locking mode is stuck in EXCLUSIVE.


c2: Write-Ahead Logging
c3: 9. Sometimes Queries Return SQLITE_BUSY In WAL Mode
c5: wal.html#sometimes_queries_return_sqlite_busy_in_wal_mode
c4:
The second advantage of WAL-mode is that
writers do not block readers and readers to do not block writers.
This is mostly true.
But there are some obscure cases where a query against a WAL-mode
database can return SQLITE_BUSY, so applications should be prepared
for that happenstance.

Cases where a query against a WAL-mode database can return SQLITE_BUSY
include the following:


If another database connection has the database mode open
in exclusive locking mode then all queries against the
database will return SQLITE_BUSY.  Both Chrome and Firefox open their
database files in exclusive locking mode, so attempts to read Chrome or
Firefox databases while the applications are running will run into this
problem, for example.


When the last connection to a particular database is closing, that
connection will acquire an exclusive lock for a short time while it
cleans up the WAL and shared-memory files.  If a second database tries
to open and query the database while the first connection
is still in the middle
of its cleanup process, the second connection might get an SQLITE_BUSY
error.


If the last connection to a database crashed, then the first new
connection to open the database will start a recovery process.  An
exclusive lock is held during recovery.  So if a third database connection
tries to jump in and query while the second connection is running recovery,
the third connection will get an SQLITE_BUSY error.


c2: Write-Ahead Logging
c3: 10. Backwards Compatibility
c5: wal.html#backwards_compatibility
c4:
The database file format is unchanged for WAL mode.  However, the
WAL file and the wal-index are new concepts and so older versions of 
SQLite will not know
how to recover a crashed SQLite database that was operating in WAL mode
when the crash occurred.
To prevent older versions of SQLite (prior to version 3.7.0, 2010-07-22)
from trying to recover
a WAL-mode database (and making matters worse) the database file format
version numbers (bytes 18 and 19 in the database header)
are increased from 1 to 2 in WAL mode.
Thus, if an older version of SQLite attempts to connect to an SQLite
database that is operating in WAL mode, it will report an error along
the lines of "file is encrypted or is not a database".

One can explicitly change out of WAL mode using a pragma such as
this:


PRAGMA journal_mode=DELETE;



Deliberately changing out of WAL mode changes the database file format
version numbers back to 1 so that older versions of SQLite can once again 
access the database file.


c2: Dynamic Memory Allocation In SQLite
c3: Overview
c5: malloc.html#overview
c4:
SQLite uses dynamic memory allocation to obtain
memory for storing various objects
(ex: database connections and prepared statements) and to build
a memory cache of the database file and to hold the results of queries.
Much effort has gone into making the dynamic memory allocation subsystem
of SQLite reliable, predictable, robust, secure, and efficient.

This document provides an overview of dynamic memory allocation within 
SQLite.  The target audience is software engineers who are tuning their
use of SQLite for peak performance in demanding environments.
Nothing in this document is required knowledge for using SQLite.  The
default settings and configuration for SQLite will work well in most
applications.  However, the information contained in this document may
be useful to engineers who are tuning SQLite to comply with special
requirements or to run under unusual circumstances.


c2: Dynamic Memory Allocation In SQLite
c3: 1. Features
c5: malloc.html#_features
c4:
The SQLite core and its memory allocation subsystem provides the 
following capabilities:



Robust against allocation failures.
If a memory allocation ever fails (that is to say, 
if malloc() or realloc() ever return NULL)
then SQLite will recover gracefully.   SQLite will first attempt
to free memory from unpinned cache pages then retry the allocation
request.  
Failing that, SQLite will either stop what
it is doing and return the
SQLITE_NOMEM error code back up to the application or it will
make do without the requested memory.



No memory leaks.
The application is responsible for destroying any objects it allocates.
(For example, the application must use sqlite3_finalize() on 
every prepared statement and sqlite3_close() on every 
database connection.)   But as long as
the application cooperates, SQLite will never leak memory.  This is
true even in the face of memory allocation failures or other system
errors.



Memory usage limits.
The sqlite3_soft_heap_limit64() mechanism allows the application to
set a memory usage limit that SQLite strives to stay below.  SQLite
will attempt to reuse memory from its caches rather than allocating new
memory as it approaches the soft limit.



Zero-malloc option.
The application can optionally provide SQLite with several buffers of bulk memory
at startup and SQLite will then use those provided buffers for all of
its memory allocation needs and never call system malloc() or free().



Application-supplied memory allocators.
The application can provide SQLite with pointers to alternative 
memory allocators at start-time.  The alternative memory allocator
will be used in place of system malloc() and free().



Proof against breakdown and fragmentation.
SQLite can be configured so that, subject to certain usage constraints
detailed below, it is guaranteed to never fail a memory allocation
or fragment the heap.
This property is important to long-running, high-reliability
embedded systems where a memory allocation error could contribute
to an overall system failure.



Memory usage statistics.
Applications can see how much memory they are using and detect when
memory usage is approaching or exceeding design boundaries.




Plays well with memory debuggers.
Memory allocation in SQLite is structured so that standard
third-party memory debuggers (such as dmalloc or 
valgrind) can be used to verify correct
memory allocation behavior.


Minimal calls to the allocator.
The system malloc() and free() implementations are inefficient
on many systems.  SQLite strives to reduce overall processing time
by minimizing its use of malloc() and free().



Open access.
Pluggable SQLite extensions or even the application itself can 
access to the same underlying memory allocation
routines used by SQLite through the
sqlite3_malloc(), sqlite3_realloc(), and sqlite3_free() interfaces.


c2: Dynamic Memory Allocation In SQLite
c3: 2. Testing
c5: malloc.html#_testing
c4:
Most of the code in the SQLite source tree is devoted purely to 
testing and verification.  Reliability is important to SQLite.
Among the tasks of the test infrastructure is to ensure that
SQLite does not misuse dynamically allocated memory, that SQLite
does not leak memory, and that SQLite responds
correctly to a dynamic memory allocation failure.

The test infrastructure verifies that SQLite does not misuse
dynamically allocated memory by using a specially instrumented
memory allocator.  The instrumented memory allocator is enabled
at compile-time using the SQLITE_MEMDEBUG option.  The instrumented
memory allocator is much slower than the default memory allocator and
so its use is not recommended in production.  But when
enabled during testing, 
the instrumented memory allocator performs the following checks:


Bounds checking.
The instrumented memory allocator places sentinel values at both ends
of each memory allocation to verify that nothing within SQLite writes
outside the bounds of the allocation.

Use of memory after freeing.
When each block of memory is freed, every byte is overwritten with a
nonsense bit pattern.  This helps to ensure that no memory is ever
used after having been freed.

Freeing memory not obtained from malloc.
Each memory allocation from the instrumented memory allocator contains
sentinels used to verify that every allocation freed came
from prior malloc.

Uninitialized memory.
The instrumented memory allocator initializes each memory allocation
to a nonsense bit pattern to help ensure that the user makes no
assumptions about the content of allocation memory.


Regardless of whether or not the instrumented memory allocator is
used, SQLite keeps track of how much memory is currently checked out.
There are hundreds of test scripts used for testing SQLite.  At the
end of each script, all objects are destroyed and a test is made to
ensure that all  memory has been freed.  This is how memory
leaks are detected.  Notice that memory leak detection is in force at
all times, during test builds and during production builds.  Whenever
one of the developers runs any individual test script, memory leak
detection is active.  Hence memory leaks that do arise during development
are quickly detected and fixed.


The response of SQLite to out-of-memory (OOM) errors is tested using
a specialized memory allocator overlay that can simulate memory failures.
The overlay is a layer that is inserted in between the memory allocator
and the rest of SQLite.  The overlay passes most memory allocation
requests straight through to the underlying allocator and passes the
results back up to the requester.  But the overlay can be set to 
cause the Nth memory allocation to fail.  To run an OOM test, the overlay
is first set to fail on the first allocation attempt.  Then some test
script is run and verification that the allocation was correctly caught
and handled is made.  Then the overlay is set to fail on the second
allocation and the test repeats.  The failure point continues to advance
one allocation at a time until the entire test procedure runs to
completion without hitting a memory allocation error.  This whole
test sequence run twice.  On the first pass, the
overlay is set to fail only the Nth allocation.  On the second pass,
the overlay is set to fail the Nth and all subsequent allocations.

Note that the memory leak detection logic continues to work even
when the OOM overlay is being used.  This verifies that SQLite
does not leak memory even when it encounters memory allocation errors.
Note also that the OOM overlay can work with any underlying memory
allocator, including the instrumented memory allocator that checks
for memory allocation misuse.  In this way it is verified that 
OOM errors do not induce other kinds of memory usage errors.

Finally, we observe that the instrumented memory allocator and the
memory leak detector both work over the entire SQLite test suite and
the TCL test suite provides over 99% statement test coverage and that
the TH3 test harness provides 100% branch test coverage
with no leak leaks. This is
strong evidence that dynamic memory allocation is used correctly
everywhere within SQLite.


c2: Dynamic Memory Allocation In SQLite
c3: 2.1. Use of reallocarray()
c5: malloc.html#_use_of_reallocarray_
c4:
The reallocarray() interface is a recent innovation (circa 2014)
from the OpenBSD community that grow out of efforts to prevent the
next "heartbleed" bug by avoiding 32-bit integer
arithmetic overflow on memory allocation size computations.  The
reallocarray() function has both unit-size and count parameters.
To allocate memory sufficient to hold an array of N elements each X-bytes
in size, one calls "reallocarray(0,X,N)".  This is preferred over
the traditional technique of invoking "malloc(X*N)" as reallocarray()
eliminates the risk that the X*N multiplication will overflow and
cause malloc() to return a buffer that is a different size from what
the application expected.

SQLite does not use reallocarray().  The reason is that reallocarray()
is not useful to SQLite.  It turns out that SQLite never does memory
allocations that are the simple product of two integers.  Instead, SQLite
does allocations of the form "X+C" or "N*X+C" or "M*N*X+C" or
"N*X+M*Y+C", and so forth.  The reallocarray() interface is not helpful
in avoiding integer overflow in those cases.

Nevertheless, integer overflow in the computation of memory allocation
sizes is a concern that SQLite would like to deal with.  To prevent
problems, all SQLite internal memory allocations occur using thin wrapper
functions that take a signed 64-bit integer size parameter.  The SQLite 
source code is audited to ensure that all size computations are carried 
out using 64-bit signed integers as well.  SQLite will
refuse to allocate more than about 2GB of memory at one go.  (In common
use, SQLite seldom ever allocates more than about 8KB of memory at a time
so a 2GB allocation limit is not a burden.)  So the 64-bit size parameter
provides lots of headroom for detecting overflows.  The same audit that
verifies that all size computations are done as 64-bit signed integers
also verifies that it is impossible to overflow a 64-bit integer
during the computation.

The code audits used to ensure that memory allocation size computations
do not overflow in SQLite are repeated prior to every SQLite release.


c2: Dynamic Memory Allocation In SQLite
c3: 3. Configuration
c5: malloc.html#_configuration
c4:
The default memory allocation settings in SQLite are appropriate
for most applications.  However, applications with unusual or particularly
strict requirements may want to adjust the configuration to more closely
align SQLite to their needs.
Both compile-time and start-time configuration options are available.


c2: Dynamic Memory Allocation In SQLite
c3: 3.1. Alternative low-level memory allocators
c5: malloc.html#_alternative_low_level_memory_allocators
c4:
The SQLite source code includes several different memory allocation
modules that can be selected at compile-time, or to a limited extent
at start-time.


c2: Dynamic Memory Allocation In SQLite
c3: 3.1.1. The default memory allocator
c5: malloc.html#the_default_memory_allocator
c4:
By default, SQLite uses the malloc(), realloc(), and free() routines
from the standard C library for its memory allocation needs.  These routines
are surrounded by a thin wrapper that also provides a "memsize()" function
that will return the size of an existing allocation.  The memsize() function
is needed to keep an accurate count of the number of bytes of outstanding
memory; memsize() determines how many bytes to remove from the outstanding
count when an allocation is freed.  The default allocator implements
memsize() by always allocating 8 extra bytes on each malloc() request and
storing the size of the allocation in that 8-byte header.

The default memory allocator is recommended for most applications.
If you do not have a compelling need to use an alternative memory
allocator, then use the default.


c2: Dynamic Memory Allocation In SQLite
c3: 3.1.2. The debugging memory allocator
c5: malloc.html#the_debugging_memory_allocator
c4:
If SQLite is compiled with the SQLITE_MEMDEBUG compile-time option,
then a different, heavy wrapper is used around system malloc(), realloc(), 
and free().
The heavy wrapper allocates around 100 bytes of extra space
with each allocation.  The extra space is used to place sentinel values 
at both ends of the allocation returned to the SQLite core.  When an
allocation is freed,
these sentinels are checked to make sure the SQLite core did not overrun
the buffer in either direction.  When the system library is GLIBC, the 
heavy wrapper also makes use of the GNU backtrace() function to examine
the stack and record the ancestor functions of the malloc() call.  When
running the SQLite test suite, the heavy wrapper also records the name of
the current test case.  These latter two features are useful for
tracking down the source of memory leaks detected by the test suite.

The heavy wrapper that is used when SQLITE_MEMDEBUG is set also
makes sure each new allocation is filled with nonsense data prior to
returning the allocation to the caller.  And as soon as an allocation
is free, it is again filled with nonsense data.  These two actions help
to ensure that the SQLite core does not make assumptions about the state
of newly allocated memory and that memory allocations are not used after
they have been freed.

The heavy wrapper employed by SQLITE_MEMDEBUG is intended for use
only during testing, analysis, and debugging of SQLite.  The heavy wrapper
has a significant performance and memory overhead and probably should not
be used in production.


c2: Dynamic Memory Allocation In SQLite
c3: 3.1.3. The Win32 native memory allocator
c5: malloc.html#the_win32_native_memory_allocator
c4:
If SQLite is compiled for Windows with the SQLITE_WIN32_MALLOC
compile-time option, then a different, thin wrapper is used around
HeapAlloc(), HeapReAlloc(), and HeapFree().  The thin wrapper uses the
configured SQLite heap, which will be different from the default process
heap if the SQLITE_WIN32_HEAP_CREATE compile-time option is used.  In
addition, when an allocation is made or freed, HeapValidate() will be
called if SQLite is compiled with assert() enabled and the
SQLITE_WIN32_MALLOC_VALIDATE compile-time option.


c2: Dynamic Memory Allocation In SQLite
c3: 3.1.4. Zero-malloc memory allocator
c5: malloc.html#zero_malloc_memory_allocator
c4:
When SQLite is compiled with the SQLITE_ENABLE_MEMSYS5 option, an
alternative memory allocator that does not use malloc() is included in the
build.  The SQLite developers refer to this alternative memory allocator
as "memsys5".  Even when it is included in the build, memsys5 is 
disabled by default.
To enable memsys5, the application must invoke the following SQLite 
interface at start-time:


sqlite3_config(SQLITE_CONFIG_HEAP, pBuf, szBuf, mnReq);


In the call above, pBuf is a pointer to a large, contiguous chunk
of memory space that SQLite will use to satisfy all of its memory
allocation needs.   pBuf might point to a static array or it might
be memory obtained from some other application-specific mechanism.
szBuf is an integer that is the number of bytes of memory space
pointed to by pBuf.  mnReq is another integer that is the
minimum size of an allocation.  Any call to sqlite3_malloc(N) where
N is less than mnReq will be rounded up to mnReq.  mnReq must be
a power of two.  We shall see later that the mnReq parameter is
important in reducing the value of n and hence the minimum memory
size requirement in the Robson proof.

The memsys5 allocator is designed for use on embedded systems, 
though there is nothing to prevent its use on workstations.
The szBuf is typically between a few hundred kilobytes up to a few
dozen megabytes, depending on system requirements and memory budget.

The algorithm used by memsys5 can be called "power-of-two,
first-fit".  The sizes of all memory allocation 
requests are rounded up to a power of two and the request is satisfied
by the first free slot in pBuf that is large enough.  Adjacent freed
allocations are coalesced using a buddy system. When used appropriately,
this algorithm provides mathematical guarantees against fragmentation and
breakdown, as described further below.


c2: Dynamic Memory Allocation In SQLite
c3: 3.1.5. Experimental memory allocators
c5: malloc.html#experimental_memory_allocators
c4:
The name "memsys5" used for the zero-malloc memory allocator implies
that there are several additional memory allocators available, and indeed
there are.  The default memory allocator is "memsys1".  The debugging
memory allocator is "memsys2".  Those have already been covered.

If SQLite is compiled with SQLITE_ENABLE_MEMSYS3 then another
zero-malloc memory allocator, similar to memsys5, is included in the
source tree.  The memsys3 allocator, like memsys5, must be activated
by a call to sqlite3_config(SQLITE_CONFIG_HEAP,...).  Memsys3
uses the memory buffer supplied as its source for all memory allocations.
The difference between memsys3 and memsys5 is that memsys3 uses a
different memory allocation algorithm that seems to work well in
practice, but which does not provide mathematical
guarantees against memory fragmentation and breakdown.  Memsys3 was
a predecessor to memsys5.  The SQLite developers now believe that 
memsys5 is superior to
memsys3 and that all applications that need a zero-malloc memory
allocator should use memsys5 in preference to memsys3.  Memsys3 is
considered both experimental and deprecated and will likely be removed 
from the source tree in a future release of SQLite.

Memsys4 and memsys6 were experimental memory allocators
introduced in around 2007 and subsequently removed from the
source tree in around 2008, after it became clear that they
added no new value.

Other experimental memory allocators might be added in future releases
of SQLite.  One may anticipate that these will be called memsys7, memsys8,
and so forth.


c2: Dynamic Memory Allocation In SQLite
c3: 3.1.6. Application-defined memory allocators
c5: malloc.html#application_defined_memory_allocators
c4:
New memory allocators do not have to be part of the SQLite source tree
nor included in the sqlite3.c amalgamation.  Individual applications can
supply their own memory allocators to SQLite at start-time.

To cause SQLite to use a new memory allocator, the application
simply calls:


sqlite3_config(SQLITE_CONFIG_MALLOC, pMem);


In the call above, pMem is a pointer to an sqlite3_mem_methods object
that defines the interface to the application-specific memory allocator.
The sqlite3_mem_methods object is really just a structure containing
pointers to functions to implement the various memory allocation primitives.


In a multi-threaded application, access to the sqlite3_mem_methods
is serialized if and only if SQLITE_CONFIG_MEMSTATUS is enabled.
If SQLITE_CONFIG_MEMSTATUS is disabled then the methods in
sqlite3_mem_methods must take care of their own serialization needs.


c2: Dynamic Memory Allocation In SQLite
c3: 3.1.7. Memory allocator overlays
c5: malloc.html#memory_allocator_overlays
c4:
An application can insert layers or "overlays" in between the
SQLite core and the underlying memory allocator.
For example, the out-of-memory test logic
for SQLite uses an overlay that can simulate memory allocation
failures.

An overlay can be created by using the


sqlite3_config(SQLITE_CONFIG_GETMALLOC, pOldMem);


interface to obtain pointers to the existing memory allocator.
The existing allocator is saved by the overlay and is used as
a fallback to do real memory allocation.  Then the overlay is
inserted in place of the existing memory allocator using
the sqlite3_config(SQLITE_CONFIG_MALLOC,...) as described
above.


c2: Dynamic Memory Allocation In SQLite
c3: 3.1.8. No-op memory allocator stub
c5: malloc.html#no_op_memory_allocator_stub
c4:
If SQLite is compiled with the SQLITE_ZERO_MALLOC option, then
the default memory allocator is omitted and replaced by a stub
memory allocator that never allocates any memory.  Any calls to the
stub memory allocator will report back that no memory is available.

The no-op memory allocator is not useful by itself.  It exists only
as a placeholder so that SQLite has a memory allocator to link against
on systems that may not have malloc(), free(), or realloc() in their
standard library.
An application that is compiled with SQLITE_ZERO_MALLOC will need to
use sqlite3_config() together with SQLITE_CONFIG_MALLOC or
SQLITE_CONFIG_HEAP to specify a new alternative memory allocator
before beginning to use SQLite.


c2: Dynamic Memory Allocation In SQLite
c3: 3.2. Page cache memory
c5: malloc.html#_page_cache_memory
c4:
In most applications, the database page cache subsystem within 
SQLite uses more dynamically allocated memory than all other parts
of SQLite combined.  It is not unusual to see the database page cache
consume over 10 times more memory than the rest of SQLite combined.

SQLite can be configured to make page cache memory allocations from
a separate and distinct memory pool of fixed-size
slots.  This can have two advantages:



Because allocations are all the same size, the memory allocator can
operate much faster.  The allocator need not bother with coalescing 
adjacent free slots or searching for a slot
of an appropriate size.  All unallocated memory slots can be stored on
a linked list.  Allocating consists of removing the first entry from the
list.  Deallocating is simply adding an entry to the beginning of the list.



With a single allocation size, the n parameter in the
Robson proof is 1, and the total memory space required by the allocator
(N) is exactly equal to maximum memory used (M).  
No additional memory is required to cover fragmentation overhead, thus 
reducing memory requirements.  This is particularly important for the
page cache memory since the page cache constitutes the largest component
of the memory needs of SQLite.



The page-cache memory allocator is disabled by default.
An application can enable it at start-time as follows:


sqlite3_config(SQLITE_CONFIG_PAGECACHE, pBuf, sz, N);


The pBuf parameter is a pointer to a contiguous range of bytes that
SQLite will use for page-cache memory allocations.  The buffer must be
at least sz*N bytes in size.  The "sz" parameter
is the size of each page-cache allocation.  N is the maximum 
number of available allocations.

If SQLite needs a page-cache entry that is larger than "sz" bytes or
if it needs more than N entries, it falls back to using the
general-purpose memory allocator.


c2: Dynamic Memory Allocation In SQLite
c3: 3.3. Lookaside memory allocator
c5: malloc.html#_lookaside_memory_allocator
c4:
SQLite database connections make many
small and short-lived memory allocations.
This occurs most commonly when compiling SQL statements using
sqlite3_prepare_v2() but also to a lesser extent when running
prepared statements using sqlite3_step().  These small memory
allocations are used to hold things such as the names of tables
and columns, parse tree nodes, individual query results values,
and B-Tree cursor objects.  There are consequently
many calls to malloc() and free() - so many calls that malloc() and
free() end up using a significant fraction of the CPU time assigned
to SQLite.

SQLite version 3.6.1 (2008-08-06)
introduced the lookaside memory allocator to
help reduce the memory allocation load.  In the lookaside allocator,
each database connection preallocates a single large chunk of memory
(typically in the range of 60 to 120 kilobytes) and divides that chunk
up into small fixed-size "slots" of around 100 to 1000 byte each.  This
becomes the lookaside memory pool.  Thereafter, memory allocations
associated with the database connection and that are not too large
are satisfied using one of the lookaside pool slots rather than by calling
the general-purpose memory allocator.  Larger allocations continue to
use the general-purpose memory allocator, as do allocations that occur
when the lookaside pool slots are all checked out.  
But in many cases, the memory
allocations are small enough and there are few enough outstanding that
the new memory requests can be satisfied from the lookaside
pool.

Because lookaside allocations are always the same size, the allocation
and deallocation algorithms are very quick.  There is no
need to coalesce adjacent free slots or search for a slot
of a particular size.  Each database connection maintains a singly-linked
list of unused slots.  Allocation requests simply pull the first
element of this list.  Deallocations simply push the element back onto
the front of the list.
Furthermore, each database connection is assumed to already be
running in a single thread (there are mutexes already in
place to enforce this) so no additional mutexing is required to 
serialize access to the lookaside slot freelist.
Consequently, lookaside memory
allocations and deallocations are very fast.  In speed tests on
Linux and Mac OS X workstations, SQLite has shown overall performance
improvements as high as 10% and 15%, depending on the workload how
and lookaside is configured.

The size of the lookaside memory pool has a global default value
but can also be configured on a connection-by-connection basis.
To change the default size of the lookaside memory pool at
compile-time, use the 
-DSQLITE_DEFAULT_LOOKASIDE=SZ,N
option.
To change the default size of the lookaside memory pool at
start-time, use the sqlite3_config() interface:


sqlite3_config(SQLITE_CONFIG_LOOKASIDE, sz, cnt);


The "sz" parameter is the size in bytes of each lookaside slot.
The "cnt" parameter is
the total number of lookaside memory slots per database connection.
The total amount
of lookaside memory allocated to each database connection is
sz*cnt bytes. 


The lookaside pool can be changed for an individual
database connection "db" using this call:


sqlite3_db_config(db, SQLITE_DBCONFIG_LOOKASIDE, pBuf, sz, cnt);


The "pBuf" parameter is a pointer to memory space that will be
used for the lookaside memory pool.  If pBuf is NULL, then SQLite
will obtain its own space for the memory pool using sqlite3_malloc().
The "sz" and "cnt" parameters are the size of each lookaside slot
and the number of slots, respectively.  If pBuf is not NULL, then it
must point to at least sz*cnt bytes of memory.

The lookaside configuration can only be changed while there are
no outstanding lookaside allocations for the database connection.
Hence, the configuration should be set immediately after creating the 
database connection using sqlite3_open() (or equivalent) and before
evaluating any SQL statements on the connection.


c2: Dynamic Memory Allocation In SQLite
c3: 3.3.1. Two-Size Lookaside
c5: malloc.html#two_size_lookaside
c4:
Beginning with SQLite version 3.31.0 (2020-01-22),
lookaside supports two memory pools, each with a different size
slot.  The small-slot pool uses 128-byte slots and the large-slot
pool uses whatever size is specified by SQLITE_DBCONFIG_LOOKASIDE
(defaulting to 1200 bytes).  Splitting the pool in two like this
allows memory allocations to be covered by lookaside more often
while at the same time reducing per-database-connection heap usage
from 120KB down to 48KB.


Configuration continues to use the SQLITE_DBCONFIG_LOOKASIDE or
SQLITE_CONFIG_LOOKASIDE configuration options, as described above,
with parameters "sz" and "cnt".  The total heap space used for
lookaside continues to be sz*cnt bytes.  But the space is allocated
between the small-slot lookaside and big-slot lookaside, with
preference given to small-slot lookaside.  The total number of
slots will usually exceed "cnt", since "sz" is typically much
larger than the small-slot size of 128 bytes.


The default lookaside configuration has changed from 100 slots
of 1200 bytes each (120KB) to be 40 slots of 1200 bytes each
(48KB).  This space ends up being allocated as 93 slots of
128 bytes each and 30 slots of 1200 bytes each.  So more lookaside
slots are available but much less heap space is used.


The default lookaside configuration, the size of the small-slots,
and the details of how heap space is allocated between small-slots
and big-slots, are all subject to change from one release to the
next.


c2: Dynamic Memory Allocation In SQLite
c3: 3.4. Memory status
c5: malloc.html#_memory_status
c4:
By default, SQLite keeps statistics on its memory usage.  These
statistics are useful in helping to determine how much memory an
application really needs.  The statistics can also be used in
high-reliability system to determine
if the memory usage is coming close to or exceeding the limits 
of the Robson proof and hence that the memory allocation subsystem is 
liable to breakdown.

Most memory statistics are global, and therefore the tracking of
statistics must be serialized with a mutex.  Statistics are turned 
on by default, but an option exists to disable them.  By disabling 
memory statistics,
SQLite avoids entering and leaving a mutex on each memory allocation
and deallocation.  That savings can be noticeable on systems where
mutex operations are expensive.  To disable memory statistics, the
following interface is used at start-time:


sqlite3_config(SQLITE_CONFIG_MEMSTATUS, onoff);


The "onoff" parameter is true to enable the tracking of memory
statistics and false to disable statistics tracking.

Assuming statistics are enabled, the following routine can be used
to access them:


sqlite3_status(verb, &amp;current, &amp;highwater, resetflag);


The "verb" argument determines what statistic is accessed.
There are various verbs defined.  The
list is expected to grow as the sqlite3_status() interface matures.
The current value the selected parameter is written into integer 
"current" and the highest historical value
is written into integer "highwater".  If resetflag is true, then
the high-water mark is reset down to the current value after the call
returns.

A different interface is used to find statistics associated with a
single database connection:


sqlite3_db_status(db, verb, &amp;current, &amp;highwater, resetflag);


This interface is similar except that it takes a pointer to
a database connection as its first argument and returns statistics about
that one object rather than about the entire SQLite library.
The sqlite3_db_status() interface currently only recognizes a
single verb SQLITE_DBSTATUS_LOOKASIDE_USED, though additional verbs
may be added in the future.

The per-connection statistics do not use global variables and hence
do not require mutexes to update or access.  Consequently the
per-connection statistics continue to function even if
SQLITE_CONFIG_MEMSTATUS is turned off.


c2: Dynamic Memory Allocation In SQLite
c3: 3.5. Setting memory usage limits
c5: malloc.html#_setting_memory_usage_limits
c4:
The sqlite3_soft_heap_limit64() interface can be used to set an
upper bound on the total amount of outstanding memory that the
general-purpose memory allocator for SQLite will allow to be outstanding
at one time.  If attempts are made to allocate more memory than specified
by the soft heap limit, then SQLite will first attempt to free cache
memory before continuing with the allocation request.  The soft heap
limit mechanism only works if memory statistics are enabled and
it works best
if the SQLite library is compiled with the SQLITE_ENABLE_MEMORY_MANAGEMENT
compile-time option.

The soft heap limit is "soft" in this sense:  If SQLite is not able
to free up enough auxiliary memory to stay below the limit, it goes
ahead and allocates the extra memory and exceeds its limit.  This occurs
under the theory that it is better to use additional memory than to fail
outright.

As of SQLite version 3.6.1 (2008-08-06), 
the soft heap limit only applies to the
general-purpose memory allocator.  The soft heap limit does not know
about or interact with
the pagecache memory allocator or the lookaside memory allocator.
This deficiency will likely be addressed in a future release.


c2: Dynamic Memory Allocation In SQLite
c3: 4. Mathematical Guarantees Against Memory Allocation Failures
c5: malloc.html#_mathematical_guarantees_against_memory_allocation_failures
c4:
The problem of dynamic memory allocation, and specifically the
problem of a memory allocator breakdown, has been studied by
J. M. Robson and the results published as:


J. M. Robson.  "Bounds for Some Functions Concerning Dynamic
Storage Allocation".  Journal of the Association for
Computing Machinery, Volume 21, Number 8, July 1974,
pages 491-499.


Let us use the following notation (similar but not identical to
Robson's notation):



N

The amount of raw memory needed by the memory allocation system
in order to guarantee that no memory allocation will ever fail.

M

The maximum amount of memory that the application ever has checked out
at any point in time.

n

The ratio of the largest memory allocation to the smallest.  We assume
that every memory allocation size is an integer multiple of the smallest memory
allocation size.




Robson proves the following result:


N = M*(1 + (log2 n)/2) - n + 1


Colloquially, the Robson proof shows that in order to guarantee
breakdown-free operation, any memory allocator must use a memory pool
of size N which exceeds the maximum amount of memory ever
used M by a multiplier that depends on n, 
the ratio of the largest to the smallest allocation size.  In other
words, unless all memory allocations are of exactly the same size
(n=1) then the system needs access to more memory than it will
ever use at one time.  Furthermore, we see that the amount of surplus
memory required grows rapidly as the ratio of largest to smallest
allocations increases, and so there is strong incentive to keep all
allocations as near to the same size as possible.

Robson's proof is constructive. 
He provides an algorithm for computing a sequence of allocation
and deallocation operations that will lead to an allocation failure due to
memory fragmentation if available memory is as much as one byte
less than N.
And, Robson shows that a power-of-two first-fit memory allocator
(such as implemented by memsys5) will never fail a memory allocation
provided that available memory is N or more bytes.

The values M and n are properties of the application.
If an application is constructed in such a way that both M and
n are known, or at least have known upper bounds, and if the
application uses
the memsys5 memory allocator and is provided with N bytes of
available memory space using SQLITE_CONFIG_HEAP
then Robson proves that no memory allocation request will ever fail
within the application.
To put this another way, the application developer can select a value
for N that will guarantee that no call to any SQLite interface
will ever return SQLITE_NOMEM.  The memory pool will never become
so fragmented that a new memory allocation request cannot be satisfied.
This is an important property for
applications where a software fault could cause injury, physical harm, or
loss of irreplaceable data.


c2: Dynamic Memory Allocation In SQLite
c3: 4.1. Computing and controlling parameters M and n
c5: malloc.html#_computing_and_controlling_parameters_m_and_n
c4:
The Robson proof applies separately to each of the memory allocators
used by SQLite:


The general-purpose memory allocator (memsys5).
The pagecache memory allocator.
The lookaside memory allocator.


For allocators other than memsys5,
all memory allocations are of the same size.  Hence, n=1
and therefore N=M.  In other words, the memory pool need
be no larger than the largest amount of memory in use at any given moment.

The usage of pagecache memory is somewhat harder to control in
SQLite version 3.6.1, though mechanisms are planned for subsequent
releases that will make controlling pagecache memory much easier.
Prior to the introduction of these new mechanisms, the only way
to control pagecache memory is using the cache_size pragma.

Safety-critical applications will usually want to modify the
default lookaside memory configuration so that when the initial
lookaside memory buffer is allocated during sqlite3_open() the
resulting memory allocation is not so large as to force the n
parameter to be too large.  In order to keep n under control,
it is best to try to keep the largest memory allocation below 2 or 4
kilobytes.  Hence, a reasonable default setup for the lookaside
memory allocator might any one of the following:


sqlite3_config(SQLITE_CONFIG_LOOKASIDE, 32, 32);  /* 1K */
sqlite3_config(SQLITE_CONFIG_LOOKASIDE, 64, 32);  /* 2K */
sqlite3_config(SQLITE_CONFIG_LOOKASIDE, 32, 64);  /* 2K */
sqlite3_config(SQLITE_CONFIG_LOOKASIDE, 64, 64);  /* 4K */


Another approach is to initially disable the lookaside memory
allocator:


sqlite3_config(SQLITE_CONFIG_LOOKASIDE, 0, 0);


Then let the application maintain a separate pool of larger
lookaside memory buffers that it can distribute to database connections
as they are created.  In the common case, the application will only
have a single database connection and so the lookaside memory pool
can consist of a single large buffer.


sqlite3_db_config(db, SQLITE_DBCONFIG_LOOKASIDE, aStatic, 256, 500);


The lookaside memory allocator is really intended as performance
optimization, not as a method for assuring breakdown-free memory allocation,
so it is not unreasonable to completely disable the lookaside memory
allocator for safety-critical operations.

The general purpose memory allocator is the most difficult memory pool
to manage because it supports allocations of varying sizes.  Since 
n is a multiplier on M we want to keep n as small
as possible.  This argues for keeping the minimum allocation size for
memsys5 as large as possible.  In most applications, the
lookaside memory allocator is able to handle small allocations.  So
it is reasonable to set the minimum allocation size for memsys5 to
2, 4 or even 8 times the maximum size of a lookaside allocation.  
A minimum allocation size of 512 is a reasonable setting.

Further to keeping n small, one desires to keep the size of
the largest memory allocations under control.
Large requests to the general-purpose memory allocator
might come from several sources:


SQL table rows that contain large strings or BLOBs.
Complex SQL queries that compile down to large prepared statements.
SQL parser objects used internally by sqlite3_prepare_v2().
Storage space for database connection objects.
Page cache memory allocations that overflow into the general-purpose
    memory allocator.
Lookaside buffer allocations for new database connections.


The last two allocations can be controlled and/or eliminated by
configuring the pagecache memory allocator,
and lookaside memory allocator appropriately, as described above.
The storage space required for database connection objects depends
to some extent on the length of the filename of the database file, but
rarely exceeds 2KB on 32-bit systems.  (More space is required on
64-bit systems due to the increased size of pointers.)
Each parser object uses about 1.6KB of memory.  Thus, elements 3 through 7
above can easily be controlled to keep the maximum memory allocation
size below 2KB.

If the application is designed to manage data in small pieces,
then the database should never contain any large strings or BLOBs
and hence element 1 above should not be a factor.  If the database
does contain large strings or BLOBs, they should be read using
incremental BLOB I/O and rows that contain the
large strings or BLOBs should never be update by any means other
than incremental BLOB I/O.  Otherwise, the 
sqlite3_step() routine will need to read the entire row into
contiguous memory at some point, and that will involve at least
one large memory allocation.

The final source of large memory allocations is the space to hold
the prepared statements that result from compiling complex SQL
operations.  Ongoing work by the SQLite developers is reducing the
amount of space required here.  But large and complex queries might
still require prepared statements that are several kilobytes in
size.  The only workaround at the moment is for the application to
break complex SQL operations up into two or more smaller and simpler 
operations contained in separate prepared statements.

All things considered, applications should normally be able to
hold their maximum memory allocation size below 2K or 4K.  This
gives a value for log2(n) of 2 or 3.  This will
limit N to between 2 and 2.5 times M.

The maximum amount of general-purpose memory needed by the application
is determined by such factors as how many simultaneous open 
database connection and prepared statement objects the application
uses, and on the complexity of the prepared statements.  For any
given application, these factors are normally fixed and can be
determined experimentally using SQLITE_STATUS_MEMORY_USED.
A typical application might only use about 40KB of general-purpose
memory.  This gives a value of N of around 100KB.


c2: Dynamic Memory Allocation In SQLite
c3: 4.2. Ductile failure
c5: malloc.html#_ductile_failure
c4:
If the memory allocation subsystems within SQLite are configured
for breakdown-free operation but the actual memory usage exceeds
design limits set by the Robson proof, SQLite will usually continue 
to operate normally.
The pagecache memory allocator
and the lookaside memory allocator automatically failover
to the memsys5 general-purpose memory allocator.  And it is usually the
case that the memsys5 memory allocator will continue to function
without fragmentation even if M and/or n exceeds the limits
imposed by the Robson proof.  The Robson proof shows that it is 
possible for a memory allocation to break down and fail in this 
circumstance, but such a failure requires an especially
despicable sequence of allocations and deallocations - a sequence that
SQLite has never been observed to follow.  So in practice it is usually
the case that the limits imposed by Robson can be exceeded by a
considerable margin with no ill effect.

Nevertheless, application developers are admonished to monitor
the state of the memory allocation subsystems and raise alarms when
memory usage approaches or exceeds Robson limits.  In this way,
the application will provide operators with abundant warning well
in advance of failure.
The memory statistics interfaces of SQLite provide the application with
all the mechanism necessary to complete the monitoring portion of
this task.


c2: Dynamic Memory Allocation In SQLite
c3: 5. Stability Of Memory Interfaces
c5: malloc.html#_stability_of_memory_interfaces
c4:
Update: As of SQLite version 3.7.0 (2010-07-21), 
all of SQLite memory allocation interfaces
are considered stable and will be supported in future releases.


c2: SQLite Shared-Cache Mode
c3: 1. SQLite Shared-Cache Mode
c5: sharedcache.html#sqlite_shared_cache_mode
c4:
Starting with version 3.3.0 (2006-01-11), 
SQLite includes a special "shared-cache"
mode (disabled by default) intended for use in embedded servers. If
shared-cache mode is enabled and a thread establishes multiple connections
to the same database, the connections share a single data and schema cache.
This can significantly reduce the quantity of memory and IO required by
the system.

In version 3.5.0 (2007-09-04), 
shared-cache mode was modified so that the same
cache can be shared across an entire process rather than just within
a single thread.  Prior to this change, there were restrictions on
passing database connections between threads.  Those restrictions were
dropped in 3.5.0 update.  This document describes shared-cache mode
as of version 3.5.0.

Shared-cache mode changes the semantics
of the locking model in some cases. The details are described by
this document. A basic understanding of the normal SQLite locking model (see
File Locking And Concurrency In SQLite Version 3
for details) is assumed.


c2: SQLite Shared-Cache Mode
c3: 2. Shared-Cache Locking Model
c5: sharedcache.html#shared_cache_locking_model
c4:
Externally, from the point of view of another process or thread, two
or more database connections using a shared-cache appear as a single 
connection. The locking protocol used to arbitrate between multiple 
shared-caches or regular database users is described elsewhere.







Figure 1

Figure 1 depicts an example runtime configuration where three 
database connections have been established. Connection 1 is a normal
SQLite database connection. Connections 2 and 3 share a cache 
The normal locking
protocol is used to serialize database access between connection 1 and
the shared cache. The internal protocol used to serialize (or not, see
"Read-Uncommitted Isolation Mode" below) access to the shared-cache by
connections 2 and 3 is described in the remainder of this section.


There are three levels to the shared-cache locking model, 
transaction level locking, table level locking and schema level locking. 
They are described in the following three sub-sections.


c2: SQLite Shared-Cache Mode
c3: 2.1. Transaction Level Locking
c5: sharedcache.html#transaction_level_locking
c4:
SQLite connections can open two kinds of transactions, read and write
transactions. This is not done explicitly, a transaction is implicitly a
read-transaction until it first writes to a database table, at which point
it becomes a write-transaction.

At most one connection to a single shared cache may open a 
write transaction at any one time. This may co-exist with any number of read 
transactions.


c2: SQLite Shared-Cache Mode
c3: 2.2. Table Level Locking
c5: sharedcache.html#table_level_locking
c4:
When two or more connections use a shared-cache, locks are used to 
serialize concurrent access attempts on a per-table basis. Tables support 
two types of locks, "read-locks" and "write-locks". Locks are granted to
connections - at any one time, each database connection has either a
read-lock, write-lock or no lock on each database table.


At any one time, a single table may have any number of active read-locks
or a single active write lock. To read data from a table, a connection must 
first obtain a read-lock. To write to a table, a connection must obtain a 
write-lock on that table. If a required table lock cannot be obtained,
the query fails and SQLITE_LOCKED is returned to the caller.
 

Once a connection obtains a table lock, it is not released until the
current transaction (read or write) is concluded.


c2: SQLite Shared-Cache Mode
c3: 2.2.1. Read-Uncommitted Isolation Mode
c5: sharedcache.html#read_uncommitted_isolation_mode
c4:
The behaviour described above may be modified slightly by using the 
read_uncommitted pragma to change the isolation level from serialized 
(the default), to read-uncommitted.

 A database connection in read-uncommitted mode does not attempt 
to obtain read-locks before reading from database tables as described 
above. This can lead to inconsistent query results if another database
connection modifies a table while it is being read, but it also means that
a read-transaction opened by a connection in read-uncommitted mode can
neither block nor be blocked by any other connection.

Read-uncommitted mode has no effect on the locks required to write to
database tables (i.e. read-uncommitted connections must still obtain 
write-locks and hence database writes may still block or be blocked). 
Also, read-uncommitted mode has no effect on the sqlite_schema
locks required by the rules enumerated below (see section 
"Schema (sqlite_schema) Level Locking").



  /* Set the value of the read-uncommitted flag:
  **
  **   True  -> Set the connection to read-uncommitted mode.
  **   False -> Set the connection to serialized (the default) mode.
  */
  PRAGMA read_uncommitted = &lt;boolean&gt;;

  /* Retrieve the current value of the read-uncommitted flag */
  PRAGMA read_uncommitted;


c2: SQLite Shared-Cache Mode
c3: 2.3. Schema (sqlite_schema) Level Locking
c5: sharedcache.html#schema_sqlite_schema_level_locking
c4:
The sqlite_schema table supports shared-cache read and write 
locks in the same way as all other database tables (see description 
above). The following special rules also apply:



A connection must obtain a read-lock on sqlite_schema before 
accessing any database tables or obtaining any other read or write locks.
Before executing a statement that modifies the database schema (i.e. 
a CREATE or DROP TABLE statement), a connection must obtain a write-lock on 
sqlite_schema.

A connection may not compile an SQL statement if any other connection
is holding a write-lock on the sqlite_schema table of any attached
database (including the default database, "main").


c2: SQLite Shared-Cache Mode
c3: 3. Thread Related Issues
c5: sharedcache.html#thread_related_issues
c4:
In SQLite versions 3.3.0 through 3.4.2 when shared-cache mode is enabled, 
a database connection may only be
used by the thread that called sqlite3_open() to create it.
And a connection could only share cache with another connection in the
same thread.
These restrictions were dropped beginning with SQLite 
version 3.5.0 (2007-09-04).


c2: SQLite Shared-Cache Mode
c3: 4. Shared Cache And Virtual Tables
c5: sharedcache.html#shared_cache_and_virtual_tables
c4:
In older versions of SQLite,
shared cache mode could not be used together with virtual tables.
This restriction was removed in SQLite version 3.6.17 (2009-08-10).


c2: SQLite Shared-Cache Mode
c3: 5. Enabling Shared-Cache Mode
c5: sharedcache.html#enabling_shared_cache_mode
c4:
Shared-cache mode is enabled on a per-process basis. Using the C 
interface, the following API can be used to globally enable or disable
shared-cache mode:



int sqlite3_enable_shared_cache(int);


Each call to sqlite3_enable_shared_cache() affects subsequent database
connections created using sqlite3_open(), sqlite3_open16(), or
sqlite3_open_v2().  Database connections that already exist are
unaffected.  Each call to sqlite3_enable_shared_cache() overrides
all previous calls within the same process.


Individual database connections created using sqlite3_open_v2() can
choose to participate or not participate in shared cache mode by using
the SQLITE_OPEN_SHAREDCACHE or SQLITE_OPEN_PRIVATECACHE flags the
third parameter.  The use of either of these flags overrides the
global shared cache mode setting established by sqlite3_enable_shared_cache().
No more than one of the flags should be used; if both SQLITE_OPEN_SHAREDCACHE
and SQLITE_OPEN_PRIVATECACHE flags are used in the third argument to
sqlite3_open_v2() then the behavior is undefined.

When URI filenames are used, the "cache" query parameter can be used
to specify whether or not the database will use shared cache.  Use
"cache=shared" to enable shared cache and "cache=private" to disable
shared cache.   The ability to use URI query parameters to specify the
cache sharing behavior of a database connection allows cache sharing to
be controlled in ATTACH statements.  For example:


ATTACH 'file:aux.db?cache=shared' AS aux;


c2: SQLite Shared-Cache Mode
c3: 6. Shared Cache And In-Memory Databases
c5: sharedcache.html#shared_cache_and_in_memory_databases
c4:
Beginning with SQLite version 3.7.13 (2012-06-11), 
shared cache can be used on
in-memory databases, provided that the database is created using
a URI filename.  For backwards compatibility, shared cache is always
disabled for in-memory
databases if the unadorned name ":memory:" is used to open the database.
Prior to version 3.7.13, shared cache was always
disabled for in-memory databases regardless of the database name used,
current system shared cache setting, or query parameters or flags.



Enabling shared-cache for an in-memory database allows two or more
database connections in the same process to have access to the same
in-memory database.  An in-memory database in shared cache is automatically
deleted and memory is reclaimed when the last connection to that database
closes.


c2: Why Is SQLite Coded In C
c3: 1. C Is Best
c5: whyc.html#c_is_best
c4:
Note: Sections 2.0 and 3.0 of this article were added in response
to comments on 
Hacker News and
Reddit.



Since its inception on 2000-05-29, SQLite has been implemented in generic C.
C was and continues to be the best language for implementing a software
library like SQLite.  There are no plans to recode SQLite in any other
programming language at this time.


The reasons why C is the best language to implement SQLite include:



 Performance
 Compatibility
 Low-dependency
 Stability


c2: Why Is SQLite Coded In C
c3: 1.1. Performance
c5: whyc.html#performance
c4:
An intensively used low-level library like SQLite needs to be fast.
(And SQLite is fast, see Internal Versus External BLOBs and
35% Faster Than The Filesystem for example.)

C is a great language for writing fast code.  C is sometimes
described as "portable assembly language".  It enables to developers
to code as close to the underlying hardware as possible while still
remaining portable across platforms.

Other programming languages sometimes claim to be "as fast as C".
But no other language claims to be faster than C for general-purpose
programming, because none are.


c2: Why Is SQLite Coded In C
c3: 1.2. Compatibility
c5: whyc.html#compatibility
c4:
Nearly all systems have the ability to call libraries
written in C.  This is not true of other implementation languages.

So, for example, Android applications written in Java are able to
invoke SQLite (through an adaptor).  Maybe it would have been more
convenient for Android if SQLite had been coded in Java as that would
make the interface simpler.  However, on iPhone applications are coded
in Objective-C or Swift, neither of which have the ability to call
libraries written in Java.  Thus, SQLite would be unusable on iPhones
had it been written in Java.


c2: Why Is SQLite Coded In C
c3: 1.3. Low-Dependency
c5: whyc.html#low_dependency
c4:
Libraries written in C do not have a huge run-time dependency.
In its minimum configuration, SQLite requires only the following
routines from the standard C library:






 memcmp()
 memcpy()
 memmove()
 memset()


&nbsp;&nbsp;&nbsp;


 strcmp()
 strlen()
 strncmp()







In a more complete build, SQLite also uses library routines like
malloc() and free() and operating system interfaces for opening, reading,
writing, and closing files.  But even then, the number of dependencies
is very small.  Other "modern" language, in contrast, often require
multi-megabyte runtimes loaded with thousands and thousands of interfaces.


c2: Why Is SQLite Coded In C
c3: 1.4. Stability
c5: whyc.html#stability
c4:
The C language is old and boring.
It is a well-known and well-understood language.
This is exactly what one wants when developing a module like SQLite.
Writing a small, fast, and reliable database engine is hard enough as it
is without the implementation language changing out from under you with
each update to the implementation language specification.


c2: Why Is SQLite Coded In C
c3: 2. Why Isn't SQLite Coded In An Object-Oriented Language?
c5: whyc.html#why_isn_t_sqlite_coded_in_an_object_oriented_language_
c4:
Some programmers cannot imagine developing a complex system like
SQLite in a language that is not "object oriented".  So why is
SQLite not coded in C++ or Java?



Libraries written in C++ or Java can generally only be used by
applications written in the same language. It is difficult to
get an application written in Haskell or Java to invoke a library
written in C++.  On the other hand, libraries written in C are
callable from any programming language.


Object-Oriented is a design pattern, not a programming language.
You can do object-oriented programming in any language you want,
including assembly language.  Some languages (ex: C++ or Java) make
object-oriented easier.  But you can still do object-oriented programming
in languages like C.


Object-oriented is not the only valid design pattern.
Many programmers have been taught to think purely in terms of
objects.  And, to be fair, objects are often a good way to
decompose a problem.  But objects are not the only way, and are
not always the best way to decompose a problem.  Sometimes good old
procedural code is easier to write, easier to maintain and understand,
and faster than object-oriented code.


When SQLite was first being developed, Java was a young and immature
language.  C++ was older, but was undergoing such growing pains that
it was difficult to find any two C++ compilers that worked the same
way.  So C was definitely a better choice back when SQLite was first
being developed.  The situation is less stark now, but there is little
to no benefit in recoding SQLite at this point.


c2: Why Is SQLite Coded In C
c3: 3. Why Isn't SQLite Coded In A "Safe" Language?
c5: whyc.html#why_isn_t_sqlite_coded_in_a_safe_language_
c4:
There has lately been a lot of interest in "safe" programming languages
like Rust or Go in which it is impossible, or is at least difficult, to make
common programming errors like memory leaks or array overruns.  So the
question often arises as to why SQLite is not coded in a "safe" language.



None of the safe programming languages existed for the first 10 years
of SQLite's existence.  SQLite could be recoded in Go or Rust, but doing
so would probably introduce far more bugs than would be fixed, and it
seems also likely to result in slower code.


Safe programming languages solve the easy problems: memory leaks,
use-after-free errors, array overruns, etc.  Safe languages provide
no help beyond ordinary C code in solving the rather more difficult
problem of computing a correct answer to an SQL statement.


Safe languages are often touted for helping to prevent
security vulnerabilities. True enough, but SQLite is
not a particularly security-sensitive library.  If an application is
running untrusted and unverified SQL, then it already has much bigger
security issues (SQL injection) that no "safe" language will fix.

It is true that applications sometimes import complete binary SQLite
database files from untrusted sources, and such imports could present a
possible attack vector.  However, those code paths in SQLite are
limited and are extremely well tested.  And pre-validation routines 
are available to applications that want to read untrusted databases
that can help detect possible attacks prior to use.


Some "safe" languages (ex: Go) dislike the use of assert().  But
the use of assert() is a vital part of keeping SQLite maintainable.
The lack of assert() in Go is a show-stopper as far as the developers
of SQLite are concerned.  See the The Use Of assert() In SQLite
article for additional information.


Safe languages insert additional machine branches to do things like
verify that array accesses are in-bounds.  In correct code, those
branches are never taken.  That means that the machine code cannot
be 100% branch tested, which is an important component of SQLite's
quality strategy.


Safe languages usually want to abort if they encounter an out-of-memory
(OOM) situation.  SQLite is designed to recover gracefully from an OOM.
It is unclear how this could be accomplished in the current crop of
safe languages.


All of the existing safe languages are new.  The developers of SQLite
applaud the efforts of computer language researchers in trying to
develop languages that are easier to program safely.  We encourage these
efforts to continue.  Be we ourselves are more interested in old and
boring languages when it comes to implementing SQLite.



All that said, it is possible that SQLite might
one day be recoded in Rust.  Recoding SQLite in Go is unlikely
since Go hates assert().  But Rust is a possibility.  Some
preconditions that must occur before SQLite is recoded in Rust
include:



 Rust needs to mature a little more, stop changing so fast, and
     move further toward being old and boring.
 Rust needs to demonstrate that it can be used to create general-purpose
     libraries that are callable from all other programming languages.
 Rust needs to demonstrate that it can produce object code that
     works on obscure embedded devices, including devices that lack
     an operating system.
 Rust needs to pick up the necessary tooling that enables one to
     do 100% branch coverage testing of the compiled binaries.
 Rust needs a mechanism to recover gracefully from OOM errors.
 Rust needs to demonstrate that it can do the kinds of work that
     C does in SQLite without a significant speed penalty.



If you are a "rustacean" and feel that Rust already meets the
preconditions listed above, and that SQLite should be recoded in
Rust, then you are welcomed and encouraged
to contact the SQLite developers privately
and argue your case.


c2: SQLite FTS5 Extension
c3: 1. Overview of FTS5
c5: fts5.html#overview_of_fts5
c4:
FTS5 is an SQLite virtual table module that provides 
full-text search
functionality to database applications. In their most elementary form, 
full-text search engines allow the user to efficiently search a large 
collection of documents for the subset that contain one or more instances of a
search term. The search functionality provided to world wide web users by
Google is, among other things, a full-text search
engine, as it allows users to search for all documents on the web that contain,
for example, the term "fts5".

To use FTS5, the user creates an FTS5 virtual table with one or more
columns. For example:

CREATE VIRTUAL TABLE email USING fts5(sender, title, body);


It is an error to add types, constraints or PRIMARY KEY declarations to 
a CREATE VIRTUAL TABLE statement used to create an FTS5 table. Once created,
an FTS5 table may be populated using INSERT, UPDATE or DELETE statements
like any other table. Like any other table with no PRIMARY KEY declaration, an
FTS5 table has an implicit INTEGER PRIMARY KEY field named rowid. 

Not shown in the example above is that there are also 
various options that may be provided to FTS5 as
part of the CREATE VIRTUAL TABLE statement to configure various aspects of the
new table. These may be used to modify the way in which the FTS5 table extracts
terms from documents and queries, to create extra indexes on disk to speed up
prefix queries, or to create an FTS5 table that acts as an index on content
stored elsewhere.

Once populated, there are three ways to execute a full-text query against
the contents of an FTS5 table:

 Using a MATCH operator in the WHERE clause of a SELECT statement, or
     Using an equals ("=") operator in the WHERE clause of a SELECT statement, or
     using the table-valued function syntax.


If using the MATCH or = operators, the expression to the left of the MATCH
   operator is usually the name of the FTS5 table (the exception is when 
   specifying a column-filter). The expression on the right
   must be a text value specifying the term to search for. For the table-valued
   function syntax, the term to search for is specified as the first table argument.
   For example:

-- Query for all rows that contain at least once instance of the term
-- "fts5" (in any column). The following three queries are equivalent.
SELECT * FROM email WHERE email MATCH 'fts5';
SELECT * FROM email WHERE email = 'fts5';
SELECT * FROM email('fts5');


 By default, FTS5 full-text searches are case-independent. Like any other
SQL query that does not contain an ORDER BY clause, the example above returns
results in an arbitrary order. To sort results by relevance (most to least
relevant), an ORDER BY may be added to a full-text query as follows:

-- Query for all rows that contain at least once instance of the term
-- "fts5" (in any column). Return results in order from best to worst
-- match.  
SELECT * FROM email WHERE email MATCH 'fts5' ORDER BY rank;


 As well as the column values and rowid of a matching row, an application
may use FTS5 auxiliary functions to retrieve extra information regarding
the matched row. For example, an auxiliary function may be used to retrieve
a copy of a column value for a matched row with all instances of the matched
term surrounded by html &lt;b&gt;&lt;/b&gt; tags. Auxiliary functions are
invoked in the same way as SQLite scalar functions, except that the name
of the FTS5 table is specified as the first argument. For example:

-- Query for rows that match "fts5". Return a copy of the "body" column
-- of each row with the matches surrounded by &lt;b&gt;&lt;/b&gt; tags.
SELECT highlight(email, 2, '&lt;b&gt;', '&lt;/b&gt;') FROM email('fts5');


A description of the available auxiliary functions, and more details
regarding configuration of the special "rank" column, are 
available below. Custom auxiliary functions may also be implemented in C and registered with
FTS5, just as custom SQL functions may be registered with the SQLite core.

 As well as searching for all rows that contain a term, FTS5 allows 
the user to search for rows that contain:


   any terms that begin with a specified prefix,
   "phrases" - sequences of terms or prefix terms that must feature in a
       document for it to match the query, 
   sets of terms, prefix terms or phrases that appear within a specified
       proximity of each other (these are called "NEAR queries"), or
   boolean combinations of any of the above.


 Such advanced searches are requested by providing a more complicated 
FTS5 query string as the text to the right of the MATCH operator (or =
operator, or as the first argument to a table-valued function syntax). The 
full query syntax is described here.


c2: SQLite FTS5 Extension
c3: 2.1. Building FTS5 as part of SQLite
c5: fts5.html#building_fts5_as_part_of_sqlite
c4:
As of version 3.9.0 (2015-10-14),
FTS5 is included as part of the SQLite amalgamation.
If using one of the two autoconf build system, FTS5 is
enabled by specifying the "--enable-fts5" option when running the configure
script.  (FTS5 is currently disabled by default for the
source-tree configure script and enabled by default for
the amalgamation configure script, but these defaults might
change in the future.)

Or, if sqlite3.c is compiled using some other build system, by arranging for
the SQLITE_ENABLE_FTS5 pre-processor symbol to be defined.


c2: SQLite FTS5 Extension
c3: 2.2. Building a Loadable Extension
c5: fts5.html#building_a_loadable_extension
c4:
Alternatively, FTS5 may be built as a loadable extension.

The canonical FTS5 source code consists of a series of *.c and other files
in the "ext/fts5" directory of the SQLite source tree. A build process reduces
this to just two files - "fts5.c" and "fts5.h" - which may be used to build an
SQLite loadable extension.


   Obtain the latest SQLite code from fossil.
   Create a Makefile as described in How To Compile SQLite.
   Build the "fts5.c" target. Which also creates fts5.h.


$ wget -c http://www.sqlite.org/src/tarball/SQLite-trunk.tgz?uuid=trunk -O SQLite-trunk.tgz
.... output ...
$ tar -xzf SQLite-trunk.tgz
$ cd SQLite-trunk
$ ./configure && make fts5.c
... lots of output ...
$ ls fts5.&#91;ch]
fts5.c        fts5.h



  The code in "fts5.c" may then be compiled into a loadable extension or
  statically linked into an application as described in 
  Compiling Loadable Extensions. There are two entry points defined, both
  of which do the same thing:


   sqlite3_fts_init
   sqlite3_fts5_init



  The other file, "fts5.h", is not required to compile the FTS5 extension. 
  It is used by applications that implement custom FTS5 tokenizers or auxiliary functions.


c2: SQLite FTS5 Extension
c3: 3. Full-text Query Syntax
c5: fts5.html#full_text_query_syntax
c4:
The following block contains a summary of the FTS query syntax in BNF form.
A detailed explanation follows.

&lt;phrase&gt;    := string &#91;*]
&lt;phrase&gt;    := &lt;phrase&gt; + &lt;phrase&gt;
&lt;neargroup&gt; := NEAR ( &lt;phrase&gt; &lt;phrase&gt; ... &#91;, N] )
&lt;query&gt;     := &#91; &#91;-] &lt;colspec&gt; :] &#91;&#94;] &lt;phrase&gt;
&lt;query&gt;     := &#91; &#91;-] &lt;colspec&gt; :] &lt;neargroup&gt;
&lt;query&gt;     := &#91; &#91;-] &lt;colspec&gt; :] ( &lt;query&gt; )
&lt;query&gt;     := &lt;query&gt; AND &lt;query&gt;
&lt;query&gt;     := &lt;query&gt; OR &lt;query&gt;
&lt;query&gt;     := &lt;query&gt; NOT &lt;query&gt;
&lt;colspec&gt;   := colname
&lt;colspec&gt;   := { colname1 colname2 ... }


c2: SQLite FTS5 Extension
c3: 3.1. FTS5 Strings
c5: fts5.html#fts5_strings
c4:
Within an FTS expression a string may be specified in one of two ways:


   By enclosing it in double quotes ("). Within a string, any embedded
       double quote characters may be escaped SQL-style - by adding a second
       double-quote character.

   As an FTS5 bareword that is not "AND", "OR" or "NOT" (case sensitive). 
       An FTS5 bareword is a string of one or more consecutive characters that
       are all either:
       
       
          Non-ASCII range characters (i.e. unicode codepoints greater 
              than 127), or 
          One of the 52 upper and lower case ASCII characters, or
          One of the 10 decimal digit ASCII characters, or
          The underscore character (unicode codepoint 96).
          The substitute character (unicode codepoint 26).
       

       Strings that include any other characters must be quoted. Characters
       that are not currently allowed in barewords, are not quote characters and
       do not currently serve any special purpose in FTS5 query expressions may
       at some point in the future be allowed in barewords or used to implement
       new query functionality. This means that queries that are currently
       syntax errors because they include such a character outside of a quoted
       string may be interpreted differently by some future version of FTS5.


c2: SQLite FTS5 Extension
c3: 3.2. FTS5 Phrases
c5: fts5.html#fts5_phrases
c4:
FTS queries are made up of phrases. A phrase is an ordered list of 
one or more tokens. A string is transformed into a phrase by passing it to
the FTS table tokenizer. Two phrases can be concatenated into a single 
large phrase using the "+" operator. For example, assuming the tokenizer
module being used tokenizes the input "one.two.three" to three separate
tokens, the following four queries all specify the same phrase:

... MATCH '"one two three"'
... MATCH 'one + two + three'
... MATCH '"one two" + three'
... MATCH 'one.two.three'



A phrase matches a document if the document contains at least one sub-sequence
of tokens that matches the sequence of tokens that make up the phrase.


c2: SQLite FTS5 Extension
c3: 3.3. FTS5 Prefix Queries
c5: fts5.html#fts5_prefix_queries
c4:
If a "*" character follows a string within an FTS expression, then the final
token extracted from the string is marked as a prefix token. As you
might expect, a prefix token matches any document token of which it is a 
prefix. For example, the first two queries in the following block will match
any document that contains the token "one" immediately followed by the token
"two" and then any token that begins with "thr".

... MATCH '"one two thr" * '
... MATCH 'one + two + thr*'
... MATCH '"one two thr*"'      -- May not work as expected!


The final query in the block above may not work as expected. Because the
"*" character is inside the double-quotes, it will be passed to the tokenizer,
which will likely discard it (or perhaps, depending on the specific tokenizer
in use, include it as part of the final token) instead of recognizing it as
a special FTS character.


c2: SQLite FTS5 Extension
c3: 3.4. FTS5 Initial Token Queries
c5: fts5.html#fts5_initial_token_queries
c4:
If a "&#94;" character appears immediately before a phrase that is not part of a
NEAR query, then that phrase only matches a document only if it starts at the
first token in a column. The "&#94;" syntax may be combined with a 
column filter, but may not be inserted into the middle of
a phrase.

... MATCH '&#94;one'              -- first token in any column must be "one"
... MATCH '&#94; one + two'       -- phrase "one two" must appear at start of a column
... MATCH '&#94; "one two"'       -- same as previous 
... MATCH 'a : &#94;two'          -- first token of column "a" must be "two"
... MATCH 'NEAR(&#94;one, two)'   -- syntax error! 
... MATCH 'one + &#94;two'        -- syntax error! 
... MATCH '"&#94;one two"'        -- May not work as expected!


c2: SQLite FTS5 Extension
c3: 3.5. FTS5 NEAR Queries
c5: fts5.html#fts5_near_queries
c4:
Two or more phrases may be grouped into a NEAR group. A NEAR group
is specified by the token "NEAR" (case sensitive) followed by an open
parenthesis character, followed by two or more whitespace separated phrases, optionally followed by a comma and the numeric parameter N, followed by
a close parenthesis. For example:

... MATCH 'NEAR("one two" "three four", 10)'
... MATCH 'NEAR("one two" thr* + four)'


If no N parameter is supplied, it defaults to 10. A NEAR group
matches a document if the document contains at least one clump of tokens that: 

 
   contains at least one instance of each phrase, and 
   for which the number of tokens between the end of the first phrase 
       and the beginning of the last phrase in the clump is less than or equal to N.


For example:

CREATE VIRTUAL TABLE f USING fts5(x);
INSERT INTO f(rowid, x) VALUES(1, 'A B C D x x x E F x');

... MATCH 'NEAR(e d, 4)';                      -- Matches!
... MATCH 'NEAR(e d, 3)';                      -- Matches!
... MATCH 'NEAR(e d, 2)';                      -- Does not match!

... MATCH 'NEAR("c d" "e f", 3)';              -- Matches!
... MATCH 'NEAR("c"   "e f", 3)';              -- Does not match!

... MATCH 'NEAR(a d e, 6)';                    -- Matches!
... MATCH 'NEAR(a d e, 5)';                    -- Does not match!

... MATCH 'NEAR("a b c d" "b c" "e f", 4)';    -- Matches!
... MATCH 'NEAR("a b c d" "b c" "e f", 3)';    -- Does not match!


c2: SQLite FTS5 Extension
c3: 3.6. FTS5 Column Filters
c5: fts5.html#fts5_column_filters
c4:
A single phrase or NEAR group may be restricted to matching text within a
specified column of the FTS table by prefixing it with the column name 
followed by a colon character. Or to a set of columns by prefixing it
with a whitespace separated list of column names enclosed in parenthesis
("curly brackets") followed by a colon character. Column names may be specified
using either of the two forms described for strings above. Unlike strings that
are part of phrases, column names are not passed to the tokenizer module.
Column names are case-insensitive in the usual way for SQLite column names -
upper/lower case equivalence is understood for ASCII-range characters only.

... MATCH 'colname : NEAR("one two" "three four", 10)'
... MATCH '"colname" : one + two + three'

... MATCH '{col1 col2} : NEAR("one two" "three four", 10)'
... MATCH '{col2 col1 col3} : one + two + three'



If a column filter specification is preceded by a "-" character, then
it is interpreted as a list of column not to match against. For example:

-- Search for matches in all columns except "colname"
... MATCH '- colname : NEAR("one two" "three four", 10)'

-- Search for matches in all columns except "col1", "col2" and "col3"
... MATCH '- {col2 col1 col3} : one + two + three'



Column filter specifications may also be applied to arbitrary expressions
enclosed in parenthesis. In this case the column filter applies to all 
phrases within the expression. Nested column filter operations may only 
further restrict the subset of columns matched, they can not be used to 
re-enable filtered columns. For example:

-- The following are equivalent:
... MATCH '{a b} : ( {b c} : "hello" AND "world" )'
... MATCH '(b : "hello") AND ({a b} : "world")'



Finally, a column filter for a single column may be specified by using
the column name as the LHS of a MATCH operator (instead of the usual
table name). For example:

-- Given the following table
CREATE VIRTUAL TABLE ft USING fts5(a, b, c);

-- The following are equivalent
SELECT * FROM ft WHERE b MATCH 'uvw AND xyz';
SELECT * FROM ft WHERE ft MATCH 'b : (uvw AND xyz)';

-- This query cannot match any rows (since all columns are filtered out): 
SELECT * FROM ft WHERE b MATCH 'a : xyz';


c2: SQLite FTS5 Extension
c3: 3.7. FTS5 Boolean Operators
c5: fts5.html#fts5_boolean_operators
c4:
Phrases and NEAR groups may be arranged into expressions using boolean
operators. In order of precedence, from highest (tightest grouping) to
lowest (loosest grouping), the operators are:


  Operator Function

  &lt;query1&gt; NOT &lt;query2&gt; 
      Matches if query1 matches and query2 does not match.

  &lt;query1&gt; AND &lt;query2&gt; 
      Matches if both query1 and query2 match.

  &lt;query1&gt; OR &lt;query2&gt; 
      Matches if either query1 or query2 match.




Parenthesis may be used to group expressions in order to modify operator
precedence in the usual ways. For example:

-- Matches documents that contain at least one instance of either "one"
-- or "two", but do not contain any instances of token "three".
... MATCH 'one OR two NOT three'

-- Match all documents that contain the token "two" but not "three", or
-- contain the token "one".
... MATCH 'one OR (two NOT three)'



Phrases and NEAR groups may also be connected by implicit AND operators.
For simplicity, these are not shown in the BNF grammar above. Essentially, any
sequence of phrases or NEAR groups (including those restricted to matching
specified columns) separated only by whitespace are handled as if there were an
implicit AND operator between each pair of phrases or NEAR groups. Implicit
AND operators are never inserted after or before an expression enclosed in
parenthesis. For example:

... MATCH 'one two three'         -- 'one AND two AND three'
... MATCH 'three "one two"'       -- 'three AND "one two"'
... MATCH 'NEAR(one two) three'   -- 'NEAR(one two) AND three'
... MATCH 'one OR two three'      -- 'one OR two AND three'

... MATCH '(one OR two) three'    -- Syntax error!
... MATCH 'func(one two)'         -- Syntax error!


c2: SQLite FTS5 Extension
c3: 4. FTS5 Table Creation and Initialization
c5: fts5.html#fts5_table_creation_and_initialization
c4:
Each argument specified as part of a "CREATE VIRTUAL TABLE ... USING fts5 
..." statement is either a column declaration or a configuration option. A
column declaration consists of one or more whitespace separated FTS5
barewords or string literals quoted in any manner acceptable to SQLite.

The first string or bareword in a column declaration is the column name. It
is an error to attempt to name an fts5 table column "rowid" or "rank", or to
assign the same name to a column as is used by the table itself. This is not
supported.

Each subsequent string or bareword in a column declaration is a column
option that modifies the behaviour of that column. Column options are
case-independent. Unlike the SQLite core, FTS5 considers unrecognized column
options to be errors. Currently, the only option recognized is 
"UNINDEXED" (see below).

A configuration option consists of an FTS5 bareword - the option name -
followed by an "=" character, followed by the option value. The option value is
specified using either a single FTS5 bareword or a string literal, again quoted
in any manner acceptable to the SQLite core. For example:

CREATE VIRTUAL TABLE mail USING fts5(sender, title, body, tokenize = 'porter ascii');


 There are currently the following configuration options:


   The "tokenize" option, used to configure a custom tokenizer.
   The "prefix" option, used to add prefix indexes
       to an FTS5 table.
   The "content" option, used to make the FTS5 table an 
       external content or contentless table.
   The "content_rowid" option, used to set the rowid field of an 
       external content table.
   The "columnsize" option, used to configure
       whether or not the size in tokens of each value in the FTS5 table is
       stored separately within the database.
   The "detail" option. This option may be used 
       to reduce the size of the FTS index on disk by omitting some information
       from it.


c2: SQLite FTS5 Extension
c3: 4.1. The UNINDEXED column option
c5: fts5.html#the_unindexed_column_option
c4:
The contents of columns qualified with the UNINDEXED column option are not
added to the FTS index. This means that for the purposes of MATCH queries and
FTS5 auxiliary functions, the column contains no matchable tokens. 

For example, to avoid adding the contents of the "uuid" field to the FTS
index:
CREATE VIRTUAL TABLE customers USING fts5(name, addr, uuid UNINDEXED);


c2: SQLite FTS5 Extension
c3: 4.2. Prefix Indexes
c5: fts5.html#prefix_indexes
c4:
By default, FTS5 maintains a single index recording the location of each
token instance within the document set. This means that querying for complete
tokens is fast, as it requires a single lookup, but querying for a prefix 
token can be slow, as it requires a range scan. For example, to query for
the prefix token "abc*" requires a range scan of all tokens greater than
or equal to "abc" and less than "abd".

 A prefix index is a separate index that records the location of all
instances of prefix tokens of a certain length in characters used to speed
up queries for prefix tokens. For example, optimizing a query for prefix
token "abc*" requires a prefix index of three-character prefixes.

 To add prefix indexes to an FTS5 table, the "prefix" option is set to
either a single positive integer or a text value containing a white-space
separated list of one or more positive integer values. A prefix index is
created for each integer specified. If more than one "prefix" option is
specified as part of a single CREATE VIRTUAL TABLE statement, all apply.

-- Two ways to create an FTS5 table that maintains prefix indexes for
-- two and three character prefix tokens.
CREATE VIRTUAL TABLE ft USING fts5(a, b, prefix='2 3');
CREATE VIRTUAL TABLE ft USING fts5(a, b, prefix=2, prefix=3);


c2: SQLite FTS5 Extension
c3: 4.3. Tokenizers
c5: fts5.html#tokenizers
c4:
The CREATE VIRTUAL TABLE "tokenize" option is used to configure the
specific tokenizer used by the FTS5 table. The option argument must be either
an FTS5 bareword, or an SQL text literal. The text of the argument is itself
treated as a white-space series of one or more FTS5 barewords or SQL text
literals. The first of these is the name of the tokenizer to use. The second
and subsequent list elements, if they exist, are arguments passed to the
tokenizer implementation.

 Unlike option values and column names, SQL text literals intended as
tokenizers must be quoted using single quote characters. For example:

-- The following are all equivalent
CREATE VIRTUAL TABLE t1 USING fts5(x, tokenize = 'porter ascii');
CREATE VIRTUAL TABLE t1 USING fts5(x, tokenize = "porter ascii");
CREATE VIRTUAL TABLE t1 USING fts5(x, tokenize = "'porter' 'ascii'");
CREATE VIRTUAL TABLE t1 USING fts5(x, tokenize = '''porter'' ''ascii''');

-- But this will fail:
CREATE VIRTUAL TABLE t1 USING fts5(x, tokenize = '"porter" "ascii"');

-- This will fail too:
CREATE VIRTUAL TABLE t1 USING fts5(x, tokenize = 'porter' 'ascii');




FTS5 features three built-in tokenizer modules, described in subsequent
sections:


   The unicode61 tokenizer, based on the Unicode 6.1 standard. This
       is the default.

   The ascii tokenizer, which assumes all characters outside of
  the ASCII codepoint range (0-127) are to be treated as token characters.

   The porter tokenizer, which implements the 
porter stemming algorithm.


 It is also possible to create custom tokenizers for FTS5. The API for doing so is described here.


c2: SQLite FTS5 Extension
c3: 4.3.1. Unicode61 Tokenizer
c5: fts5.html#unicode61_tokenizer
c4:
The unicode tokenizer classifies all unicode characters as either 
"separator" or "token" characters. By default all space and punctuation
characters, as defined by Unicode 6.1, are considered separators, and all 
other characters as token characters. More specifically, all unicode 
characters assigned to a 

general category beginning with "L" or "N" (letters and numbers,
specifically) or to category "Co" ("other, private use") are considered tokens.
All other characters are separators.
 
Each contiguous run of one or more token characters is considered to be a
token. The tokenizer is case-insensitive according to the rules defined by
Unicode 6.1.

 By default, diacritics are removed from all Latin script characters. This
means, for example, that "A", "a", "&#192;", "&#224;", "&#194;" and "&#226;"
are all considered to be equivalent.

 Any arguments following "unicode61" in the token specification are treated
as a list of alternating option names and values. Unicode61 supports the
following options:


   Option  Usage
   remove_diacritics
  This option should be set to "0", "1" or "2". The default value is "1".
  If it is set to "1" or "2", then diacritics are removed from Latin script
  characters as described above. However, if it is set to "1", then diacritics
  are not removed in the fairly uncommon case where a single unicode codepoint
  is used to represent a character with more that one diacritic. For example,
  diacritics are not removed from codepoint 0x1ED9 ("LATIN SMALL LETTER O WITH
  CIRCUMFLEX AND DOT BELOW"). This is technically a bug, but cannot be fixed
  without creating backwards compatibility problems. If this option is set to
  "2", then diacritics are correctly removed from all Latin characters.

   categories
  This option may be used to modify the set of Unicode general categories
  that are considered to correspond to token characters. The argument must
  consist of a space separated list of two-character general category
  abbreviations (e.g. "Lu" or "Nd"), or of the same with the second character
  replaced with an asterisk ("*"), interpreted as a glob pattern. The default
  value is "L* N* Co".

   tokenchars
   This option is used to specify additional unicode characters that 
  should be considered token characters, even if they are white-space or
  punctuation characters according to Unicode 6.1. All characters in the
  string that this option is set to are considered token characters.

   separators
   This option is used to specify additional unicode characters that 
  should be considered as separator characters, even if they are token
  characters according to Unicode 6.1. All characters in the string that 
  this option is set to are considered separators.


 For example:

-- Create an FTS5 table that does not remove diacritics from Latin
-- script characters, and that considers hyphens and underscore characters
-- to be part of tokens. 
CREATE VIRTUAL TABLE ft USING fts5(a, b, 
    tokenize = "unicode61 remove_diacritics 0 tokenchars '-_'"
);


 or:

-- Create an FTS5 table that, as well as the default token character classes,
-- considers characters in class "Mn" to be token characters.
CREATE VIRTUAL TABLE ft USING fts5(a, b, 
    tokenize = "unicode61 categories 'L* N* Co Mn'"
);


 The fts5 unicode61 tokenizer is byte-for-byte compatible with the fts3/4
unicode61 tokenizer.


c2: SQLite FTS5 Extension
c3: 4.3.2. Ascii Tokenizer
c5: fts5.html#ascii_tokenizer
c4:
The Ascii tokenizer is similar to the Unicode61 tokenizer, except that:


   All non-ASCII characters (those with codepoints greater than 127) are
  always considered token characters. If any non-ASCII characters are specified
  as part of the separators option, they are ignored.  

   Case-folding is only performed for ASCII characters. So while "A" and
  "a" are considered to be equivalent, "&#195;" and "&#227;" are distinct.

   The remove_diacritics option is not supported.


 For example:

-- Create an FTS5 table that uses the ascii tokenizer, but does not
-- consider numeric characters to be part of tokens.
CREATE VIRTUAL TABLE ft USING fts5(a, b, 
    tokenize = "ascii separators '0123456789'"
);


c2: SQLite FTS5 Extension
c3: 4.3.3. Porter Tokenizer
c5: fts5.html#porter_tokenizer
c4:
The porter tokenizer is a wrapper tokenizer. It takes the output of some
other tokenizer and applies the 
porter stemming algorithm
to each token before it returns it to FTS5. This allows search terms like
"correction" to match similar words such as "corrected" or "correcting". The
porter stemmer algorithm is designed for use with English language terms 
only - using it with other languages may or may not improve search utility.

 By default, the porter tokenizer operates as a wrapper around the default
tokenizer (unicode61). Or, if one or more extra arguments are added to the
"tokenize" option following "porter", they are treated as a specification for
the underlying tokenizer that the porter stemmer uses. For example:

-- Two ways to create an FTS5 table that uses the porter tokenizer to
-- stem the output of the default tokenizer (unicode61). 
CREATE VIRTUAL TABLE t1 USING fts5(x, tokenize = porter); 
CREATE VIRTUAL TABLE t1 USING fts5(x, tokenize = 'porter unicode61');

-- A porter tokenizer used to stem the output of the unicode61 tokenizer,
-- with diacritics removed before stemming.
CREATE VIRTUAL TABLE t1 USING fts5(x, tokenize = 'porter unicode61 remove_diacritics 1');


c2: SQLite FTS5 Extension
c3: 4.4. External Content and Contentless Tables
c5: fts5.html#external_content_and_contentless_tables
c4:
Normally, when a row is inserted into an FTS5 table, as well as the various
full-text index entries and other data a copy of the row is stored in a private
table managed by the FTS5 module. When column values are requested from the
FTS5 table by the user or by an auxiliary function implementation, they are
read from this private table. The "content" option may be used to create an
FTS5 table that stores only FTS full-text index entries. Because the column
values themselves are usually much larger than the associated full-text index
entries, this can save significant database space.


There are two ways to use the "content" option:

   By setting it to an empty string to create a contentless FTS5 table. In
       this case FTS5 assumes that the original column values are unavailable
       to it when processing queries. Full-text queries and some auxiliary
       functions can still be used, but no column values apart from the rowid
       may be read from the table.

   By setting it to the name of a database object (table, virtual table or
       view) that may be queried by FTS5 at any time to retrieve the column
       values. This is known as an "external content" table. In this case all
       FTS5 functionality may be used, but it is the responsibility of the user
       to ensure that the contents of the full-text index are consistent with
       the named database object. If they are not, query results may be
       unpredictable.


c2: SQLite FTS5 Extension
c3: 4.4.1. Contentless Tables
c5: fts5.html#contentless_tables
c4:
A contentless FTS5 table is created by setting the "content" option to
an empty string. For example:

CREATE VIRTUAL TABLE f1 USING fts5(a, b, c, content='');


 Contentless FTS5 tables do not support UPDATE or DELETE statements, or
INSERT statements that do not supply a non-NULL value for the rowid field.
Contentless tables do not support REPLACE conflict handling. REPLACE 
and INSERT OR REPLACE statements are treated as regular INSERT statements.
Rows may be deleted from a contentless table using an FTS5 delete command.

 Attempting to read any column value except the rowid from a contentless
FTS5 table returns an SQL NULL value.


c2: SQLite FTS5 Extension
c3: 4.4.2. External Content Tables
c5: fts5.html#external_content_tables
c4:
An external content FTS5 table is created by setting the content 
option to the name of a table, virtual table or view (hereafter the "content
table") within the same database. Whenever column values are required by
FTS5, it queries the content table as follows, with the rowid of the row
for which values are required bound to the SQL variable:

SELECT &lt;content_rowid&gt;, &lt;cols&gt; FROM &lt;content&gt; WHERE &lt;content_rowid&gt; = ?;


 In the above, &lt;content&gt; is replaced by the name of the content table.
By default, &lt;content_rowid&gt; is replaced by the literal text "rowid". Or,
if the "content_rowid" option is set within the CREATE VIRTUAL TABLE statement,
by the value of that option. &lt;cols&gt; is replaced by a comma-separated list
of the FTS5 table column names. For example:

-- If the database schema is: 
CREATE TABLE tbl (a, b, c, d INTEGER PRIMARY KEY);
CREATE VIRTUAL TABLE fts USING fts5(a, c, content=tbl, content_rowid=d);

-- Fts5 may issue queries such as:
SELECT d, a, c FROM tbl WHERE d = ?;


 The content table may also be queried as follows:

SELECT &lt;content_rowid&gt;, &lt;cols&gt; FROM &lt;content&gt; ORDER BY &lt;content_rowid&gt; ASC;
SELECT &lt;content_rowid&gt;, &lt;cols&gt; FROM &lt;content&gt; ORDER BY &lt;content_rowid&gt; DESC;


 It is still the responsibility of the user to ensure that the contents of
an external content FTS5 table are kept up to date with the content table. 
One way to do this is with triggers. For example:

-- Create a table. And an external content fts5 table to index it.
CREATE TABLE tbl(a INTEGER PRIMARY KEY, b, c);
CREATE VIRTUAL TABLE fts_idx USING fts5(b, c, content='tbl', content_rowid='a');

-- Triggers to keep the FTS index up to date.
CREATE TRIGGER tbl_ai AFTER INSERT ON tbl BEGIN
  INSERT INTO fts_idx(rowid, b, c) VALUES (new.a, new.b, new.c);
END;
CREATE TRIGGER tbl_ad AFTER DELETE ON tbl BEGIN
  INSERT INTO fts_idx(fts_idx, rowid, b, c) VALUES('delete', old.a, old.b, old.c);
END;
CREATE TRIGGER tbl_au AFTER UPDATE ON tbl BEGIN
  INSERT INTO fts_idx(fts_idx, rowid, b, c) VALUES('delete', old.a, old.b, old.c);
  INSERT INTO fts_idx(rowid, b, c) VALUES (new.a, new.b, new.c);
END;


 Like contentless tables, external content tables do not support REPLACE
conflict handling. Any operations that specify REPLACE conflict handling are
handled using ABORT.


c2: SQLite FTS5 Extension
c3: 4.5. The Columnsize Option
c5: fts5.html#the_columnsize_option
c4:
Normally, FTS5 maintains a special backing table within the database that
stores the size of each column value in tokens inserted into the main FTS5
table in a separate table. This backing table is used by the
xColumnSize API function, which is in turn used by
the built-in bm25 ranking function (and is likely to be useful 
to other ranking functions as well).

In order to save space, this backing table may be omitted by setting the
columnsize option to zero. For example:

-- A table without the xColumnSize() values stored on disk:
CREATE VIRTUAL TABLE ft USING fts5(a, b, c, columnsize=0);

-- Three equivalent ways of creating a table that does store the
-- xColumnSize() values on disk:
CREATE VIRTUAL TABLE ft USING fts5(a, b, c);
CREATE VIRTUAL TABLE ft USING fts5(a, b, c, columnsize=1);
CREATE VIRTUAL TABLE ft USING fts5(a, b, columnsize='1', c);


 It is an error to set the columnsize option to any value other than
0 or 1.

 If an FTS5 table is configured with columnsize=0 but is not a
contentless table, the xColumnSize API function
still works, but runs much more slowly. In this case, instead of reading
the value to return directly from the database, it reads the text value
itself and count the tokens within it on demand.

Or, if the table is also a contentless table,
then the following apply:


   The xColumnSize API always returns -1. There is no way to determine 
       the number of tokens in a value stored within a contentless FTS5 table
       configured with columnsize=0.

   Each inserted row must be accompanied by an explicitly specified rowid
       value. If a contentless table is configured with columnsize=0,
       attempting to insert a NULL value into the rowid is an SQLITE_MISMATCH
       error.

   All queries on the table must be full-text queries. In other words,
       they must use the MATCH or = operator with the table-name column as the
       left-hand operand, or else use the table-valued function syntax. Any
       query that is not a full-text query results in an error.


 The name of the table in which the xColumnSize values are stored
(unless columnsize=0 is specified) is "&lt;name&gt;_docsize", where 
&lt;name&gt; is the name of the FTS5 table itself. The 
sqlite3_analyzer
tool may be used on an existing database in order to determine how much
space might be saved by recreating an FTS5 table using columnsize=0.


c2: SQLite FTS5 Extension
c3: 4.6. The Detail Option
c5: fts5.html#the_detail_option
c4:
For each term in a document, the FTS index maintained by FTS5 
stores the rowid of the document, the column number of the column that contains
the term and the offset of the term within the column value. The "detail"
option may be used to omit some of this information. This reduces the space
that the index consumes within the database file, but also reduces the
capability and efficiency of the system.

 The detail option may be set to "full" (the default value), "column" or
"none". For example:

-- The following two lines are equivalent (because the default value
-- of "detail" is "full". 
CREATE VIRTUAL TABLE ft1 USING fts5(a, b, c);
CREATE VIRTUAL TABLE ft1 USING fts5(a, b, c, detail=full);

CREATE VIRTUAL TABLE ft2 USING fts5(a, b, c, detail=column);
CREATE VIRTUAL TABLE ft3 USING fts5(a, b, c, detail=none);


If the detail option is set to column, then for each term the FTS
index records the rowid and column number only, omitting the term offset
information. This results in the following restrictions:


   NEAR queries are not available.
   Phrase queries are not available.
   Assuming the table is not also a 
  contentless table, the 
  xInstCount, xInst, 
  xPhraseFirst and xPhraseNext
  are slower than usual. This is because instead of reading the required data
  directly from the FTS index they have to load and tokenize the document text 
  on demand.
   If the table is also a contentless table, the xInstCount, xInst, 
  xPhraseFirst and xPhraseNext APIs behave as if the current row contains no
  phrase matches at all (i.e. xInstCount() returns 0).

  
If the detail option is set to none, then for each term the FTS
index records just the rowid is stored. Both column and offset information
are omitted. As well as the restrictions itemized above for detail=column
mode, this imposes the following extra limitations:


   Column filter queries are not available.
   Assuming the table is not also a contentless table, the 
  xPhraseFirstColumn and 
  xPhraseNextColumn are slower than usual. 

   If the table is also a contentless table, the xPhraseFirstColumn and
  xPhraseNextColumn APIs behave as if the current row contains no phrase
  matches at all (i.e. xPhraseFirstColumn() sets the iterator to EOF).


 In one test that indexed a large set of emails (1636 MiB on disk), the FTS
index was 743 MiB on disk with detail=full, 340 MiB with detail=column and 134
MiB with detail=none.


c2: SQLite FTS5 Extension
c3: 5. Auxiliary Functions
c5: fts5.html#_auxiliary_functions_
c4:
Auxiliary functions are similar to SQL scalar functions,
except that they may only be used within full-text queries (those that use
the MATCH operator) on an FTS5 table. Their results are calculated based not
only on the arguments passed to them, but also on the current match and 
matched row. For example, an auxiliary function may return a numeric value
indicating the accuracy of the match (see the bm25() function), 
or a fragment of text from the matched row that contains one or more 
instances of the search terms (see the snippet() function).

To invoke an auxiliary function, the name of the FTS5 table should be
specified as the first argument. Other arguments may follow the first,
depending on the specific auxiliary function being invoked. For example, to
invoke the "highlight" function:

SELECT highlight(email, 2, '&lt;b&gt;', '&lt;/b&gt;') FROM email WHERE email MATCH 'fts5'


The built-in auxiliary functions provided as part of FTS5 are described in
the following section. Applications may also implement 
custom auxiliary functions in C.


c2: SQLite FTS5 Extension
c3: 5.1. Built-in Auxiliary Functions
c5: fts5.html#built_in_auxiliary_functions
c4:
FTS5 provides three built-in auxiliary functions:


   The bm25() auxiliary function returns a real value
       reflecting the accuracy of the current match. Better matches are
       assigned numerically lower values.

   The highlight() auxiliary function returns a copy
       of the text from one of the columns of the current match with each
       instance of a queried term within the result surrounded by specified
       markup (for example "&lt;b&gt;" and "&lt;/b&gt;").

   The snippet() auxiliary function selects a short
       fragment of text from one of the columns of the matched row and returns
       it with each instance of a queried term surrounded by markup in
       the same manner as the highlight() function. The fragment of text is
       selected so as to maximize the number of queried terms it contains.


c2: SQLite FTS5 Extension
c3: 5.1.1. The bm25() function
c5: fts5.html#the_bm25_function
c4:
The built-in auxiliary function bm25() returns a real value indicating
how well the current row matches the full-text query. The better the match,
the numerically smaller the value returned. A query such as the following may
be used to return matches in order from best to worst match:

SELECT * FROM fts WHERE fts MATCH ? ORDER BY bm25(fts)


 In order to calculate a documents score, the full-text query is separated
    into its component phrases. The bm25 score for document D and 
    query Q is then calculated as follows:

 

 In the above, nPhrase is the number of phrases in the query.
    |D| is the number of tokens in the current document, and
    avgdl is the average number of tokens in all documents within the
    FTS5 table.  k1 and b are both constants,
    hard-coded at 1.2 and 0.75 respectively.

 The "-1" term at the start of the formula is not found in most
implementations of the BM25 algorithm. Without it, a better match is assigned
a numerically higher BM25 score. Since the default sorting order is
"ascending", this means that appending "ORDER BY bm25(fts)" to a query would
cause results to be returned in order from worst to best. The "DESC" keyword
would be required in order to return the best matches first. In order to
avoid this pitfall, the FTS5 implementation of BM25 multiplies the result
by -1 before returning it, ensuring that better matches are assigned
numerically lower scores.

 IDF(qi) is the inverse-document-frequency of query 
    phrase i. It is calculated as follows, where N is the total
    number of rows in the FTS5 table and n(qi) is the total
    number of rows that contain at least one instance of phrase i:

 

 Finally, f(qi,D) is the phrase frequency of phrase 
i. By default, this is simply the number of occurrences of the phrase
within the current row. However, by passing extra real value arguments to 
the bm25() SQL function, each column of the table may be assigned a different
weight and the phrase frequency calculated as follows:

 

 where wc is the weight assigned to column c and
n(qi,c) is the number of occurrences of phrase i in
column c of the current row. The first argument passed to bm25()
following the table name is the weight assigned to the leftmost column of
the FTS5 table. The second is the weight assigned to the second leftmost
column, and so on. If there are not enough arguments for all table columns,
remaining columns are assigned a weight of 1.0. If there are too many 
trailing arguments, the extras are ignored. For example:

-- Assuming the following schema:
CREATE VIRTUAL TABLE email USING fts5(sender, title, body);

-- Return results in bm25 order, with each phrase hit in the "sender"
-- column considered the equal of 10 hits in the "body" column, and
-- each hit in the "title" column considered as valuable as 5 hits in
-- the "body" column.
SELECT * FROM email WHERE email MATCH ? ORDER BY bm25(email, 10.0, 5.0);


Refer to wikipedia for 
more information regarding
BM25 and its variants.


c2: SQLite FTS5 Extension
c3: 5.1.2. The highlight() function
c5: fts5.html#the_highlight_function
c4:
The highlight() function returns a copy of the text from a specified 
column of the current row with extra markup text inserted to mark the start 
and end of phrase matches. 

The highlight() must be invoked with exactly three arguments following 
the table name. To be interpreted as follows:


   An integer indicating the index of the FTS table column to read the 
       text from. Columns are numbered from left to right starting at zero.

   The text to insert before each phrase match.

   The text to insert after each phrase match.


For example:

-- Return a copy of the text from the leftmost column of the current
-- row, with phrase matches marked using html "b" tags.
SELECT highlight(fts, 0, '&lt;b&gt;', '&lt;/b&gt;') FROM fts WHERE fts MATCH ?


In cases where two or more phrase instances overlap (share one or more
tokens in common), a single open and close marker is inserted for each set
of overlapping phrases. For example:

-- Assuming this:
CREATE VIRTUAL TABLE ft USING fts5(a);
INSERT INTO ft VALUES('a b c x c d e');
INSERT INTO ft VALUES('a b c c d e');
INSERT INTO ft VALUES('a b c d e');

-- The following SELECT statement returns these three rows:
--   '&#91;a b c&#93; x &#91;c d e&#93;'
--   '&#91;a b c&#93; &#91;c d e&#93;'
--   '&#91;a b c d e&#93;'
SELECT highlight(ft, 0, '&#91;', '&#93;') FROM ft WHERE ft MATCH 'a+b+c AND c+d+e';


c2: SQLite FTS5 Extension
c3: 5.1.3. The snippet() function
c5: fts5.html#the_snippet_function
c4:
The snippet() function is similar to highlight(), except that instead of
returning entire column values, it automatically selects and extracts a
short fragment of document text to process and return. The snippet() function
must be passed five parameters following the table name argument:


   An integer indicating the index of the FTS table column to select
       the returned text from. Columns are numbered from left to right 
       starting at zero. A negative value indicates that the column should
       be automatically selected.

   The text to insert before each phrase match within the returned text.

   The text to insert after each phrase match within the returned text.

   The text to add to the start or end of the selected text to indicate
       that the returned text does not occur at the start or end of its column,
       respectively.

   The maximum number of tokens in the returned text. This must be greater
       than zero and equal to or less than 64.


c2: SQLite FTS5 Extension
c3: 5.2. Sorting by Auxiliary Function Results
c5: fts5.html#sorting_by_auxiliary_function_results
c4:
All FTS5 tables feature a special hidden column named "rank". If the
current query is not a full-text query (i.e. if it does not include a MATCH
operator), the value of the "rank" column is always NULL. Otherwise, in a
full-text query, column rank contains by default the same value as would be
returned by executing the bm25() auxiliary function with no trailing 
arguments.

 The difference between reading from the rank column and using the bm25()
function directly within the query is only significant when sorting by the
returned value. In this case, using "rank" is faster than using bm25().

-- The following queries are logically equivalent. But the second may
-- be faster, particularly if the caller abandons the query before
-- all rows have been returned (or if the queries were modified to 
-- include LIMIT clauses).
SELECT * FROM fts WHERE fts MATCH ? ORDER BY bm25(fts);
SELECT * FROM fts WHERE fts MATCH ? ORDER BY rank;


 Instead of using bm25() with no trailing arguments, the specific auxiliary
function mapped to the rank column may be configured either on a per-query
basis, or by setting a different persistent default for the FTS table.

 In order to change the mapping of the rank column for a single query, 
a term similar to either of the following is added to the WHERE clause of a 
query:

rank MATCH 'auxiliary-function-name(arg1, arg2, ...)'
rank = 'auxiliary-function-name(arg1, arg2, ...)'


 The right-hand-side of the MATCH or = operator must be a constant
expression that evaluates to a string consisting of the auxiliary function to
invoke, followed by zero or more comma separated arguments within parenthesis.
Arguments must be SQL literals. For example:

-- The following queries are logically equivalent. But the second may
-- be faster. See above. 
SELECT * FROM fts WHERE fts MATCH ? ORDER BY bm25(fts, 10.0, 5.0);
SELECT * FROM fts WHERE fts MATCH ? AND rank MATCH 'bm25(10.0, 5.0)' ORDER BY rank;


 The table-valued function syntax may also be used to specify an alternative
ranking function. In this case the text describing the ranking function should
be specified as the second table-valued function argument. The following three
queries are equivalent:

SELECT * FROM fts WHERE fts MATCH ? AND rank MATCH 'bm25(10.0, 5.0)' ORDER BY rank;
SELECT * FROM fts WHERE fts = ? AND rank = 'bm25(10.0, 5.0)' ORDER BY rank;
SELECT * FROM fts WHERE fts(?, 'bm25(10.0, 5.0)') ORDER BY rank;


 The default mapping of the rank column for a table may be modified 
using the FTS5 rank configuration option.


c2: SQLite FTS5 Extension
c3: 6.1. The 'automerge' Configuration Option
c5: fts5.html#the_automerge_configuration_option
c4:
Instead of using a single data structure on disk to store the full-text
  index, FTS5 uses a series of b-trees. Each time a new transaction is
  committed, a new b-tree containing the contents of the committed transaction
  is written into the database file. When the full-text index is queried, each
  b-tree must be queried individually and the results merged before being
  returned to the user.


  In order to prevent the number of b-trees in the database from becoming too
  large (slowing down queries), smaller b-trees are periodically merged into
  single larger b-trees containing the same data. By default, this happens
  automatically within INSERT, UPDATE or DELETE statements that modify the
  full-text index. The 'automerge' parameter determines how many smaller
  b-trees are merged together at a time. Setting it to a small value can
  speed up queries (as they have to query and merge the results from fewer 
  b-trees), but can also slow down writing to the database (as each INSERT,
  UPDATE or DELETE statement has to do more work as part of the automatic
  merging process).


  Each of the b-trees that make up the full-text index is assigned to a "level"
  based on its size. Level-0 b-trees are the smallest, as they contain the
  contents of a single transaction. Higher level b-trees are the result of
  merging two or more level-0 b-trees together and so they are larger. FTS5
  begins to merge b-trees together once there exist M or more b-trees 
  with the same level, where M is the value of the 'automerge' 
  parameter.


  The maximum allowed value for the 'automerge' parameter is 16. The default
  value is 4. Setting the 'automerge' parameter to 0 disables the automatic 
  incremental merging of b-trees altogether.

INSERT INTO ft(ft, rank) VALUES('automerge', 8);


c2: SQLite FTS5 Extension
c3: 6.2. The 'crisismerge' Configuration Option
c5: fts5.html#the_crisismerge_configuration_option
c4:
The 'crisismerge' option is similar to 'automerge', in that it determines
how and how often the component b-trees that make up the full-text index are
merged together. Once there exist C or more b-trees on a single level
within the full-text index, where C is the value of the 'crisismerge'
option, all b-trees on the level are immediately merged into a single b-tree.

The difference between this option and the 'automerge' option is that when
the 'automerge' limit is reached FTS5 only begins to merge the b-trees
together. Most of the work is performed as part of subsequent INSERT, 
UPDATE or DELETE operations. Whereas when the 'crisismerge' limit is reached,
the offending b-trees are all merged immediately. This means that an INSERT,
UPDATE or DELETE that triggers a crisis-merge may take a long time to 
complete.

The default 'crisismerge' value is 16. There is no maximum limit. Attempting
to set the 'crisismerge' parameter to a value of 0 or 1 is equivalent to
setting it to the default value (16). It is an error to attempt to set the
'crisismerge' option to a negative value.

INSERT INTO ft(ft, rank) VALUES('crisismerge', 16);


c2: SQLite FTS5 Extension
c3: 6.3. The 'delete' Command
c5: fts5.html#the_delete_command
c4:
This command is only available with external content and contentless tables. It
is used to delete the index entries associated with a single row from the
full-text index. This command and the delete-all
command are the only ways to remove entries from the full-text index of a
contentless table.

 In order to use this command to delete a row, the text value 'delete' 
must be inserted into the special column with the same name as the table.
The rowid of the row to delete is inserted into the rowid column. The
values inserted into the other columns must match the values currently
stored in the table. For example:

-- Insert a row with rowid=14 into the fts5 table.
INSERT INTO ft(rowid, a, b, c) VALUES(14, $a, $b, $c);

-- Remove the same row from the fts5 table.
INSERT INTO ft(ft, rowid, a, b, c) VALUES('delete', 14, $a, $b, $c);


 If the values "inserted" into the text columns as part of a 'delete'
command are not the same as those currently stored within the table, the
results may be unpredictable.

 The reason for this is easy to understand: When a document is inserted
into the FTS5 table, an entry is added to the full-text index to record the
position of each token within the new document. When a document is removed,
the original data is required in order to determine the set of entries that
need to be removed from the full-text index. So if the data supplied to FTS5
when a row is deleted using this command is different from that used to
determine the set of token instances when it was inserted, some full-text 
index entries may not be correctly deleted, or FTS5 may try to remove index 
entries that do not exist. This can leave the full-text index in an
unpredictable state, making future query results unreliable.


c2: SQLite FTS5 Extension
c3: 6.4. The 'delete-all' Command
c5: fts5.html#the_delete_all_command
c4:
This command is only available with external content and contentless tables. It
deletes all entries from the full-text index.

INSERT INTO ft(ft) VALUES('delete-all');


c2: SQLite FTS5 Extension
c3: 6.5. The 'integrity-check' Command
c5: fts5.html#the_integrity_check_command
c4:
This command is used to verify that the full-text index is consistent 
with the contents of the FTS5 table or content 
table. It is not available with contentless tables.

The integrity-check command is invoked by inserting the text value
'integrity-check' into the special column with the same name as the FTS5
table. For example:

INSERT INTO ft(ft) VALUES('integrity-check');


If the full-text index is consistent with the contents of the table, the
INSERT used to invoke the integrity-check command succeeds. Or, if any
discrepancy is found, it fails with an SQLITE_CORRUPT_VTAB error.


c2: SQLite FTS5 Extension
c3: 6.6. The 'merge' Command
c5: fts5.html#the_merge_command
c4:
INSERT INTO ft(ft, rank) VALUES('merge', 500);


 This command merges b-tree structures together until roughly N pages
of merged data have been written to the database, where N is the absolute
value of the parameter specified as part of the 'merge' command. The size of
each page is as configured by the FTS5 pgsz option.

 If the parameter is a positive value, B-tree structures are only eligible
for merging if one of the following is true:


   There are U or more such b-trees on a
       single level (see the documentation for the FTS5 automerge option
       for an explanation of b-tree levels), where U is the value assigned
       to the FTS5 usermerge option option.
   A merge has already been started (perhaps by a 'merge' command that
       specified a negative parameter).


 It is possible to tell whether or not the 'merge' command found any 
b-trees to merge together by checking the value returned by the
sqlite3_total_changes() API before and after the command is executed. If
the difference between the two values is 2 or greater, then work was performed.
If the difference is less than 2, then the 'merge' command was a no-op. In this
case there is no reason to execute the same 'merge' command again, at least
until after the FTS table is next updated.

 If the parameter is negative, and there are B-tree structures on more than
one level within the FTS index, all B-tree structures are assigned to the same
level before the merge operation is commenced. Additionally, if the parameter
is negative, the value of the usermerge configuration option is not 
respected - as few as two b-trees from the same level may be merged together.

 The above means that executing the 'merge' command with a negative
parameter until the before and after difference in the return value of
sqlite3_total_changes() is less than two optimizes the FTS index in the
same way as the FTS5 optimize command. However, if a new b-tree is added
to the FTS index while this process is ongoing, FTS5 will move the new 
b-tree to the same level as the existing b-trees and restart the merge. To
avoid this, only the first call to 'merge' should specify a negative parameter.
Each subsequent call to 'merge' should specify a positive value so that the
merge started by the first call is run to completion even if new b-trees are
added to the FTS index.


c2: SQLite FTS5 Extension
c3: 6.7. The 'optimize' Command
c5: fts5.html#the_optimize_command
c4:
This command merges all individual b-trees that currently make up the
full-text index into a single large b-tree structure. This ensures that the
full-text index consumes the minimum space within the database and is in the
fastest form to query.

Refer to the documentation for the FTS5 automerge option for more details
regarding the relationship between the full-text index and its component
b-trees.

INSERT INTO ft(ft) VALUES('optimize');


Because it reorganizes the entire FTS index, the optimize command can 
take a long time to run. The FTS5 merge command can be used to divide
the work of optimizing the FTS index into multiple steps. To do this:


   Invoke the 'merge' command once with the parameter set to -N, then
   Invoke the 'merge' command zero or more times with the parameter set to N.


where N is the number of pages of data to merge within each invocation of
the merge command. The application should stop invoking merge when the
difference in the value returned by the sqlite3_total_changes() function before
and after the merge command drops to below two. The merge commands may be
issued as part of the same or separate transactions, and by the same or
different database clients. Refer to the documentation for the 
merge command for further details.


c2: SQLite FTS5 Extension
c3: 6.8. The 'pgsz' Configuration Option
c5: fts5.html#the_pgsz_configuration_option
c4:
This command is used to set the persistent "pgsz" option.

 The full-text index maintained by FTS5 is stored as a series of fixed-size
blobs in a database table. It is not strictly necessary for all blobs that make
up a full-text index to be the same size. The pgsz option determines the size
of all blobs created by subsequent index writers. The default value is 1000.

INSERT INTO ft(ft, rank) VALUES('pgsz', 4072);


c2: SQLite FTS5 Extension
c3: 6.9. The 'rank' Configuration Option
c5: fts5.html#the_rank_configuration_option
c4:
This command is used to set the persistent "rank" option.

 The rank option is used to change the default auxiliary function mapping
for the rank column. The option should be set to a text value in the same
format as described for "rank MATCH ?" terms 
above. For example:

INSERT INTO ft(ft, rank) VALUES('rank', 'bm25(10.0, 5.0)');


c2: SQLite FTS5 Extension
c3: 6.10. The 'rebuild' Command
c5: fts5.html#the_rebuild_command
c4:
This command first deletes the entire full-text index, then rebuilds it
based on the contents of the table or content
table.  It is not available with contentless
tables.

INSERT INTO ft(ft) VALUES('rebuild');


c2: SQLite FTS5 Extension
c3: 6.11. The 'usermerge' Configuration Option
c5: fts5.html#the_usermerge_configuration_option
c4:
This command is used to set the persistent "usermerge" option.

 The usermerge option is similar to the automerge and crisismerge options.
It is the minimum number of b-tree segments that will be merged together by
a 'merge' command with a positive parameter. For example:

INSERT INTO ft(ft, rank) VALUES('usermerge', 4);


 The default value of the usermerge option is 4. The minimum allowed value
is 2, and the maximum 16.


c2: SQLite FTS5 Extension
c3: 7. Extending FTS5
c5: fts5.html#extending_fts5
c4:
FTS5 features APIs allowing it to be extended by:


   Adding new auxiliary functions implemented in C, and
   Adding new tokenizers, also implemented in C.


 The built-in tokenizers and auxiliary functions described in this
document are all implemented using the publicly available API described
below.

 Before a new auxiliary function or tokenizer implementation may be 
registered with FTS5, an application must obtain a pointer to the "fts5_api"
structure. There is one fts5_api structure for each database connection with
which the FTS5 extension is registered. To obtain the pointer, the application
invokes the SQL user-defined function fts5() with a single argument.  That
argument must be set to a pointer to a pointer to an fts5_api object
using the sqlite3_bind_pointer() interface.
The following example code demonstrates the technique:

/*
** Return a pointer to the fts5_api pointer for database connection db.
** If an error occurs, return NULL and leave an error in the database 
** handle (accessible using sqlite3_errcode()/errmsg()).
*/
fts5_api *fts5_api_from_db(sqlite3 *db){
  fts5_api *pRet = 0;
  sqlite3_stmt *pStmt = 0;

  if( SQLITE_OK==sqlite3_prepare(db, "SELECT fts5(?1)", -1, &pStmt, 0) ){
    sqlite3_bind_pointer(pStmt, (void*)&pRet, "fts5_api_ptr", NULL);
    sqlite3_step(pStmt);
  }
  sqlite3_finalize(pStmt);
  return pRet;
}


Backwards Compatibility Warning:
Prior to SQLite version 3.20.0 (2017-08-01), the fts5() worked slightly
differently.  Older applications that extend FTS5 must be revised to use 
the new technique shown above.

 The fts5_api structure is defined as follows. It exposes three methods, 
one each for registering new auxiliary functions and tokenizers, and one for
retrieving existing tokenizer. The latter is intended to facilitate the
implementation of "tokenizer wrappers" similar to the built-in
porter tokenizer.

typedef struct fts5_api fts5_api;
struct fts5_api {
  int iVersion;                   /* Currently always set to 2 */

  /* Create a new tokenizer */
  int (*xCreateTokenizer)(
    fts5_api *pApi,
    const char *zName,
    void *pContext,
    fts5_tokenizer *pTokenizer,
    void (*xDestroy)(void*)
  );

  /* Find an existing tokenizer */
  int (*xFindTokenizer)(
    fts5_api *pApi,
    const char *zName,
    void **ppContext,
    fts5_tokenizer *pTokenizer
  );

  /* Create a new auxiliary function */
  int (*xCreateFunction)(
    fts5_api *pApi,
    const char *zName,
    void *pContext,
    fts5_extension_function xFunction,
    void (*xDestroy)(void*)
  );
};


 To invoke a method of the fts5_api object, the fts5_api pointer itself
should be passed as the methods first argument followed by the other, method
specific, arguments. For example:

rc = pFts5Api->xCreateTokenizer(pFts5Api, ... other args ...);


 The fts5_api structure methods are described individually in the following
sections.


c2: SQLite FTS5 Extension
c3: 7.1. Custom Tokenizers
c5: fts5.html#custom_tokenizers
c4:
To create a custom tokenizer, an application must implement three
functions: a tokenizer constructor (xCreate), a destructor (xDelete) and a
function to do the actual tokenization (xTokenize). The type of each
function is as for the member variables of the fts5_tokenizer struct:

typedef struct Fts5Tokenizer Fts5Tokenizer;
typedef struct fts5_tokenizer fts5_tokenizer;
struct fts5_tokenizer {
  int (*xCreate)(void*, const char **azArg, int nArg, Fts5Tokenizer **ppOut);
  void (*xDelete)(Fts5Tokenizer*);
  int (*xTokenize)(Fts5Tokenizer*, 
      void *pCtx,
      int flags,            /* Mask of FTS5_TOKENIZE_* flags */
      const char *pText, int nText, 
      int (*xToken)(
        void *pCtx,         /* Copy of 2nd argument to xTokenize() */
        int tflags,         /* Mask of FTS5_TOKEN_* flags */
        const char *pToken, /* Pointer to buffer containing token */
        int nToken,         /* Size of token in bytes */
        int iStart,         /* Byte offset of token within input text */
        int iEnd            /* Byte offset of end of token within input text */
      )
  );
};

/* Flags that may be passed as the third argument to xTokenize() */
#define FTS5_TOKENIZE_QUERY     0x0001
#define FTS5_TOKENIZE_PREFIX    0x0002
#define FTS5_TOKENIZE_DOCUMENT  0x0004
#define FTS5_TOKENIZE_AUX       0x0008

/* Flags that may be passed by the tokenizer implementation back to FTS5
** as the third argument to the supplied xToken callback. */
#define FTS5_TOKEN_COLOCATED    0x0001      /* Same position as prev. token */


 The implementation is registered with the FTS5 module by calling the
xCreateTokenizer() method of the fts5_api object. If there is already a
tokenizer with the same name, it is replaced.  If a non-NULL xDestroy parameter
is passed to xCreateTokenizer(), it is invoked with a copy of the pContext
pointer passed as the only argument when the database handle is closed or when
the tokenizer is replaced.

 If successful, xCreateTokenizer() returns SQLITE_OK. Otherwise, it
returns an SQLite error code. In this case the xDestroy function is not 
invoked.

 When an FTS5 table uses the custom tokenizer, the FTS5 core calls xCreate()
once to create a tokenizer, then xTokenize() zero or more times to tokenize
strings, then xDelete() to free any resources allocated by xCreate(). More
specifically:


xCreate:
   This function is used to allocate and initialize a tokenizer instance.
   A tokenizer instance is required to actually tokenize text.

   The first argument passed to this function is a copy of the (void*)
   pointer provided by the application when the fts5_tokenizer object
   was registered with FTS5 (the third argument to xCreateTokenizer()). 
   The second and third arguments are an array of nul-terminated strings
   containing the tokenizer arguments, if any, specified following the
   tokenizer name as part of the CREATE VIRTUAL TABLE statement used
   to create the FTS5 table.

   The final argument is an output variable. If successful, (*ppOut) 
   should be set to point to the new tokenizer handle and SQLITE_OK
   returned. If an error occurs, some value other than SQLITE_OK should
   be returned. In this case, fts5 assumes that the final value of *ppOut 
   is undefined.

 xDelete:
   This function is invoked to delete a tokenizer handle previously
   allocated using xCreate(). Fts5 guarantees that this function will
   be invoked exactly once for each successful call to xCreate().

 xTokenize:
   This function is expected to tokenize the nText byte string indicated 
   by argument pText. pText may or may not be nul-terminated. The first
   argument passed to this function is a pointer to an Fts5Tokenizer object
   returned by an earlier call to xCreate().

   The second argument indicates the reason that FTS5 is requesting
   tokenization of the supplied text. This is always one of the following
   four values:

    FTS5_TOKENIZE_DOCUMENT - A document is being inserted into
            or removed from the FTS table. The tokenizer is being invoked to
            determine the set of tokens to add to (or delete from) the
            FTS index.

        FTS5_TOKENIZE_QUERY - A MATCH query is being executed 
            against the FTS index. The tokenizer is being called to tokenize 
            a bareword or quoted string specified as part of the query.

        (FTS5_TOKENIZE_QUERY | FTS5_TOKENIZE_PREFIX) - Same as
            FTS5_TOKENIZE_QUERY, except that the bareword or quoted string is
            followed by a "*" character, indicating that the last token
            returned by the tokenizer will be treated as a token prefix.

        FTS5_TOKENIZE_AUX - The tokenizer is being invoked to 
            satisfy an fts5_api.xTokenize() request made by an auxiliary
            function. Or an fts5_api.xColumnSize() request made by the same
            on a columnsize=0 database.  
   

   For each token in the input string, the supplied callback xToken() must
   be invoked. The first argument to it should be a copy of the pointer
   passed as the second argument to xTokenize(). The third and fourth
   arguments are a pointer to a buffer containing the token text, and the
   size of the token in bytes. The 4th and 5th arguments are the byte offsets
   of the first byte of and first byte immediately following the text from
   which the token is derived within the input.

   The second argument passed to the xToken() callback ("tflags") should
   normally be set to 0. The exception is if the tokenizer supports 
   synonyms. In this case see the discussion below for details.

   FTS5 assumes the xToken() callback is invoked for each token in the 
   order that they occur within the input text.

   If an xToken() callback returns any value other than SQLITE_OK, then
   the tokenization should be abandoned and the xTokenize() method should
   immediately return a copy of the xToken() return value. Or, if the
   input buffer is exhausted, xTokenize() should return SQLITE_OK. Finally,
   if an error occurs with the xTokenize() implementation itself, it
   may abandon the tokenization and return any error code other than
   SQLITE_OK or SQLITE_DONE.


c2: SQLite FTS5 Extension
c3: 7.1.1. Synonym Support
c5: fts5.html#synonym_support
c4:
Custom tokenizers may also support synonyms. Consider a case in which a
   user wishes to query for a phrase such as "first place". Using the 
   built-in tokenizers, the FTS5 query 'first + place' will match instances
   of "first place" within the document set, but not alternative forms
   such as "1st place". In some applications, it would be better to match
   all instances of "first place" or "1st place" regardless of which form
   the user specified in the MATCH query text.

   There are several ways to approach this in FTS5:

    By mapping all synonyms to a single token. In this case, using
            the above example, this means that the tokenizer returns the
            same token for inputs "first" and "1st". Say that token is in
            fact "first", so that when the user inserts the document "I won
            1st place" entries are added to the index for tokens "i", "won",
            "first" and "place". If the user then queries for '1st + place',
            the tokenizer substitutes "first" for "1st" and the query works
            as expected.

        By querying the index for all synonyms of each query term
            separately. In this case, when tokenizing query text, the
            tokenizer may provide multiple synonyms for a single term 
            within the document. FTS5 then queries the index for each 
            synonym individually. For example, faced with the query:

   ... MATCH 'first place'


            the tokenizer offers both "1st" and "first" as synonyms for the
            first token in the MATCH query and FTS5 effectively runs a query 
            similar to:

   ... MATCH '(first OR 1st) place'


            except that, for the purposes of auxiliary functions, the query
            still appears to contain just two phrases - "(first OR 1st)" 
            being treated as a single phrase.

        By adding multiple synonyms for a single term to the FTS index.
            Using this method, when tokenizing document text, the tokenizer
            provides multiple synonyms for each token. So that when a 
            document such as "I won first place" is tokenized, entries are
            added to the FTS index for "i", "won", "first", "1st" and
            "place".

            This way, even if the tokenizer does not provide synonyms
            when tokenizing query text (it should not - to do so would be
            inefficient), it doesn't matter if the user queries for 
            'first + place' or '1st + place', as there are entries in the
            FTS index corresponding to both forms of the first token.
   

   Whether it is parsing document or query text, any call to xToken that
   specifies a tflags argument with the FTS5_TOKEN_COLOCATED bit
   is considered to supply a synonym for the previous token. For example,
   when parsing the document "I won first place", a tokenizer that supports
   synonyms would call xToken() 5 times, as follows:

   xToken(pCtx, 0, "i",                      1,  0,  1);
xToken(pCtx, 0, "won",                    3,  2,  5);
xToken(pCtx, 0, "first",                  5,  6, 11);
xToken(pCtx, FTS5_TOKEN_COLOCATED, "1st", 3,  6, 11);
xToken(pCtx, 0, "place",                  5, 12, 17);


   It is an error to specify the FTS5_TOKEN_COLOCATED flag the first time
   xToken() is called. Multiple synonyms may be specified for a single token
   by making multiple calls to xToken(FTS5_TOKEN_COLOCATED) in sequence. 
   There is no limit to the number of synonyms that may be provided for a
   single token.

   In many cases, method (1) above is the best approach. It does not add 
   extra data to the FTS index or require FTS5 to query for multiple terms,
   so it is efficient in terms of disk space and query speed. However, it
   does not support prefix queries very well. If, as suggested above, the
   token "first" is substituted for "1st" by the tokenizer, then the query:

   ... MATCH '1s*'


   will not match documents that contain the token "1st" (as the tokenizer
   will probably not map "1s" to any prefix of "first").

   For full prefix support, method (3) may be preferred. In this case, 
   because the index contains entries for both "first" and "1st", prefix
   queries such as 'fi*' or '1s*' will match correctly. However, because
   extra entries are added to the FTS index, this method uses more space
   within the database.

   Method (2) offers a midpoint between (1) and (3). Using this method,
   a query such as '1s*' will match documents that contain the literal 
   token "1st", but not "first" (assuming the tokenizer is not able to
   provide synonyms for prefixes). However, a non-prefix query like '1st'
   will match against "1st" and "first". This method does not require
   extra disk space, as no extra entries are added to the FTS index. 
   On the other hand, it may require more CPU cycles to run MATCH queries,
   as separate queries of the FTS index are required for each synonym.

   When using methods (2) or (3), it is important that the tokenizer only
   provide synonyms when tokenizing document text (method (2)) or query
   text (method (3)), not both. Doing so will not cause any errors, but is
   inefficient.


c2: SQLite FTS5 Extension
c3: 7.2. Custom Auxiliary Functions
c5: fts5.html#custom_auxiliary_functions
c4:
Implementing a custom auxiliary function is similar to implementing a
scalar SQL function. The implementation
should be a C function of type fts5_extension_function, defined as follows:

typedef struct Fts5ExtensionApi Fts5ExtensionApi;
typedef struct Fts5Context Fts5Context;
typedef struct Fts5PhraseIter Fts5PhraseIter;

typedef void (*fts5_extension_function)(
  const Fts5ExtensionApi *pApi,   /* API offered by current FTS version */
  Fts5Context *pFts,              /* First arg to pass to pApi functions */
  sqlite3_context *pCtx,          /* Context for returning result/error */
  int nVal,                       /* Number of values in apVal[] array */
  sqlite3_value **apVal           /* Array of trailing arguments */
);


 The implementation is registered with the FTS5 module by calling the
xCreateFunction() method of the fts5_api object. If there is already an
auxiliary function with the same name, it is replaced by the new function.
If a non-NULL xDestroy parameter is passed to xCreateFunction(), it is invoked
with a copy of the pContext pointer passed as the only argument when the
database handle is closed or when the registered auxiliary function is
replaced.

 If successful, xCreateFunction() returns SQLITE_OK. Otherwise, it
returns an SQLite error code. In this case the xDestroy function is not 
invoked.

 The final three arguments passed to the auxiliary function callback are
similar to the three arguments passed to the implementation of a scalar SQL
function. All arguments except the first passed to the auxiliary function are
available to the implementation in the apVal&#91;&#93; array. The
implementation should return a result or error via the content handle pCtx.

 The first argument passed to an auxiliary function callback is a pointer
to a structure containing methods that may be invoked in order to obtain
information regarding the current query or row. The second argument is an
opaque handle that should be passed as the first argument to any such method 
invocation. For example, the following auxiliary function definition returns
the total number of tokens in all columns of the current row:

/*
** Implementation of an auxiliary function that returns the number
** of tokens in the current row (including all columns).
*/
static void column_size_imp(
  const Fts5ExtensionApi *pApi,
  Fts5Context *pFts,
  sqlite3_context *pCtx,
  int nVal,
  sqlite3_value **apVal
){
  int rc;
  int nToken;
  rc = pApi->xColumnSize(pFts, -1, &nToken);
  if( rc==SQLITE_OK ){
    sqlite3_result_int(pCtx, nToken);
  }else{
    sqlite3_result_error_code(pCtx, rc);
  }
}


The following section describes the API offered to auxiliary function
implementations in detail. Further examples may be found in the "fts5_aux.c"
file of the source code.


c2: SQLite FTS5 Extension
c3: 7.2.1. Custom Auxiliary Functions API Reference
c5: fts5.html#_custom_auxiliary_functions_api_reference_
c4:
struct Fts5ExtensionApi {
  int iVersion;                   /* Currently always set to 3 */

  void *(*xUserData)(Fts5Context*);

  int (*xColumnCount)(Fts5Context*);
  int (*xRowCount)(Fts5Context*, sqlite3_int64 *pnRow);
  int (*xColumnTotalSize)(Fts5Context*, int iCol, sqlite3_int64 *pnToken);

  int (*xTokenize)(Fts5Context*, 
    const char *pText, int nText, /* Text to tokenize */
    void *pCtx,                   /* Context passed to xToken() */
    int (*xToken)(void*, int, const char*, int, int, int)       /* Callback */
  );

  int (*xPhraseCount)(Fts5Context*);
  int (*xPhraseSize)(Fts5Context*, int iPhrase);

  int (*xInstCount)(Fts5Context*, int *pnInst);
  int (*xInst)(Fts5Context*, int iIdx, int *piPhrase, int *piCol, int *piOff);

  sqlite3_int64 (*xRowid)(Fts5Context*);
  int (*xColumnText)(Fts5Context*, int iCol, const char **pz, int *pn);
  int (*xColumnSize)(Fts5Context*, int iCol, int *pnToken);

  int (*xQueryPhrase)(Fts5Context*, int iPhrase, void *pUserData,
    int(*)(const Fts5ExtensionApi*,Fts5Context*,void*)
  );
  int (*xSetAuxdata)(Fts5Context*, void *pAux, void(*xDelete)(void*));
  void *(*xGetAuxdata)(Fts5Context*, int bClear);

  int (*xPhraseFirst)(Fts5Context*, int iPhrase, Fts5PhraseIter*, int*, int*);
  void (*xPhraseNext)(Fts5Context*, Fts5PhraseIter*, int *piCol, int *piOff);

  int (*xPhraseFirstColumn)(Fts5Context*, int iPhrase, Fts5PhraseIter*, int*);
  void (*xPhraseNextColumn)(Fts5Context*, Fts5PhraseIter*, int *piCol);
};




void *(*xUserData)(Fts5Context*)

Return a copy of the context pointer the extension function was 
   registered with.



int (*xColumnTotalSize)(Fts5Context*, int iCol, sqlite3_int64 *pnToken)

If parameter iCol is less than zero, set output variable *pnToken
   to the total number of tokens in the FTS5 table. Or, if iCol is
   non-negative but less than the number of columns in the table, return
   the total number of tokens in column iCol, considering all rows in 
   the FTS5 table.



   If parameter iCol is greater than or equal to the number of columns
   in the table, SQLITE_RANGE is returned. Or, if an error occurs (e.g.
   an OOM condition or IO error), an appropriate SQLite error code is 
   returned.



int (*xColumnCount)(Fts5Context*)

Return the number of columns in the table.



int (*xColumnSize)(Fts5Context*, int iCol, int *pnToken)

If parameter iCol is less than zero, set output variable *pnToken
   to the total number of tokens in the current row. Or, if iCol is
   non-negative but less than the number of columns in the table, set
   *pnToken to the number of tokens in column iCol of the current row.



   If parameter iCol is greater than or equal to the number of columns
   in the table, SQLITE_RANGE is returned. Or, if an error occurs (e.g.
   an OOM condition or IO error), an appropriate SQLite error code is 
   returned.



   This function may be quite inefficient if used with an FTS5 table
   created with the "columnsize=0" option.



int (*xColumnText)(Fts5Context*, int iCol, const char **pz, int *pn)

This function attempts to retrieve the text of column iCol of the
   current document. If successful, (*pz) is set to point to a buffer
   containing the text in utf-8 encoding, (*pn) is set to the size in bytes
   (not characters) of the buffer and SQLITE_OK is returned. Otherwise,
   if an error occurs, an SQLite error code is returned and the final values
   of (*pz) and (*pn) are undefined.



int (*xPhraseCount)(Fts5Context*)

Returns the number of phrases in the current query expression.



int (*xPhraseSize)(Fts5Context*, int iPhrase)

Returns the number of tokens in phrase iPhrase of the query. Phrases
   are numbered starting from zero.



int (*xInstCount)(Fts5Context*, int *pnInst)

Set *pnInst to the total number of occurrences of all phrases within
   the query within the current row. Return SQLITE_OK if successful, or
   an error code (i.e. SQLITE_NOMEM) if an error occurs.



   This API can be quite slow if used with an FTS5 table created with the
   "detail=none" or "detail=column" option. If the FTS5 table is created 
   with either "detail=none" or "detail=column" and "content=" option 
   (i.e. if it is a contentless table), then this API always returns 0.



int (*xInst)(Fts5Context*, int iIdx, int *piPhrase, int *piCol, int *piOff)

Query for the details of phrase match iIdx within the current row.
   Phrase matches are numbered starting from zero, so the iIdx argument
   should be greater than or equal to zero and smaller than the value
   output by xInstCount().



   Usually, output parameter *piPhrase is set to the phrase number, *piCol
   to the column in which it occurs and *piOff the token offset of the
   first token of the phrase. Returns SQLITE_OK if successful, or an error
   code (i.e. SQLITE_NOMEM) if an error occurs.



   This API can be quite slow if used with an FTS5 table created with the
   "detail=none" or "detail=column" option.



sqlite3_int64 (*xRowid)(Fts5Context*)

Returns the rowid of the current row.



int (*xTokenize)(Fts5Context*, 
    const char *pText, int nText, 
    void *pCtx,                   
    int (*xToken)(void*, int, const char*, int, int, int)       
)

Tokenize text using the tokenizer belonging to the FTS5 table.



int (*xQueryPhrase)(Fts5Context*, int iPhrase, void *pUserData,
    int(*)(const Fts5ExtensionApi*,Fts5Context*,void*)
)

This API function is used to query the FTS table for phrase iPhrase
   of the current query. Specifically, a query equivalent to:


... FROM ftstable WHERE ftstable MATCH $p ORDER BY rowid



   with $p set to a phrase equivalent to the phrase iPhrase of the
   current query is executed. Any column filter that applies to
   phrase iPhrase of the current query is included in $p. For each 
   row visited, the callback function passed as the fourth argument 
   is invoked. The context and API objects passed to the callback 
   function may be used to access the properties of each matched row.
   Invoking Api.xUserData() returns a copy of the pointer passed as 
   the third argument to pUserData.



   If the callback function returns any value other than SQLITE_OK, the
   query is abandoned and the xQueryPhrase function returns immediately.
   If the returned value is SQLITE_DONE, xQueryPhrase returns SQLITE_OK.
   Otherwise, the error code is propagated upwards.



   If the query runs to completion without incident, SQLITE_OK is returned.
   Or, if some error occurs before the query completes or is aborted by
   the callback, an SQLite error code is returned.



int (*xSetAuxdata)(Fts5Context*, void *pAux, void(*xDelete)(void*))

Save the pointer passed as the second argument as the extension function's 
   "auxiliary data". The pointer may then be retrieved by the current or any
   future invocation of the same fts5 extension function made as part of
   the same MATCH query using the xGetAuxdata() API.



   Each extension function is allocated a single auxiliary data slot for
   each FTS query (MATCH expression). If the extension function is invoked 
   more than once for a single FTS query, then all invocations share a 
   single auxiliary data context.



   If there is already an auxiliary data pointer when this function is
   invoked, then it is replaced by the new pointer. If an xDelete callback
   was specified along with the original pointer, it is invoked at this
   point.



   The xDelete callback, if one is specified, is also invoked on the
   auxiliary data pointer after the FTS5 query has finished.



   If an error (e.g. an OOM condition) occurs within this function,
   the auxiliary data is set to NULL and an error code returned. If the
   xDelete parameter was not NULL, it is invoked on the auxiliary data
   pointer before returning.



void *(*xGetAuxdata)(Fts5Context*, int bClear)

Returns the current auxiliary data pointer for the fts5 extension 
   function. See the xSetAuxdata() method for details.



   If the bClear argument is non-zero, then the auxiliary data is cleared
   (set to NULL) before this function returns. In this case the xDelete,
   if any, is not invoked.



int (*xRowCount)(Fts5Context*, sqlite3_int64 *pnRow)

This function is used to retrieve the total number of rows in the table.
   In other words, the same value that would be returned by:


SELECT count(*) FROM ftstable;



int (*xPhraseFirst)(Fts5Context*, int iPhrase, Fts5PhraseIter*, int*, int*)

This function is used, along with type Fts5PhraseIter and the xPhraseNext
   method, to iterate through all instances of a single query phrase within
   the current row. This is the same information as is accessible via the
   xInstCount/xInst APIs. While the xInstCount/xInst APIs are more convenient
   to use, this API may be faster under some circumstances. To iterate 
   through instances of phrase iPhrase, use the following code:


Fts5PhraseIter iter;
int iCol, iOff;
for(pApi->xPhraseFirst(pFts, iPhrase, &iter, &iCol, &iOff);
    iCol>=0;
    pApi->xPhraseNext(pFts, &iter, &iCol, &iOff)
){
  // An instance of phrase iPhrase at offset iOff of column iCol
}



   The Fts5PhraseIter structure is defined above. Applications should not
   modify this structure directly - it should only be used as shown above
   with the xPhraseFirst() and xPhraseNext() API methods (and by
   xPhraseFirstColumn() and xPhraseNextColumn() as illustrated below).



   This API can be quite slow if used with an FTS5 table created with the
   "detail=none" or "detail=column" option. If the FTS5 table is created 
   with either "detail=none" or "detail=column" and "content=" option 
   (i.e. if it is a contentless table), then this API always iterates
   through an empty set (all calls to xPhraseFirst() set iCol to -1).



void (*xPhraseNext)(Fts5Context*, Fts5PhraseIter*, int *piCol, int *piOff)

See xPhraseFirst above.



int (*xPhraseFirstColumn)(Fts5Context*, int iPhrase, Fts5PhraseIter*, int*)

This function and xPhraseNextColumn() are similar to the xPhraseFirst()
   and xPhraseNext() APIs described above. The difference is that instead
   of iterating through all instances of a phrase in the current row, these
   APIs are used to iterate through the set of columns in the current row
   that contain one or more instances of a specified phrase. For example:


Fts5PhraseIter iter;
int iCol;
for(pApi->xPhraseFirstColumn(pFts, iPhrase, &iter, &iCol);
    iCol>=0;
    pApi->xPhraseNextColumn(pFts, &iter, &iCol)
){
  // Column iCol contains at least one instance of phrase iPhrase
}



   This API can be quite slow if used with an FTS5 table created with the
   "detail=none" option. If the FTS5 table is created with either 
   "detail=none" "content=" option (i.e. if it is a contentless table), 
   then this API always iterates through an empty set (all calls to 
   xPhraseFirstColumn() set iCol to -1).



   The information accessed using this API and its companion
   xPhraseFirstColumn() may also be obtained using xPhraseFirst/xPhraseNext
   (or xInst/xInstCount). The chief advantage of this API is that it is
   significantly more efficient than those alternatives when used with
   "detail=column" tables.



void (*xPhraseNextColumn)(Fts5Context*, Fts5PhraseIter*, int *piCol)

See xPhraseFirstColumn above.


c2: SQLite FTS5 Extension
c3: 8. The fts5vocab Virtual Table Module
c5: fts5.html#the_fts5vocab_virtual_table_module
c4:
The fts5vocab virtual table module allows users to extract information from
an FTS5 full-text index directly. The fts5vocab module is a part of FTS5 - it 
is available whenever FTS5 is.

 Each fts5vocab table is associated with a single FTS5 table. An fts5vocab
table is usually created by specifying two arguments in place of column names
in the CREATE VIRTUAL TABLE statement - the name of the associated FTS5 table
and the type of fts5vocab table. Currently there are three types of fts5vocab
table; "row", "col" and "instance". Unless the fts5vocab table is created
within the "temp" database, it must be part of the same database as the
associated FTS5 table.

-- Create an fts5vocab "row" table to query the full-text index belonging
-- to FTS5 table "ft1".
CREATE VIRTUAL TABLE ft1_v USING fts5vocab('ft1', 'row');

-- Create an fts5vocab "col" table to query the full-text index belonging
-- to FTS5 table "ft2".
CREATE VIRTUAL TABLE ft2_v USING fts5vocab(ft2, col);

-- Create an fts5vocab "instance" table to query the full-text index
-- belonging to FTS5 table "ft3".
CREATE VIRTUAL TABLE ft3_v USING fts5vocab(ft3, instance);


 If an fts5vocab table is created in the temp database, it may be associated
with an FTS5 table in any attached database. In order to attach the fts5vocab
table to an FTS5 table located in a database other than "temp", the name of the
database is inserted before the FTS5 table name in the CREATE VIRTUAL TABLE 
arguments. For example:

-- Create an fts5vocab "row" table to query the full-text index belonging
-- to FTS5 table "ft1" in database "main".
CREATE VIRTUAL TABLE temp.ft1_v USING fts5vocab(main, 'ft1', 'row');

-- Create an fts5vocab "col" table to query the full-text index belonging
-- to FTS5 table "ft2" in attached database "aux".
CREATE VIRTUAL TABLE temp.ft2_v USING fts5vocab('aux', ft2, col);

-- Create an fts5vocab "instance" table to query the full-text index 
-- belonging to FTS5 table "ft3" in attached database "other".
CREATE VIRTUAL TABLE temp.ft2_v USING fts5vocab('aux', ft3, 'instance');


 Specifying three arguments when creating an fts5vocab table in any database
other than "temp" results in an error.

 An fts5vocab table of type "row" contains one row for each distinct term
in the associated FTS5 table. The table columns are as follows:


  ColumnContents
  term The term, as stored in the FTS5 index.
  doc  The number of rows that contain at least one instance of the term.
  cnt  The total number of instances of the term in the entire FTS5 table.


 An fts5vocab table of type "col" contains one row for each distinct term/column
combination in the associated FTS5 table. Table columns are as follows:


  ColumnContents
  term The term, as stored in the FTS5 index.
  col  The name of the FTS5 table column that contains the term.
  doc  The number of rows in the FTS5 table for which column $col
                   contains at least one instance of the term.
  cnt  The total number of instances of the term that appear in
                   column $col of the FTS5 table (considering all rows). 


 An fts5vocab table of type "instance" contains one row for each term
instance stored in the associated FTS index. Assuming the FTS5 table is
created with the 'detail' option set to 'full', table columns are as follows:


  ColumnContents
  term   The term, as stored in the FTS5 index.
  doc    The rowid of the document that contains the term instance.
  col    The name of the column that contains the term instance.
  offset The index of the term instance within its column. Terms 
                     are numbered in order of occurrence starting from 0.


 If the FTS5 table is created with the 'detail' option set to 'col', then
the offset column of an instance virtual table always contains NULL.
In this case there is one row in the table for each unique term/doc/col 
combination. Or, if the FTS5 table is created with 'detail' set to 'none',
then both offset and col always contain NULL values. For
detail=none FTS5 tables, there is one row in the fts5vocab table for each
unique term/doc combination.

Example:

-- Assuming a database created using:
CREATE VIRTUAL TABLE ft1 USING fts5(c1, c2);
INSERT INTO ft1 VALUES('apple banana cherry', 'banana banana cherry');
INSERT INTO ft1 VALUES('cherry cherry cherry', 'date date date');

-- Then querying the following fts5vocab table (type "col") returns:
--
--    apple  | c1 | 1 | 1
--    banana | c1 | 1 | 1
--    banana | c2 | 1 | 2
--    cherry | c1 | 2 | 4
--    cherry | c2 | 1 | 1
--    date   | c3 | 1 | 3
--
CREATE VIRTUAL TABLE ft1_v_col USING fts5vocab(ft1, col);

-- Querying an fts5vocab table of type "row" returns:
--
--    apple  | 1 | 1
--    banana | 1 | 3
--    cherry | 2 | 5
--    date   | 1 | 3
--
CREATE VIRTUAL TABLE ft1_v_row USING fts5vocab(ft1, row);

-- And, for type "instance"
INSERT INTO ft1 VALUES('apple banana cherry', 'banana banana cherry');
INSERT INTO ft1 VALUES('cherry cherry cherry', 'date date date');
--
--    apple  | 1 | c1 | 0
--    banana | 1 | c1 | 1
--    banana | 1 | c2 | 0
--    banana | 1 | c2 | 1
--    cherry | 1 | c1 | 2
--    cherry | 1 | c2 | 2
--    cherry | 2 | c1 | 0
--    cherry | 2 | c1 | 1
--    cherry | 2 | c1 | 2
--    date   | 2 | c2 | 0
--    date   | 2 | c2 | 1
--    date   | 2 | c2 | 2
--
CREATE VIRTUAL TABLE ft1_v_instance USING fts5vocab(ft1, instance);


c2: SQLite FTS5 Extension
c3: Appendix A: Comparison with FTS3/4
c5: fts5.html#appendix_a
c4:
Also available is the similar but more mature FTS3/4 module. 
FTS5 is a new version of FTS4 that includes various fixes and solutions for 
problems that could not be fixed in FTS4 without sacrificing backwards 
compatibility. Some of these problems are 
described below.


c2: SQLite FTS5 Extension
c3: Application Porting Guide
c5: fts5.html#_application_porting_guide_
c4:
In order to use FTS5 instead of FTS3 or FTS4, applications usually require
minimal modifications. Most of these fall into three categories - changes
required to the CREATE VIRTUAL TABLE statement used to create the FTS table,
changes required to SELECT queries used to execute queries against the table,
and changes required to applications that use FTS auxiliary functions.


c2: SQLite FTS5 Extension
c3: Changes to CREATE VIRTUAL TABLE statements
c5: fts5.html#_changes_to_create_virtual_table_statements_
c4:
The module name must be changed from "fts3" or "fts4" to "fts5".

 All type information or constraint specifications must be removed from
     column definitions. FTS3/4 ignores everything following the column name in
     a column definition, FTS5 attempts to parse it (and will report an error
     if it fails to).

 The "matchinfo=fts3" option is not available. The 
     "columnsize=0" option is equivalent.

 The notindexed= option is not available. Adding UNINDEXED
     to the column definition is equivalent.

 The ICU tokenizer is not available.

 The compress=, uncompress= and languageid= options are not available.
     There is as of yet no equivalent for their functionality.


 -- FTS3/4 statement 
CREATE VIRTUAL TABLE t1 USING fts4(
  linkid INTEGER,
  header CHAR(20),
  text VARCHAR,
  notindexed=linkid,
  matchinfo=fts3,
  tokenizer=unicode61
);

 -- FTS5 equivalent (note - the "tokenizer=unicode61" option is not
 -- required as this is the default for FTS5 anyway)
CREATE VIRTUAL TABLE t1 USING fts5(
  linkid UNINDEXED,
  header,
  text,
  columnsize=0
);


c2: SQLite FTS5 Extension
c3: Changes to SELECT statements
c5: fts5.html#_changes_to_select_statements_
c4:
The "docid" alias does not exist. Applications must use "rowid"
          instead.

   The behaviour of queries when a column-filter is specified both as
          part of the FTS query and by using a column as the LHS of a MATCH
          operator is slightly different. For a table with columns "a" and "b"
          and a query similar to:
... a MATCH 'b: string'

       FTS3/4 searches for matches in column "b". However, FTS5 always
          returns zero rows, as results are first filtered for column "b", then
          for column "a", leaving no results. In other words, in FTS3/4 the
          inner filter overrides the outer, in FTS5 both filters are applied.

   The FTS query syntax (right hand side of the MATCH operator) has
          changed in some ways. The FTS5 syntax is quite close to the FTS4
          "enhanced syntax". The main difference is that FTS5 is fussier 
          about unrecognized punctuation characters and similar within query
          strings. Most queries that work with FTS3/4 should also work with
          FTS5, and those that do not should return parse errors.


c2: SQLite FTS5 Extension
c3: Auxiliary Function Changes
c5: fts5.html#_auxiliary_function_changes_
c4:
FTS5 has no matchinfo() or offsets() function, and the snippet() function
is not as fully-featured as in FTS3/4. However, since FTS5 does provide 
an API allowing applications to create custom auxiliary functions, any
required functionality may be implemented within the application code.

 The set of built-in auxiliary functions provided by FTS5 may be 
improved upon in the future.


c2: SQLite FTS5 Extension
c3: Other Issues
c5: fts5.html#_other_issues
c4:
The functionality provided by the fts4aux module is now provided
          by fts5vocab. The schema of these two tables is slightly different.

   The FTS3/4 "merge=X,Y" command has been replaced by the 
          FTS5 merge command.

   The FTS3/4 "automerge=X" command has been replaced by the 
          FTS5 automerge option.


c2: SQLite FTS5 Extension
c3: Summary of Technical Differences
c5: fts5.html#_summary_of_technical_differences_
c4:
FTS5 is similar to FTS3/4 in that the primary task of each is to maintain
an index mapping from each unique token to a list of instances of that token 
within a set of documents, where each instance is identified by the document 
in which it appears and its position within that document. For example:

-- Given the following SQL:
CREATE VIRTUAL TABLE ft USING fts5(a, b);
INSERT INTO ft(rowid, a, b) VALUES(1, 'X Y', 'Y Z');
INSERT INTO ft(rowid, a, b) VALUES(2, 'A Z', 'Y Y');

-- The FTS5 module creates the following mapping on disk:
A --&gt; (2, 0, 0)
X --&gt; (1, 0, 0)
Y --&gt; (1, 0, 1) (1, 1, 0) (2, 1, 0) (2, 1, 1)
Z --&gt; (1, 1, 1) (2, 0, 1)


In the example above, each triple identifies the location of a token
instance by rowid, column number (columns are numbered sequentially
starting at 0 from left to right) and position within the column value (the
first token in a column value is 0, the second is 1, and so on). Using this
index, FTS5 is able to provide timely answers to queries such as "the set
of all documents that contain the token 'A'", or "the set of all documents
that contain the sequence 'Y Z'". The list of instances associated with a
single token is called an "instance-list".

The principle difference between FTS3/4 and FTS5 is that in FTS3/4,
each instance-list is stored as a single large database record, whereas
in FTS5 large instance-lists are divided between multiple database records.
This has the following implications for dealing with large databases that
contain large lists:


   FTS5 is able to load instance-lists into memory incrementally in
       order to reduce memory usage and peak allocation size. FTS3/4 very
       often loads entire instance-lists into memory.

   When processing queries that feature more than one token, FTS5 is
       sometimes able to determine that the query can be answered by
       inspecting a subset of a large instance-list. FTS3/4 almost always
       has to traverse entire instance-lists.

   If an instance-list grows so large that it exceeds
       the SQLITE_MAX_LENGTH limit, FTS3/4 is unable to handle it. FTS5
       does not have this problem. 


For these reasons, many complex queries may use less memory and run faster 
using FTS5.

Some other ways in which FTS5 differs from FTS3/4 are:


   FTS5 supports "ORDER BY rank" for returning results in order of
       decreasing relevancy.

   FTS5 features an API allowing users to create custom auxiliary 
       functions for advanced ranking and text processing applications. The
       special "rank" column may be mapped to a custom auxiliary function
       so that adding "ORDER BY rank" to a query works as expected.

   FTS5 recognizes unicode separator characters and case equivalence by
       default. This is also possible using FTS3/4, but must be explicitly
       enabled.

   The query syntax has been revised where necessary to remove
       ambiguities and to make it possible to escape special characters
       in query terms.

   By default, FTS3/4 occasionally merges together two or more of the
       b-trees that make up its full-text index within an INSERT, UPDATE or
       DELETE statement executed by the user. This means that any operation
       on an FTS3/4 table may turn out to be surprisingly slow, as FTS3/4 
       may unpredictably choose to merge together two or more large b-trees
       within it. FTS5 uses incremental merging by default, which limits
       the amount of processing that may take place within any given 
       INSERT, UPDATE or DELETE operation.


c2: SQLite FTS5 Extension
c3: Appendix B: Shadow tables created by FTS5
c5: fts5.html#appendix_b
c4:
When an FTS5 virtual table is created in a database, between 3 and 5 real
tables are created in the database. These are known as "shadow tables", and are
used by the virtual table module to store persistent data. They should not
be accessed directly by the user. Many other virtual table modules, including
FTS3 and rtree, also create and use shadow tables.

FTS5 creates the following shadow tables. In each case the actual table name
is based on the name of the FTS5 virtual table (in the following table, replace
&lt;name&gt; with the name of the virtual table to find the actual shadow
table name).


Table&nbsp;NameContents
&lt;name&gt;_data This table contains most of the full-text index data.
&lt;name&gt;_idx This table contains the remainder of the full-text
index data. It is almost always much smaller than the &lt;name&gt;_data table.
&lt;name&gt;_config Contains the values of persistent
configuration parameters.
&lt;name&gt;_content Contains the actual data inserted into the 
FTS5 table. This shadow table is not present for 
contentless or 
external content FTS5 tables.
&lt;name&gt;_docsize Contains the size of each column of each
row in the virtual table in tokens. This shadow table is not present if 
the "columnsize" option is set to 0.


c2: Code Of Conduct
c3: 1. Overview
c5: codeofconduct.html#overview
c4:
The original document we put here was more of a 
Code of Ethics of the Project Founder. 
While we stand by those principles, they are not in line with the modern
technical meaning of a Code of Conduct and have hence been renamed.


c2: Code Of Conduct
c3: 2. Code of Conduct
c5: codeofconduct.html#code_of_conduct
c4:
The SQLite community has been a kind, welcoming and well-mannered community
since its inception nearly two decades ago. In that time many of the
qualities we've worked hard to bring to our project - safety, consistency,
usability and integrity among them - have become part of our community's
culture as well. Much like the care and effort we've put into securing the
longevity of SQLite, we'd like to do the same for that community by 
clarifying our goals and expectations for all involved. To that end
the SQLite project has officially adopted the 
Mozilla Community Participation Guidelines
as its Code of Conduct with the exception that the reporting
email address is changed to
codeofconduct@sqlite.org.
While we are not doing so in reaction to any current or ongoing issues,
we believe that this will be a helpful part of maintaining the long-term
sustainability of the project.


c2: How SQLite Works
c3: 1. Background
c5: howitworks.html#background
c4:
SQLite is a software library
that translates high-level disk I/O requests generated by an application
into low-level I/O operations that
can be carried out by the operating system.  The application constructs 
high-level I/O requests using the
SQL language.
SQLite translates each high-level SQL statement into a sequence of
many low-level I/O requests (open a file, read a few bytes from a file,
write a few bytes into a file, etc.) that do the work requested by
the SQL.



An application program could do all its disk I/O by direct calls to
operating system I/O routines
or by using a key/value storage engine like 
Berkeley DB or
RocksDB (to name but two).
But there are advantages to using a higher-level interface based
on the SQL language.



SQL is a very high-level language.
A few lines of SQL can replace hundreds or thousands of lines of
procedural code.  SQL thus reduces
the amount of work needed to develop and maintain the application,
and thereby helps to reduce the number of bugs in the application.


SQL and SQLite are 
transactional.
The use of a transactional storage system makes it much easier to
reason about the behavior of the application, and to write applications
that are robust, even in the face of software bugs, hardware faults,
or power losses.


SQLite is often
faster than direct low-level I/O.
This is counterintuitive.  One would expect that a
high-level interface such as SQLite would impose a run-time penalty.
And, theoretically, that is correct.  But in practice, SQL-based
systems such as SQLite do so many behind-the-scenes optimizations
that an application developer would never have time to create and
maintain, that the SQL-based systems end up providing
a net performance gain.


c2: How SQLite Works
c3: 1.1. SQLite Is Different From Most Other SQL Databases
c5: howitworks.html#sqlite_is_different_from_most_other_sql_databases
c4:
There are many SQL-based database management systems available, besides
SQLite.  Common options include MySQL, PostgreSQL, and SQL-Server.
All these systems use the SQL langauge to communicate with the 
application, just like SQLite.  But these other systems different
from SQLite in important respects.



SQLite is a serverless software library, whereas the other
systems are client-server based.  With MySQL, PostgreSQL, SQL-Server,
and others, the application sends a message containing some SQL
over to a separate server thread or process. That separate
thread or process performs the requested I/O, then send the results
back to the application.  But there is no separate thread or
process with SQLite.  SQLite runs in the same address space as
the application, using the same program counter and heap storage.
SQLite does no interprocess communication (IPC).  When an application
sends an SQL statement into SQLite (by invoking a the appropriate
SQLite library subroutine), SQLite interprets the SQL
in the same thread as the caller.  When an SQLite API routine
returns, it does not leave behind any background tasks that
run separately from the application.


An SQLite database is a single ordinary file on disk
(with a well-defined file format).  With other
systems, a "database" is usually a large number of separate
files hidden away in obscure directories of the filesystem, or
even spread across multiple machines.  But with SQLite, a
complete database is just an ordinary disk file.


c2: How SQLite Works
c3: 2. SQL Is A Programming Language
c5: howitworks.html#sql_is_a_programming_language
c4:
The best way to understand how SQL database engines work is to
think of SQL as a programming language, not as a "query language".
Each SQL statement is a separate program.  Applications construct
SQL program source files and send them to the database engine.
The database engine compiles the SQL source code into executable
form, runs that executable, then sends the result back to the
application.


While SQL is a programming language, it is different from other
programming languages like C, Javascript, Python, or Go in that
SQL is a 
declarative language
where the others are
imperative languages.
This is an important difference that has implications for the
design of the compiler used to translate program source text into an
executable format. However, those details 
should not detract from the fact that
SQL is really just another programming language.


c2: How SQLite Works
c3: 2.1. Programming Language Processing Steps
c5: howitworks.html#programming_language_processing_steps
c4:
All programming languages are processed in two steps:



Translate the program source text into an executable format.

Run the executable generated in the previous step in order to
carry out the desired action.



All programming languages uses those two basic steps.
The main difference is in the executable format.


"Compiled" languages like C++ and Rust translate
the source text into machine code that can be directly executed
by the underlying hardware.  There exist SQL database systems
that do the same with SQL - they translate each SQL statement
directly into machine code.  But that approach is uncommon and
is not the approach taken by SQLite.


Other languages like Java, Perl, Python, and TCL
typically translate the program source text into bytecode.
This bytecode is then run through an interpreter that
reads the bytecode and carries out the desired operations.
SQLite uses this bytecode approach.  If you preceed
any SQL statement with the "EXPLAIN" keyword in SQLite,
it will show you the bytecode that is generated rather
than run the bytecode.


Another approach is to translate the program source text
into a tree of objects in memory.  This tree is the "executable".
An interpret runs the executable by walking the tree.
This is the technique used by MySQL, PostgreSQL, and
SQL-Server.


Of course, not every language fits neatly into one of the
above catagories.  This applies to both SQL database engines
and more familiar imperative programming languages.  Javascript
is famous for using a hybrid execution model, where the code
is initially compiled into a tree of objects, but might be
further translating (using just-in-time compilation) down into
more efficient bytecode or machine code, as a means of boosting
performance.


The executable format really ends up being just an implementation
detail.  The key point is that all languages have a compiler
step which translates programs into an executable format and
a run step that executes the compiled program.


c2: How SQLite Works
c3: 2.2. Compiling SQLite Programs
c5: howitworks.html#compiling_sqlite_programs
c4:
When an SQL program is submitted to SQLite, the first step is
to split the source text into "tokens".  A token might be:


A language keyword like "SELECT" or "UPDATE".
An identifier for a table or column or variable.
Punctuation characters like "," or "==" or ";".
Literal values: numeric or string constants.
Whitespace or comments.



Whitespace and comment tokens are discarded.  All other tokens
are fed into an
LALR(1) Parser
that analysis the structure of the input program and generates an
Abstract Syntax Tree (AST)
for the input program.


The parser forwards the AST on to the code generator.
The code generator is the heart of SQLite, and is where most of
the magic happens.
The code generator resolves symbolic names in the AST - matching
the names of columns and tables in the input SQL into actual
columns and tables of the database.  The code generator also does
various transformations on the AST to "optimize" it.  Finally
the code generator chooses appropriate algorithms to implement
the operations requested by the AST and constructs bytecode to
carry out those operations.


The bytecode generated by the code generator is called a
"prepared statement".  Translating SQL source text into a
prepared statement is analogous to converting a C++ program
into machine code by invoking gcc or clang.  Human-readable
source text (SQL or C++) goes in, and a machine readable executable
(bytecode or machine code) comes out.


c2: How SQLite Works
c3: 3. Further Reading
c5: howitworks.html#further_reading
c4:
The Atomic Commit document describes how SQLite implements
transactions.


The bytecode engine document has more information
on the bytecode format used by SQLite, and how to view and interpret
an SQLite prepared statement.


The SQLite query planner and
Next Generation Query Planner documents have further details
on the algorithms SQLite uses to implement SQL statements and
how it goes above choosing an appropriate algorithm for each
individual SQL statement.


c2: WAL-mode File Format
c5: walformat.html#
c4:
This document describes low-level details on how WAL mode is
implemented on unix and windows.

The separate file format description provides details on the
structure of a database file and of the write-head log file used in
WAL mode.  But details of the locking protocol and of the format
of the WAL-index are deliberately omitted since those details
are left to discretion of individual VFS implementations.  This
document fills in those missing details for the unix and windows VFSes.

For completeness, some of the higher level formatting information
contains in the file format document and elsewhere is replicated here,
when it pertains to WAL mode processing.


c2: WAL-mode File Format
c3: 1. Files On Disk
c5: walformat.html#files_on_disk
c4:
When in active use, the state of a WAL mode database is described
by three separate files:


 The main database file with an arbitrary name "X".
 The write-ahead log file, usually named "X-wal".
 The wal-index file, usually named "X-shm".


c2: WAL-mode File Format
c3: 1.1. The Main Database File
c5: walformat.html#the_main_database_file
c4:
The format of the main database file is as described in the
file format document.  The file format version numbers at offsets
18 and 19 into the main database must both be 2 to indicate that the
database is in WAL mode.  The main database may have an arbitrary
name allowed by the underlying filesystem.  No special file suffixes
are required, though ".db", ".sqlite", and ".sqlite3" seem to be
popular choices.


c2: WAL-mode File Format
c3: 1.2. The Write-Ahead-Log or "-wal" File
c5: walformat.html#the_write_ahead_log_or_wal_file
c4:
The write-ahead log or "wal" file is a roll-forward journal
that records transactions that have been committed but not yet applied
to the main database.  Details on the format of the wal file are
describe in the WAL format subsection of the main file format
document.  The wal file is named by appending the four characters
"-wal" to the end of the name of the main database file.  Except
on 8+3 filesystems, such names are not allowed, and in that case
the file suffix is changed to ".WAL".  But as 8+3 filesystems are
increasingly rare, that exceptional case can usually be ignored.


c2: WAL-mode File Format
c3: 1.3. The Wal-Index or "-shm" file
c5: walformat.html#the_wal_index_or_shm_file
c4:
The wal-index file or "shm" file is not actually used as a file.
Rather, individual database clients mmap the shm file and use it
as shared memory for coordinating access to the database and as a cache
for quickly locating frame within the wal file.  The name
of the shm file is the main database file name with the four characters
"-shm" appended.  Or, for 8+3 filesystems, the shm file is the main
database file with the suffix changed to ".SHM".

The shm does not contain any database content and is not required
to recover the database following a crash.  For that reason, the first
client to connect to a quiescent database will normally truncate the
shm file if it exists.  Since the content of the shm file does not need
to be preserved across a crash, the shm file is never fsync()-ed to disk.
In fact, if there were a mechanism by which SQLite could tell the 
operating system to never persist the shm file to disk but always hold
it in cache memory, SQLite would use that mechanism to avoid any
unnecessary disk I/O associated with the shm file.  However, no such
mechanism exists in standard posix.

Because the shm is only used to coordinate access between concurrent
clients, the shm file is omitted if exclusive locking mode
is set, as an optimization.  When exclusive locking mode is set,
SQLite uses heap memory in place of the memory-mapped shm file.


c2: WAL-mode File Format
c3: 1.4. File Lifecycles
c5: walformat.html#file_lifecycles
c4:
When a WAL mode database is in active use, all three of the above
files usually exist.  Except, the Wal-Index file is omitted if
exclusive locking mode is set.

If the last client using the database shuts down cleanly by
calling sqlite3_close(), then a checkpoint is run automatically
in order to transfer all information from the wal file
over into the main database, and both the shm file
and the wal file are unlinked.  Thus, when the database is not in
use by any client, it is usually the case that only the main
database file exists on disk.
However, if the last client did not call sqlite3_close() before it
shut down, or if the last client to disconnect was a read-only client,
then the final cleanup operation does not occur and the
shm and wal files may still exist on disk even when the database is
not in use.


c2: WAL-mode File Format
c3: 1.5. Variations
c5: walformat.html#variations
c4:
When PRAGMA locking_mode=EXCLUSIVE (exclusive locking mode) is set,
only a single client is allowed to have the database open at one time.  Since
only a single client can use the database, the shm file is omitted.
The single client uses a buffer in heap memory as a substitute for the
memory-mapped shm file.

If a read/write client invokes
sqlite3_file_control(SQLITE_FCNTL_PERSIST_WAL) prior to shutdown,
then at shutdown a checkpoint is still run, but the shm file and wal
file are not deleted.
This allows subsequent read-only clients to connect to and read the
database.


c2: WAL-mode File Format
c3: 2. The WAL-Index File Format
c5: walformat.html#the_wal_index_file_format
c4:
The WAL-index or "shm" file is used to coordinate access to the database
by multiple clients, and as a cache to help clients quickly locate frames
within the wal file.


Because the shm file is not involved in recovery, the shm file does not
need to be machine byte-order independent.  Hence, numeric values in
the shm file are written in the native byte order of the host computer,
rather than being converted into a specific cross-platform byte order as
is done with the main database file and the wal file.


The shm file consists of one or more hash tables, where each hash table
is 32768 bytes in size.  Except, a 136-byte header is carved out of the
front of the very first hash table, so the first hash table is only
32632 bytes in size.  The total size of the shm file is always a multiple
of 32768.  In most cases, the total size of the shm file is exactly 32768
bytes.  The shm file only needs to grow beyond a single hash table if
when the wal file grows very large (more than 4079 frames).  Since the
default automatic checkpoint threshold is
1000, WAL files rare reach the 4079 threshold needed to make the shm file
grow.


c2: WAL-mode File Format
c3: 2.1. The WAL-Index Header
c5: walformat.html#the_wal_index_header
c4:
The first 136 bytes of the shm file are a header.  The shm header has
three main divisions as follows:


WAL-Index Header Divisions

BytesDescription
0..47First copy of the WAL Index Information
48..95Second copy of the WAL Index Information
96..135Checkpoint Information and Locks



Individual fields of the shm header, except for the
salt values copied from the WAL header, are unsigned integers
in the native byte-order of the host machine.  The salt values
are exact copies from the WAL header and are in whatever byte
order is used by the WAL file.
The size of integers may be 8, 16, 32, or 64 bits.
A detailed breakout of the individual fields of the shm header
follows:


WAL-Index Header Details

BytesNameMeaning

0..3iVersion
The WAL-index format version number.  Always 3007000.


4..7&nbsp;
Unused padding space.  Must be zero.


8..11iChange
Unsigned integer counter, incremented with each transaction


12isInit
The "isInit" flag.  1 when the shm file has been initialized.


13bigEndCksum
True if the WAL file uses big-ending checksums.  0 if the WAL
    uses little-endian checksums.


14..15szPage
The database page size in bytes, or 1 if the page size is 65536.


16..19mxFrame
Number of valid and committed frames in the WAL file.


20..23nPage
Size of the database file in pages.


24..31aFrameCksum
Checksum of the last frame in the WAL file.


32..39aSalt
The two salt value copied from the WAL file header.
These values are in the byte-order of the WAL file, which
might be different from the native byte-order of the
machine.


40..47aCksum
A checksum over bytes 0 through 39 of this header.


48..95&nbsp;
A copy of bytes 0 through 47 of this header.


96..99nBackfill
Number of WAL frames that have already been backfilled into the database
    by prior checkpoints


100..119read-mark&#91;0..4&#93;
Five "read marks".  Each read mark is a 32-bit unsigned integer (4 bytes).


120..127&nbsp;
Unused space set aside for 8 file locks.


128..132nBackfillAttempted
Number of WAL frames that have attempted to be backfilled but which might
    not have been backfilled successfully.


132..136&nbsp;
Unused space reserved for further expansion.


c2: WAL-mode File Format
c3: 2.1.1. The mxFrame field
c5: walformat.html#the_mxframe_field
c4:
The 32-bit unsigned integer at offset 16 (and repeated at offset 64)
is the number of valid frames in the WAL.  Because WAL frame are numbered
starting with 1, mxFrame is also the index of the last valid commit frame
in the WAL. A commit frame is a frame that has a non-zero "size of database"
value in bytes 4 through 7 of the frame header, and that indicates the end
of a transaction.

When mxFrame field is zero, it indicates that the WAL is empty and that
all content should be obtained directly from the database file.

When mxFrame is equal to nBackfill, that indicates that all content
in the WAL has been written back into the database.
In that case, all content can be read directly from the database.
Furthermore, the next writer is free to reset the WAL if no other
connections hold locks on WAL_READ_LOCK(N) for N>0.

The mxFrame value is always greater than or equal to both
nBackfill and nBackfillAttempted.


c2: WAL-mode File Format
c3: 2.1.2. The nBackfill field
c5: walformat.html#the_nbackfill_field
c4:
The 32-bit unsigned integer at offset 128 in the WAL-index header
is called the "nBackfill".
this field holds the number of frames in the WAL file which
have been copied back into the main database.

The nBackfill number is never greater than mxFrame.
When nBackfill equals mxFrame, that means that the WAL content
has been completely written back into the database and it is
ok to reset the WAL if there are no locks held on any of WAL_READ_LOCK(N)
for N>0.

The nBackfill can only be increased while holding the
WAL_CKPT_LOCK.  However, nBackfill is changed to zero
during a WAL reset, and this happens while holding
the WAL_WRITE_LOCK.


c2: WAL-mode File Format
c3: 2.1.3. WAL Locks
c5: walformat.html#wal_locks
c4:
Eight bytes of space are set aside in the header to support
file locking using the xShmLock() method in the sqlite3_io_methods
object.  These eight bytes are never read nor written by SQLite since
some VFSes (ex: Windows) might implement locks using mandatory file locks.

These are the eight locks supported:


WAL-Index Locks Controlled By xShmLock()

NameOffset
xShmLockFile

WAL_WRITE_LOCK
0
120



WAL_CKPT_LOCK
1
121



WAL_RECOVER_LOCK
2
122



WAL_READ_LOCK(0)
3
123



WAL_READ_LOCK(1)
4
124



WAL_READ_LOCK(2)
5
125



WAL_READ_LOCK(3)
6
126



WAL_READ_LOCK(4)
7
127






TBD:  More information about the header


c2: WAL-mode File Format
c3: 2.2. WAL-Index Hash Tables
c5: walformat.html#wal_index_hash_tables
c4:
The hash tables in the shm file are designed to answer the
following question quickly:


FindFrame(P,M):
Given a page number P and a maximum WAL frame index M,
return the largest WAL frame index for page P that does not exceed M, 
or return NULL if there are no frames for page P that do not exceed M.



Let the datatypes "u8", "u16", and "u32" mean unsigned integers of
length 8, 16, and 32 bits, respectively.  Then, the first 32768-byte unit
of the shm file is organized as follows:


u8 aWalIndexHeader[136];
u32 aPgno[4062];
u16 aHash[8192];


The second and all subsequent 32768-byte units of the shm file are
like this:


u32 aPgno[4096];
u16 aHash[8192];


Collectively, the aPgno entries record the database page number stored
in all frames of the WAL file.  The aPgno[0] entry on the first hash table
records the database page number stored in the very first frame in the WAL 
file. The aPgno[i] entry from the first hash table is the database page number
for the i-th frame in the WAL file.  The aPgno[k] entry for the second
hash table is the database page number for the (k+4062)-th frame in the
WAL file.  The aPgno[k] entry for the n-th 32768-byte hash table in the
shm file (for n>1) holds the database page number stored in the
(k+4062+4096*(n-2))-th frame of the WAL file.

Here is a slightly different way to describe the aPgno values:
If you think of all aPgno values as a contiguous array, then
the database page number stored in the i-th frame of the WAL file
is stored in aPgno[i].  Of course, aPgno is not a contiguous array.
The first 4062 entries are on the first 32768-byte unit of the shm
file and subsequent values are in 4096 entry chunks in later units
of the shm file.

One way to compute FindFrame(P,M) would be to scan the aPgno
array starting with the M-th entry and working backwards towards
the beginning and return J where aPgno[J]==P.  Such an algorithm would
work, and it would be faster than searching the whole WAL file for
the latest frame with page number P.  But the search can be made
much faster still by using the aHash structure.

A database page number P is mapped into a hash value
using the following hash function:


h = (P * 383)%8192


This function maps every page number into an integer between 0
and 8191 inclusive.  The aHash field of each 32768-byte shm file unit
maps P values into indexes of the aPgno field of the same unit as
follows:


 Compute the hash value: h = P * 383
 Let X be the largest set of consecutive integers {h, h+1, h+2, ..., h+N}
     such that for every j in X, aPgno&#91;j%8192]!=0.  The X set will be empty
     if aPgno&#91;h%8192]==0.  The X set is easily computed by starting with
     the value h%8192, and adding h%8192 to X and incrementing h until
     encountering the first aPgno&#91;h%8192] entry that is zero.
 The set X contains the index in aPgno of every entry in the current
     32768-byte unit of the shm file that might possible be a solution
     to the FindFrame(P,M) function.  Each of these entries must be
     checked separately to ensure that the aPgno value is P and that the
     frame number does not exceed M.  The largest frame number that passes
     those two tests is the answer.


Each entry in the aPgno array has a single corresponding entry
in the aHash array.  There are more available slots in aHash than
there are in aPgno.  The unused slots in aHash are filled with zero.
And since there are guaranteed to be unused slots in aHash, that means
the loop that computes X is guaranteed to terminate.  The expected size
of X is less than 2.  The worst case is that X will be the same as the
number of entries in aPgno, in which case the algorithm runs at about
the same speed as a linear scan of aPgno.  But that worst case performance
is exceedingly rare.  Usually, the size of X will be small and the use
of the aHash array allows one to compute FindFrame(P,M) much faster.

Here is an alternative way of describing the hash look-up algorithm:
Start with h = (P * 383)%8192 and look at aHash[h] and subsequent entries,
wrapping around to zero when h reaches 8192, until finding an entry with
aHash[h]==0.  All aPgno entries having a page number of P will have an
index that is one of the aHash[h] values thusly computed.
But not all the computed aHash[h] values will
meet the matching criteria, so you must check them independently.  The
speed advantage comes about because normally this set of h values is
very small.

Note that each 32768-byte unit of the shm file has its own aHash and
aPgno arrays.  The aHash array for a single unit is only helpful in finding
aPgno entries in that same unit.  The overall FindFrame(P,M) function
needs to do hash lookups beginning with the latest unit and working
backwards to the oldest unit until it finds an answer.


c2: WAL-mode File Format
c3: 2.3. Locking Matrix
c5: walformat.html#locking_matrix
c4:
Access is coordinated in WAL mode using both the legacy DELETE-mode
locks controlled by the xLock and xUnlock methods of the sqlite3_io_methods
object and the WAL locks controlled by the xShmLock method of the
sqlite3_io_methods object.

Conceptually, there is just a single DELETE-mode lock.  The DELETE-mode
lock for a single database connection can be in exactly one of the
following states:

 SQLITE_LOCK_NONE (unlocked)
 SQLITE_LOCK_SHARED (reading)
 SQLITE_LOCK_RESERVED (reading, waiting to write)
 SQLITE_LOCK_PENDING (new readers blocked, waiting to write)
 SQLITE_LOCK_EXCLUSIVE (writing)

The DELETE-mode locks are stored on the lock-byte page of the
main database file.
Only SQLITE_LOCK_SHARED and SQLITE_LOCK_EXCLUSIVE are factors for WAL-mode
databases.
The other locking states are used in rollback-mode, but not in WAL-mode.

The WAL-mode locks are described above.


c2: WAL-mode File Format
c3: 2.3.1. How the various locks are used
c5: walformat.html#how_the_various_locks_are_used
c4:
The following rules show how each of the locks is used.


SQLITE_LOCK_SHARED

All connections hold SQLITE_LOCK_SHARED continuously while attached
to a WAL-mode database.  This is true for both read/write connections
and read-only connections.
The SQLITE_LOCK_SHARED lock is held even by connections that are
not within transaction.
This is different from rollback mode, where the SQLITE_LOCK_SHARED is
released at the end of each transaction.

SQLITE_LOCK_EXCLUSIVE

Connections hold an exclusive lock when change in between WAL mode
and any of the various rollback-modes.  Connections might also attempt to
obtain an EXCLUSIVE lock when they disconnect from WAL mode.  If
a connection is able to obtain an EXCLUSIVE lock, that means it is the
only connection to the database and so it may attempt to checkpoint
and then delete the WAL-index and WAL files.

When a connection is holding a SHARED lock on the main database,
that will prevent any other connection from acquiring the EXCLUSIVE
lock, which in turn prevents the WAL-index and WAL files from being
deleted out from under other users, and prevents a transition out of
WAL-mode while other users are accessing the database in WAL-mode.


WAL_WRITE_LOCK

The WAL_WRITE_LOCK is only locked exclusively.  There is never a shared
lock taken on WAL_WRITE_LOCK.

An EXCLUSIVE WAL_WRITE_LOCK is held by any connection that is appending
content to the end of the WAL.  Hence, only a single process at a time
can append content to the WAL.  If a WAL reset occurs as a consequence of
a write, then the nBackfill field of the WAL-index header is reset to
zero while holding this lock.
An EXCLUSIVE is also held WAL_WRITE_LOCK, and on several other locking
bytes, when a connection is running recovery on the shared WAL-index.

WAL_CKPT_LOCK

The WAL_CKPT_LOCK is only locked exclusively.  There is never a shared
lock taken on WAL_CKPT_LOCK.

An EXCLUSIVE WAL_CKPT_LOCK is held by any connection that is running
a checkpoint.  The nBackfill field of the WAL-index header may be
increased while holding this exclusive lock, but it may not be decreased.
An EXCLUSIVE is also held WAL_CKPT_LOCK, and on several other locking
bytes, when a connection is running recovery on the shared WAL-index.

WAL_RECOVER_LOCK

The WAL_RECOVER_LOCK is only locked exclusively.  There is never a shared
lock taken on WAL_RECOVER_LOCK.

An EXCLUSIVE WAL_RECOVER_LOCK is held by any connection that is running
recovery to reconstruct the shared WAL-index.

A read-only connection that is rebuilding its private heap-memory WAL-index
does not hold this lock.  (It cannot, since read-only connections are not
allowed to hold any exclusive locks.)  This lock is only held when rebuilding
the global shared WAL-index contained in the memory-mapped SHM file.

In addition to locking this byte, a connection running recovery also
gets an exclusive lock on all other WAL locks except for WAL_READ_LOCK(0).

WAL_READ_LOCK(N)

There are five separate read locks, numbers 0 through 4.
Read locks may be either SHARED or EXCLUSIVE.
Connections obtain a shared lock on one of the read locks bytes while
they are within a transaction.
Connections also obtain an exclusive lock on read locks, one at a time,
for the brief moment while they are updating the values of the corresponding
read-marks.
Read locks 1 through 4 are held exclusively when running recovery.

Each read lock byte corresponds to one of the five 32-bit read-mark
integers located in bytes 100 through 119 of the WAL-index header, as
follows:


Lock NameLock offsetRead-mark nameRead-mark offset
WAL_READ_LOCK(0)123read-mark[0]100..103
WAL_READ_LOCK(1)124read-mark[1]104..107
WAL_READ_LOCK(2)125read-mark[2]108..111
WAL_READ_LOCK(3)126read-mark[3]112..115
WAL_READ_LOCK(4)127read-mark[4]116..119



When a connection holds a shared lock on WAL_READ_LOCK(N), that is a
promise by the connection that it will use the WAL and not the database
file for any database pages that are modified by the first
read-mark[N] entries in the WAL.
The read-mark[0] is always zero.  If a connection holds a shared lock
on WAL_READ_LOCK(0), that means the connection expects to be able to ignore
the WAL and read any content it wants from the main database.
If N>0 then the connection is free to use more of the WAL file beyond
read-mark[N] if it wants to, up to the first mxFrame frames.
But when a connection holds a shared lock on WAL_READ_LOCK(0), that is a
promise that it will never read content from the WAL
and will acquire all content directly from the main database.

When a checkpoint runs, if it sees a lock on WAL_READ_LOCK(N), then it
must not move WAL content into the main database for more than the first
read-mark[N] frames.  Were it to do so, it would overwrite content that
the process holding the lock was expecting to be able to read out of the
main database file.  A consequence of if this is that f the WAL file
contains more than read-mark[N] frames (if mxFrame>read-mark[N]
for any read-mark for which WAL_READ_LOCK(N) is held by another process),
then the checkpoint cannot run to completion.

When a writer wants to reset the WAL, it must ensure that there are
no locks on WAL_READ_LOCK(N) for N>0 because such locks indicate
that some other connection is still using the current WAL file and 
a WAL reset would delete content out from those other connections.  It is
ok for a WAL reset to occur if other connections are holding WAL_READ_LOCK(0)
because by holding WAL_READ_LOCK(0), those other connections are promising
not to use any content from the WAL.


c2: WAL-mode File Format
c3: 2.3.2. Operations that require locks and which locks those operations use
c5: walformat.html#operations_that_require_locks_and_which_locks_those_operations_use
c4:
Transition into and out of WAL-mode

The SQLITE_LOCK_EXCLUSIVE lock must be held by a connection that wants
to transition into our out of WAL mode.
Transitioning into WAL mode is, therefore, just like any other write
transaction, since every write transaction in rollback mode requires
the SQLITE_LOCK_EXCLUSIVE lock.
If the database file is already in WAL mode (hence if the desire it to change
it back into rollback mode) and if there are two
or more connections to the database, then each of these connections will
be holding an SQLITE_LOCK_SHARED lock.  That means that the
SQLITE_LOCK_EXCLUSIVE cannot be obtained, and the transition out of
WAL mode will not be allowed.  This prevents one connection from deleting
WAL mode out from under another.  It also means that the only way to move
a database from WAL mode into rollback mode is to close all but one
connection to the database.


Close a connection to a WAL mode database

When a database connection closes (via sqlite3_close() or
sqlite3_close_v2()), an attempt is made to acquire
SQLITE_LOCK_EXCLUSIVE.  If this attempt is successful, that means
the connection that is closing is the last connection to the database.
In that case, it is desirable to clean up the WAL and WAL-index files,
so the closing connection runs a checkpoint (while holding
SQLITE_LOCK_EXCLUSIVE) and the deletes both the WAL and WAL-index files.
The SQLITE_LOCK_EXCLUSIVE is not released until after both the
WAL and WAL-index files have been deleted.

If the application invokes
sqlite3_file_control(SQLITE_FCNTL_PERSIST_WAL) on the database
connection prior to closing, then the final checkpoint is still
run but the WAL and WAL-index files are
not deleted as they normally would be.
This leaves the database in a state that allows other processes
without write permission on the database, WAL, or WAL-index files
to open the database read-only.
If the WAL and WAL-index files are missing, then a process that
lacks permission to create and initialize those files will not be
able to open the database, unless the database is designated
as immutable using the immutable query parameter.

Reconstruct the global shared WAL-index during recovery

All of the WAL-index locks, except for WAL_READ_LOCK(0),
are held exclusively while reconstructing the global shared WAL-index
during recovery.  

Append a new transaction to the end of the WAL

An exclusive lock is held on WAL_WRITE_LOCK while adding new
frame onto the end of a WAL file.  

Read content from the database and WAL as part of a transaction

Run a checkpoint

Reset the WAL file

A WAL reset means to rewind the WAL and start adding new frames
at the beginning.  This occurs while appending new frames to a WAL
that has mxFrame equal to nBackfill and which has no locks on
WAL_READ_LOCK(1) through WAL_READ_LOCK(4).  The WAL_WRITE_LOCK is
held.


c2: WAL-mode File Format
c3: 3. Recovery
c5: walformat.html#recovery
c4:
Recovery is the process of rebuilding the WAL-index so that it is
synchronized with the WAL.


Recovery is run by the first thread to connect to a WAL-mode database.
Recovery restores the WAL-index so that it accurately describes the
WAL file.  If there is no WAL file present when the first thread connects
to the database, there is nothing to recover, but the recovery process
still runs to initialize the WAL-index.


If the WAL-index is implemented as a memory-mapped file and that file is
read-only to the first thread to connect, then that thread creates an
private heap-memory ersazt WAL-index and runs the recovery routine to
populate that private WAL-index.  The same data results, but it is held
privately rather that being written into the public shared memory area.


Recovery works by doing a single pass over the WAL, from beginning to end.
The checksums are verified on each frame of the WAL as it is read.  The
scan stops at the end of the file or at the first invalid checksum.
The mxFrame field is set to the index of the last valid commit frame
in WAL.  Since WAL frame numbers are indexed starting with 1, mxFrame is
also the number of valid frames in the WAL.  A "commit frame" is a frame
that has a non-zero value in bytes 4 through 7 of the frame header.
Since the recovery procedure has no way of knowing how many frames of the
WAL might have previously been copied back into the database, it initializes
the nBackfill value to zero.


During recovery of the global shared-memory WAL-index, exclusive locks are
held on WAL_WRITE_LOCK, WAL_CKPT_LOCK, WAL_RECOVER_LOCK, and WAL_READ_LOCK(1) through WAL_READ_LOCK(4).  In other words, all locks associated with the
WAL-index except for WAL_READ_LOCK(0) are held exclusively.  This prevents
any other thread from writing the database and from reading any transactions
that are held in the WAL, until the recovery is complete.


c2: The Carray() Table-Valued Function
c3: 1. Overview
c5: carray.html#overview
c4:
Carray($PTR,$N) is a table-valued function with a single column (named
"value") and zero or more rows.
The "value" of each row in the carray() is taken from a C-language array
that is $N elements long.  $PTR is a pointer to the beginning of the array.
In this way, the carray() function provides a convenient mechanism to
bind C-language arrays to SQL queries.


c2: The Carray() Table-Valued Function
c3: 2. Availability
c5: carray.html#availability
c4:
The carray() function is not compiled into SQLite by default.
It is available as a loadable extension in the
ext/misc/carray.c
source file.


c2: The Carray() Table-Valued Function
c3: 3. Details
c5: carray.html#details
c4:
The carray() function takes two or three arguments.
The first argument is a pointer to an array.  Since pointer values cannot
be specified directly in SQL, the first argument must be a parameter that
is bound to a pointer value using the sqlite3_bind_pointer() interface
using a pointer-type of "carray".
The second argument is the number of elements in the array.  The optional
third argument is a string that determines the datatype of the elements
in the C-language array.  Allowed values for the third argument are:


 'int32'
 'int64'
 'double'
 'char*'


The default datatype is 'int32'.

The carray() function can be used in the FROM clause of a query.
For example, to query two entries from the OBJ table using rowids
taken from a C-language array at address $PTR.

SELECT obj.* FROM obj, carray($PTR, 10) AS x
 WHERE obj.rowid=x.value;


This query gives the same result:

SELECT * FROM obj WHERE rowid IN carray($PTR, 10);


c2: SQLite Autoincrement
c3: 1. Summary
c5: autoinc.html#summary
c4:
The AUTOINCREMENT keyword imposes extra CPU, memory, disk space,
  and disk I/O overhead and should be avoided if not strictly needed.
  It is usually not needed.

  In SQLite, a column with type INTEGER PRIMARY KEY is an alias for the ROWID
  (except in WITHOUT ROWID tables) which is always a 64-bit signed integer.

  On an INSERT, if the ROWID or INTEGER PRIMARY KEY column is not 
  explicitly given a value, then it
  will be filled automatically with an unused integer, usually
  one more than the largest ROWID currently in use.
  This is true regardless of whether or not the AUTOINCREMENT keyword is used.

  If the AUTOINCREMENT keyword appears after INTEGER PRIMARY KEY, that
  changes the automatic ROWID assignment algorithm to prevent
  the reuse of ROWIDs over the lifetime of the database.  In other words,
  the purpose of AUTOINCREMENT is to prevent the reuse of ROWIDs from
  previously deleted rows.


c2: SQLite Autoincrement
c3: 2. Background
c5: autoinc.html#background
c4:
In SQLite, table rows normally have a 64-bit signed integer ROWID
which is unique among all rows in the same table.
(WITHOUT ROWID tables are the exception.)



You can access the ROWID of an SQLite table using one of the special column
names ROWID, _ROWID_, or OID.
Except if you declare an ordinary table column to use one of those special
names, then the use of that name will refer to the declared column not
to the internal ROWID.



If a table contains a column of type INTEGER PRIMARY KEY, then that
column becomes an alias for the ROWID.  You can then access the ROWID
using any of four different names, the original three names described above
or the name given to the INTEGER PRIMARY KEY column.  All these names are
aliases for one another and work equally well in any context.



When a new row is inserted into an SQLite table, the ROWID can either
be specified as part of the INSERT statement or it can be assigned
automatically by the database engine.  To specify a ROWID manually,
just include it in the list of values to be inserted.  For example:


CREATE TABLE test1(a INT, b TEXT);
INSERT INTO test1(rowid, a, b) VALUES(123, 5, 'hello');



If no ROWID is specified on the insert, or if the specified ROWID has a value
of NULL, then an appropriate ROWID is created
automatically.  The usual algorithm is to give the newly created row
a ROWID that is one larger than the largest ROWID in the table prior
to the insert.  If the table is initially empty, then a ROWID of 1 is
used.  If the largest ROWID is equal to the largest possible integer
(9223372036854775807) then the database
engine starts picking positive candidate ROWIDs at random until it finds one
that is not previously used.
If no unused ROWID can be found after a reasonable number of attempts,
the insert operation fails with an SQLITE_FULL error.
If no negative ROWID values are inserted explicitly, then automatically 
generated ROWID values will always be greater than zero.



The normal ROWID selection algorithm described above
will generate monotonically increasing
unique ROWIDs as long as you never use the maximum ROWID value and you never
delete the entry in the table with the largest ROWID. 
If you ever delete rows or if you ever create a row with the maximum possible
ROWID, then ROWIDs from previously deleted rows might be reused when creating
new rows and newly created ROWIDs might not be in strictly ascending order.


c2: SQLite Autoincrement
c3: 3. The AUTOINCREMENT Keyword
c5: autoinc.html#the_autoincrement_keyword
c4:
If a column has the type INTEGER PRIMARY KEY AUTOINCREMENT then a slightly
different ROWID selection algorithm is used.  
The ROWID chosen for the new row is at least one larger than the largest ROWID
that has ever before existed in that same table.  If the table has never
before contained any data, then a ROWID of 1 is used.  If the largest possible
ROWID has previously been inserted, then
new INSERTs are not allowed and any attempt to insert a new row will
fail with an SQLITE_FULL error.  
Only ROWID values from previous transactions that
were committed are considered.  ROWID values that were rolled back
are ignored and can be reused.



SQLite keeps track of the largest ROWID
using an internal table named "sqlite_sequence".
The sqlite_sequence table is created
and initialized automatically whenever a normal table that contains an
AUTOINCREMENT column is created.  The content of the sqlite_sequence table
can be modified using ordinary UPDATE, INSERT, and DELETE statements.
But making modifications to this table will likely perturb the AUTOINCREMENT
key generation algorithm.  Make sure you know what you are doing before
you undertake such changes.
The sqlite_sequence table does not track ROWID changes associated with
UPDATE statement, only INSERT statements.



The behavior implemented by the AUTOINCREMENT keyword is subtly different
from the default behavior.  With AUTOINCREMENT, rows with automatically
selected ROWIDs are guaranteed to have ROWIDs that have never been used
before by the same table in the same database.  And the automatically generated
ROWIDs are guaranteed to be monotonically increasing.  These are important
properties in certain applications.  But if your application does not
need these properties, you should probably stay with the default behavior
since the use of AUTOINCREMENT requires additional work to be done
as each row is inserted and thus causes INSERTs to run a little slower.


Note that "monotonically increasing" does not imply that the ROWID always
increases by exactly one.  One is the usual increment.  However, if an
insert fails due to (for example) a uniqueness constraint, the ROWID of
the failed insertion attempt might not be reused on subsequent inserts,
resulting in gaps in the ROWID sequence.  AUTOINCREMENT guarantees that
automatically chosen ROWIDs will be increasing but not that they will be
sequential.

Because AUTOINCREMENT keyword changes the behavior of the ROWID selection
algorithm, AUTOINCREMENT is not allowed on WITHOUT ROWID tables or on any
table column other than INTEGER PRIMARY KEY.  Any attempt to use 
AUTOINCREMENT on a WITHOUT ROWID table or on a column other than the
INTEGER PRIMARY KEY column results in an error.


c2: The CSV Virtual Table
c3: 1. Overview
c5: csv.html#overview
c4:
The CSV virtual table reads
RFC 4180 formatted comma-separated
values, and returns that content as if it were rows and columns of an SQL
table.



The CSV virtual table is useful to applications that need to bulk-load
large amounts of comma-separated value content.
The CSV virtual table is also useful as a template source file for
implementing other virtual tables.




The CSV virtual table is not built into the SQLite amalgamation.
It is available as a
separate source file
that can be compiled into a loadable extension.
Typical usage of the CSV virtual table from the
command-line shell would be something like this:

.load ./csv
CREATE VIRTUAL TABLE temp.t1 USING csv(filename='thefile.csv');
SELECT * FROM t1;



The first line of the script above causes the command-line shell to
read and activate the run-time loadable extension for CSV.  For an
application, the equivalent C-language API is
sqlite3_load_extension().
Observe that the filename extension (ex: ".dll" or ".so" or ".dylib") is
omitted from the extension filename.  Omitting the filename extension is
not required, but it helps in making the script cross-platform.  SQLite
will automatically append the appropriate extension.


The second line above creates a virtual table named "t1" that reads
the content of the file named in the argument.  The number and names of
the columns is determined automatically by reading the first line of
content.  Other options to the CSV virtual table provide the ability to
take the CSV content from a string rather than a separate file, and give 
the programmer more control over the number and names of the columns.
The options are detailed below.  The CSV virtual table is usually
created as a TEMP table so that it exists only for the current database
connection and does not become a permanent part of the database schema.
Note that there is no "CREATE TEMP VIRTUAL TABLE" command in SQLite.
Instead, prepend the "temp." schema prefix to the name of the virtual
table.


The third line of the example shows the virtual table being used, to read
all content of the CSV file.  This is perhaps the simplest possible use
of the virtual table.  The CSV virtual table can be used anywhere an ordinary
virtual table can be used.  One can use the CSV virtual table inside subqueries,
or common table expressions or add WHERE, GROUP BY, HAVING, ORDER BY,
and LIMIT clauses as required.


c2: The CSV Virtual Table
c3: 2. Arguments
c5: csv.html#arguments
c4:
The example above showed a single filename='thefile.csv' argument
for the CSV virtual table.  But other arguments are also possible.


filename=FILENAME
The filename= argument specifies an external file from which
CSV content is read.  Every CSV virtual table must have either a 
filename= argument or a data= argument and not both.

data=TEXT
The data= argument specifies that TEXT is the literal
content of the CSV file.

schema=SCHEMA
 The schema= argument specifies a CREATE TABLE statement that
the CSV virtual table passes to the sqlite3_declare_vtab() interface in
order to define the names of the columns in the virtual table.

columns=N
The columns=N argument specifies the number of columns
in the CSV file.
If the input data contains more columns than this,
then the excess columns are ignored.  If the input data contains fewer columns,
then extra columns are filled with NULL.
If the columns=N argument is omitted, the first line of the
CSV file is read to determine the number of columns.

header=BOOLEAN
or just
header
If the header argument is true then the first row of the CSV file
to be treated as a header rather than as data.  The second line of the CSV
file becomes the first row of content.
If the schema= options is omitted, then the first line of the CSV
file determines the names of the columns.


c2: The CSV Virtual Table
c3: 3. Column Names
c5: csv.html#column_names
c4:
The column names of the virtual table are determined primarily by the
schema= argument.
If the schema= argument is omitted, but header is true, then
the values found in the first line of the CSV file become the column names.
If the schema= argument is omitted and header is false, then
the columns are named "c0", "c1", "c2", and so forth.


c2: The SQLite Amalgamation
c3: 1. Executive Summary
c5: amalgamation.html#executive_summary
c4:
Over 100 separate source files are concatenated into a
single large files of C-code named "sqlite3.c" and
called "the amalgamation". The amalgamation
contains everything an application needs to embed SQLite.
The amalgamation file is more than 220,000 lines long and over 7.5
megabytes in size (as of 2018-11-24).

Combining all the code for SQLite into one big file makes SQLite
easier to deploy &mdash; there is just one file to keep track of.  
And because all code is in
a single translation unit, compilers can do better inter-procedure
optimization resulting in machine code that is between 5% and 10% faster.


c2: The SQLite Amalgamation
c3: 2. The SQLite Amalgamation
c5: amalgamation.html#the_sqlite_amalgamation
c4:
The SQLite library consists of 102 files of C code
(as of Version 3.9.0 - 2015-10-14)
in the core with 32 additional files that
implement the FTS3, FTS5, RTREE, DBSTAT, JSON1, and
RBU extensions.
Of the 102 
main source files, about 75% are C code and about 25% are C header files.
Most of these are "source" files in the sense that they are stored 
in the SQLite version control system
and are edited manually in an ordinary text editor.
But some of the C-language files are generated using scripts
or auxiliary programs.  For example, the
parse.y
file contains an LALR(1) grammar of the SQL language which is compiled
down into are parser in files "parse.c" and "parse.h" by the
Lemon parser generator.


The makefiles for SQLite have an "sqlite3.c" target for building the
file we call "the amalgamation".
The amalgamation is a single C code file, named "sqlite3.c",
that contains all C code 
for the core SQLite library and the FTS3, FTS5, RTREE,
DBSTAT, JSON1, and RBU extensions.
This file contains about 184K lines of code 
(113K if you omit blank lines and comments) and is over 6.4 megabytes
in size.
Though the various extensions are included in the
"sqlite3.c" amalgamation file, they are disabled using #ifdef statements.
Activate the extensions using compile-time options like:


 -DSQLITE_ENABLE_FTS3
 -DSQLITE_ENABLE_FTS5
 -DSQLITE_ENABLE_RTREE
 -DSQLITE_ENABLE_DBSTAT_VTAB
 -DSQLITE_ENABLE_JSON1
 -DSQLITE_ENABLE_RBU


The amalgamation contains everything you need to integrate SQLite 
into a larger project.  Just copy the amalgamation into your source 
directory and compile it along with the other C code files in your project.
(A more detailed discussion of the compilation process is
available.)
You may also want to make use of the "sqlite3.h" header file that 
defines the programming API for SQLite. 
The sqlite3.h header file is available separately. 
The sqlite3.h file is also contained within the amalgamation, in
the first few thousand lines. So if you have a copy of 
sqlite3.c but cannot seem to locate sqlite3.h, you can always 
regenerate the sqlite3.h by copying and pasting from the amalgamation.

In addition to making SQLite easier to incorporate into other
projects, the amalgamation also makes it run faster. Many
compilers are able to do additional optimizations on code when 
it is contained with in a single translation unit such as it 
is in the amalgamation. We have measured performance improvements
of between 5 and 10% when we use the amalgamation to compile 
SQLite rather than individual source files.  The downside of this
is that the additional optimizations often take the form of 
function inlining which tends to make the size of the resulting
binary image larger.


c2: The SQLite Amalgamation
c3: 3. The Split Amalgamation
c5: amalgamation.html#the_split_amalgamation
c4:
Developers sometimes experience trouble debugging the
185,000-line-long amalgamation source file because some debuggers
are only able to handle source code line numbers less than 32,768.
The amalgamation source code runs fine.  One just cannot single-step
through it in a debugger.

To circumvent this limitation, the amalgamation is also available in
a split form, consisting of files "sqlite3-1.c", "sqlite3-2.c", and
so forth, where each file is less than 32,768 lines in length and
where the concatenation of the files contain all of the code for the
complete amalgamation.  Then there is a separate source file named
"sqlite3-all.c" which basically consists of code like this:

#include "sqlite3-1.c"
#include "sqlite3-2.c"
#include "sqlite3-3.c"
#include "sqlite3-4.c"
#include "sqlite3-5.c"
#include "sqlite3-6.c"
#include "sqlite3-7.c"


Applications using the split amalgamation simply compile against
"sqlite3-all.c" instead of "sqlite3.c".  The two files work exactly
the same.  But with "sqlite3-all.c", no single source file contains more
than 32,767 lines of code, and so it is more convenient to use some
debuggers.  The downside of the split amalgamation is that it consists
of 6 C source code files instead of just 1.


c2: The SQLite Amalgamation
c3: 4. Download Copies Of The Amalgamation
c5: amalgamation.html#download_copies_of_the_amalgamation
c4:
The amalgamation and
the sqlite3.h header file are available on
the download page as a file 
named sqlite-amalgamation-X.zip
where the X is replaced by the appropriate version number.


c2: The SQLite Amalgamation
c3: 5. Building The Amalgamation From Canonical Source Code
c5: amalgamation.html#building_the_amalgamation_from_canonical_source_code
c4:
To build the amalgamation (either the full amalgamation or the
split amalgamation), first
get the canonical source code from one of the three servers.
Then, on both unix-like systems and on Windows systems that have the
free MinGW development environment
installed, the amalgamation can be built using the
following commands:

sh configure
make sqlite3.c


To build using Microsoft Visual C++, run this command:

nmake /f makefile.msc sqlite3.c


In both cases, the split amalgamation can be obtained by
substituting "sqlite3-all.c" for "sqlite3.c" as the make target.


c2: The SQLite Amalgamation
c3: 5.1. Dependencies
c5: amalgamation.html#dependencies
c4:
The build process makes extensive use of the 
Tcl scripting language.  You will need to have a
copy of TCL installed in order for the make targets above to work.
Easy-to-use installers can be obtained from http://www.tcl-lang.org/.
Many unix workstations have Tcl installed by default.


c2: List Of Virtual Tables
c3: 1. Introduction
c5: vtablist.html#introduction
c4:
A virtual table is an object that presents an SQL table
interface but which is not stored in the database file, at least
not directly.  The virtual table mechanism is a feature
of SQLite that allows SQLite to access and manipulate resources
other than bits in the database file using the powerful SQL
query language.


The table below lists a few of the virtual tables implementations
available for SQLite.  Developers can deploy these virtual tables
in their own applications, or use the implementations shown below
as templates for writing their own virtual tables.


The list below is not exhaustive.
Other virtual table implementation exist in the SQLite source tree
and elsewhere.
The list below tries to capture the more interesting virtual table
implementations.


c2: List Of Virtual Tables
c3: 2. Virtual Tables
c5: vtablist.html#virtual_tables
c4:
NameDescription
approximate_match
A demonstration of how to use a virtual table to implement
    approximate string matching.
carray
A table-valued function that allows a C-language array of integers,
    doubles, or strings to be used as table in a query.
closure
Compute the transitive closure of a set.
completion
Suggests completions for partially-entered words during interactive
    SQL input.  Used by the CLI to help implement tab-completion.
csv
A virtual table that represents a comma-separated-value or CSV file
    (RFC 4180) as a read-only
    table so that it can be used as part of a larger query.
dbstat
Provides information about the purpose and use of each page in a
    database file.  Used in the implementation of the
    sqlite3_analyzer utility program.
files_of_checkin
Provides information about all files in a single check-in in the
    Fossil version control system.  This
    virtual table is not part of the SQLite project but is included because
    it provides an example of how to use virtual tables and because it is
    used to help version control the SQLite sources.
fsdir
A table-valued function returning one row for each file in a
    selected file hierarchy of the host computer.  Used by the
    CLI to help implement the .archive command.
FTS3
A high-performance full-text search index.
FTS5
A higher-performance full-text search index
generate_series
A table-valued function returning a sequence of increasing
    integers, modeled after the table-valued function by the same
    name in PostgreSQL.
json_each
A table-valued function for decomposing a JSON string.
json_tree
A table-valued function for decomposing a JSON string.
OsQuery
Hundreds of virtual tables that publish various aspects of the
    host computer, such as the process table, user lists, active network
    connections, and so forth.
    OsQuery is a separate project, started by Facebook, hosted on
    GitHub, and intended for
    security analysis and intrusion detection
    OsQuery is not a part of the SQLite project, but is included in this
    list because it demonstrates how the SQL language and the
    SQLite virtual table mechanism
    can be leveraged to provide elegant solutions
    to important real-world problems.
pragma
Built-in table-valued functions that return the results of PRAGMA
    statements for use within ordinary SQL queries.
RTree
An implementation of the Guttmann R*Tree spatial index idea.
spellfix1
A virtual table that implements a spelling correction engine.
sqlite_btreeinfo
This experimental table-valued function provides information about
    a single B-tree in a database file, such as the depth, and estimated
    number of pages and number of entries, and so forth.
sqlite_dbpage
Key/value store for the raw database file content.  The key is the
    page number and the value is binary page content.
sqlite_memstat
Provides SQL access to the sqlite3_status64() and
    sqlite3_db_status() interfaces.
sqlite_stmt
A table-valued function containing one row for each
    prepared statement associated with an open database connection.
swarmvtab
An experimental module providing on-demand read-only access to
    multiple tables spread across multiple databases, via a single
    virtual table abstraction.
tclvar
Represents the global variables of a 
    TCL Interpreter as an SQL
    table. Used as part of the SQLite test suite.
templatevtab
A template virtual table implementation useful as a starting point
    for developers who want to write their own virtual tables
unionvtab
An experimental module providing on-demand read-only access to
    multiple tables spread across multiple databases, via a single
    virtual table abstraction.
vfsstat
A table-valued function which, in combination with a
    co-packaged VFS shim provides information on the number of
    system calls performed by SQLite.
vtablog
A virtual table that prints diagnostic information
    on stdout when its key methods are invoked.  Intended for
    interactive analysis and debugging of virtual table interfaces.
wholenumber
A virtual table returns all integers between 1 and 4294967295.
zipfile
Represent a
    ZIP Archive
    as an SQL table.  Works for both reading and writing.  Used by
    the CLI to implement the ability to read and write ZIP Archives.


c2: Vulnerabilities
c3: 1. Executive Summary
c5: cves.html#executive_summary
c4:
CVEs about SQLite probably do not apply to your use of SQLite.


All historical vulnerabilities reported against SQLite require at least
one of these preconditions:


The attacker can submit and run arbitrary SQL statements.

The attacker can submit a maliciously crafted database file to the
application that the application will then open and query.



Few real-world applications meet either of these preconditions, and hence
few real-world applications are vulnerable, even if they use older
and unpatched versions of SQLite.


The SQLite development team fixes bugs promptly,
usually within hours of discovery.  New releases of SQLite
are issued if the bug seems likely to impact real-world
applications.


Grey-hat hackers are rewarded based on the number and severity of 
CVEs that they write.  This results in a proliferation of CVEs that
have minor impact, or no impact at all, but which make
exaggerated impact claims.


Very few CVEs written about SQLite are real vulnerabilities in the 
sense that they do not give any new capabilities to an attacker.
Consider:


    Almost all CVEs written against SQLite require the ability to
    inject and run arbitrary SQL.

    The advertised consequence of most CVEs is "denial of service",
    typically by causing a crash through a NULL pointer dereference or
    a division by zero, or similar.

    But if an attacker can already run
    arbitrary SQL, they do not need a bug to cause a denial of service.
    There are plenty of perfectly legal and valid SQL statements
    that will consume unlimited CPU, memory, and disk I/O in order
    to create a denial-of-service without requiring help from bugs.

    Hence, the mere fact that an attacker has a way to inject and run
    arbitrary SQL is in and of itself a denial-of-service attack.  That
    the arbitrary SQL might also tickle a bug in SQLite and cause a
    crash is not a new vulnerability.



The SQLite developers do not write CVEs.  Any CVEs you find on
SQLite are generated by third-parties, often without any input from the
core developers.  A common scenario is that someone will report a bug in
SQLite, which will promptly be fixed, then weeks later a CVE for that bug will
appear, unbeknownst to the developers.


You should not assume that a CVE about
SQLite contains authoritative information.
CVEs often contain inaccuracies.
The SQLite developers have attempted to add clarifications and
corrections to CVEs about SQLite, but without success.


c2: Vulnerabilities
c3: 2. About CVEs
c5: cves.html#about_cves
c4:
CVEs ("Common Vulnerabilities and Exposures") are reports of software
bugs that might allow a system to be hacked.  The idea
behind CVEs is sound.  They provide a common naming scheme whereby 
software bugs that might compromise information security can be easily
tracked.

While the original idea being CVEs is sound, the current processes for
creating and managing CVEs are inadequate.  There are countless grey-hat
hackers running fuzzers against a wide-variety of open-source software
products (SQLite as well as many others) and writing up CVEs against
any problems they find.  The grey-hats are rewarded, sometimes with
prestige and sometimes financially, by the number and severity of
the CVEs they write.  This incentive results in a proliferation
of CVEs which are often not well-vetted and which can have exaggerated
impact claims.  The quality-control procedures for CVEs are unable
to cope with this flood of inputs, making it difficult to correct
exaggerated, misleading, omitted, or inaccurate claims.

This is not to say that CVEs are useless.  CVEs do still (mostly)
report actual bugs.  But in most cases the bugs are not true vulnerabilities,
in the sense that they do not contribute to data loss or compromise
in and of themselves.
It is good that bugs are reported and fixed.  But not every bug is
accessible from every application.  In the case of SQLite, most of the
bugs reported by CVEs are inaccessible in most applications.  Upgrading
to the latest version of SQLite is always a good plan, but it need not
be an emergency just because an anonymous grey-hat on the internet
wrote up a CVE.


c2: Vulnerabilities
c3: 2.1. A separate SQL injection vulnerability is usually required
c5: cves.html#a_separate_sql_injection_vulnerability_is_usually_required
c4:
Other C-libraries that process complex structured inputs will
routinely be asked to deal with unvetted inputs from untrusted
sources.  Libraries like libjpeg, or libzip, or OpenSSL are
handed input streams that come directly from potentially hostile
agents.


But database engines like SQLite are usually not this way.
The SQL scripts that are passed into SQLite come from the
(trusted) application itself, not from an attacker.  Sometimes
applications contain bugs by which an external attacker can
trick the application into sending SQL of the attackers design
into the database engine.  This is a separate bug in the
application called an
SQL Injection
vulnerability.  Since SQL text is executable code, an
SQL Injection vulnerability is actually a special case of a
Remote
Code Execution (RCE) vulnerability.  An SQL Injection is perhaps not
quite as bad as other kinds of RCEs because,
while SQL is a powerful language, it is not as convenient
for crafting an exploit as Python or shell script or raw machine code.
Nevertheless, an SQL Injection is a serious problem.


Most CVEs written about SQLite assume that the attacker is
able to run arbitrary SQL scripts in SQLite.  In most applications,
this means that there must first be an SQL Injection vulnerability
that allows the attacker to inject the malicious SQL.


A few applications do allow untrusted SQL scripts received from
potentially hostile agents to be run direct in SQLite.  The main
example of this is the Chrome and Safari web browsers, which allow
an anonymous web page to run SQL using the WebSQL feature of Javascript.
This is done inside a sandbox with tightly controlled constraints on
resources, lest the SQL script try to soak up all available memory
or CPU cycles in a denial-of-service attack.  Chrome and Safari
have the infrastructure in place to allow a hostile agent to run
code which does not harm or compromise the rest of the machine.
They have to, as they also run Javascript which could, if not
tightly controlled, do even more damage than unrestrained SQL.
Apart from Chrome and Safari, no applications known to the
SQLite developers deliberately allows an anonymous remote agent
to run arbitrary SQL text.

However, most CVEs written against SQLite flippantly assume
that an attacker is free to run any arbitrary SQL in the database
engine.  So to a good approximation, this means most CVEs
written against SQLite really only apply to SQLite as it is
used in Chrome and Safari.  Or, in other words, most CVEs
for SQLite do not apply to you unless you are one of the
developers of Chrome or Safari.


c2: Vulnerabilities
c3: 2.2. Defense Against Dark Arts
c5: cves.html#defense_against_dark_arts
c4:
Most applications can use SQLite without having to worry about
bugs in obscure SQL inputs.  If the application controls
the SQL, and the application is not deliberately trying to break
SQLite, then everything should just work.
It is not necessary to have the latest patched version of SQLite.
Any older version should work just fine.


However, there are some occasions where an application does need
to be able to safely run untrusted SQL. The SQLite developers work hard
to make SQLite safe for this purpose, though there are occasional
slip-ups.  It is good to keep up-to-date with the latest patches
in this case.  The separate defense against dark arts document
contains additional suggestions that can help prevent zero-day
attacks in cases where SQLite is given inputs that come directly
from untrusted sources.


c2: Vulnerabilities
c3: 2.3. The SQLite Developer Policy Toward CVEs
c5: cves.html#the_sqlite_developer_policy_toward_cves
c4:
SQLite developers fix all bugs in SQLite as soon as they are reported,
usually within a few hours.  The fixes are immediately available on the
public SQLite source tree.
If a bug seems like it might cause problems for existing applications,
a new patch release for SQLite will be issued.

However, the SQLite developers do not track CVEs.  There are 
various reasons for this:



The developers often do not find out about CVEs until long after the
bug is fixed.  You can see this by the fact that many CVEs reference the
bug fix in their initial report.


CVEs are a low-quality source of information about bugs in SQLite
that are likely to affect most applications.


Almost all bugs reported by CVEs are just bugs and not
true vulnerabilities.  Claiming that they are vulnerabilities is
stretching the meaning of the word "vulnerability" and the SQLite
developers do not wish to participate in that deception.


The developers have no editorial influence on the content of CVEs,
and they do not like to be controlled by groups in which they have
no voice.


c2: Vulnerabilities
c3: 3. Status Of Recent SQLite CVEs
c5: cves.html#status_of_recent_sqlite_cves
c4:
Though the SQLite developers do not consider CVEs to be a reliable
source of information about bugs in SQLite, they recognize that many
groups, and especially small teams working at the bottom of tall
bureaucracies, sometimes need to track CVEs, whether they are useful
or not.  To aid in this chore, the following table of recent CVEs
affecting SQLite is provided.

If you notice new CVEs associated with SQLite that are not in
the table below, please bring them to the attention of the developers
on the SQLite Forum so they can
be added.




CVE Number
Fix
Comments





CVE-2020-15358

3.32.3(2020-06-18)
Malicious SQL statement causes an read past the end of a heap buffer.
(details)



CVE-2020-13871

3.32.3(2020-06-18)
Malicious SQL statement causes a read-only use-after-free memory error.
(details)



CVE-2020-13632

3.32.0(2020-05-22)
Malicious SQL statement causes a read of a NULL pointer in the
  matchinfo() SQL function of the FTS3 extension, resulting in
  denial of service.
(details)



CVE-2020-13631

3.32.0(2020-05-22)
Malicious SQL statement (an ALTER TABLE that tries to rename a
  virtual table into one of its own shadow tables)
  causes an infinite loop and denial of service.
(details)



CVE-2020-13630

3.32.0(2020-05-22)
Malicious SQL statement causes a read-only use-after-free,
  possibly resulting in a incorrect output from the snippet()
  SQL function of the FTS3 extension.  There is no known
  way to exfiltrate data or crash the application using this bug.
(details)



CVE-2020-13435

3.32.1(2020-05-25)
Malicious SQL statement causes a read access to a NULL pointer and
  denial of service.
(details)



CVE-2020-13434

3.32.1(2020-05-25)
Malicious SQL statement involving the printf() SQL function results
  in an integer overflow which can overwrite the stack with over 2
  billion bytes of 0x30 or 0x20 (ASCII '0' or ' ').
  Even though this is a stack overwrite, there is no known way to
  redirect control or otherwise escalate the level of harm.
  This is a denial-of-service attack only.
(details)



CVE-2020-11656

3.32.0(2020-05-22)
Malicious SQL statement causes read-only use-after-free of memory allocation
  if SQLite is compile with -DSQLITE_DEBUG.  Does not affect release
  builds.
(details)



CVE-2020-11655

3.32.0(2020-05-22)
Malicious SQL statement causes a read using an uninitialized pointer
  and denial-of-service.
(details)



CVE-2020-9327

3.32.0(2020-05-22)
Malicious SQL statement causes a read using an uninitialized pointer
  and denial-of-service
(details)



CVE-2020-6405

3.31.0(2020-01-22)
Malicious SQL statement causes a NULL pointer dereference and
  denial-of-service
(details)



CVE-2019-20218

3.31.0(2020-01-22)
Malicious SQL statement causes an uninitialized pointer read and
  denial-of-service.
(details)



CVE-2019-19959

3.31.0(2020-01-22)
Malicious SQL statement causes a NULL pointer dereference
  in the Zipfile virtual table extension and
  denial-of-service.  This is only possible when the optional
  Zipfile virtual table extension is deployed, which is not
  the case in default builds.
(details)



CVE-2019-19926

3.31.0(2020-01-22)
Malicious SQL statement causes an uninitialized pointer read and
  denial-of-service.
(details)



CVE-2019-19925

3.31.0(2020-01-22)
Malicious SQL statement causes a NULL pointer dereference and
  in the Zipfile virtual table extension and
  denial-of-service.  This is only possible when the optional
  Zipfile virtual table extension is deployed, which is not
  the case in default builds.
(details)



CVE-2019-19924

3.31.0(2020-01-22)
Malicious SQL statement causes a uninitialized pointer reference and
  denial-of-service.
(details)



CVE-2019-19923

3.31.0(2020-01-22)
Malicious SQL statement causes a NULL pointer dereference and
  denial-of-service.
(details)



CVE-2019-19317

3.31.0(2020-01-22)
This CVE identifies a bug in a development check-in of
  SQLite.  The bug never appeared in any official SQLite release.
(details)


c2: Pointer Passing Interfaces
c3: 1. Overview
c5: bindptr.html#overview
c4:
Three new "_pointer()" interfaces were added to SQLite 3.20.0 (2017-08-01):

 sqlite3_bind_pointer(),
 sqlite3_result_pointer(), and
 sqlite3_value_pointer().


Questions and confusion quickly arose
on the mailing lists about the purpose behind these new interfaces,
why they were introduced, and what problem they solve.  This essay attempts
to answer those questions and clear up the confusion.


c2: Pointer Passing Interfaces
c3: 2. A Brief History Of Pointer Passing In SQLite
c5: bindptr.html#a_brief_history_of_pointer_passing_in_sqlite
c4:
It is sometimes convenient for SQLite extensions to
communicate non-SQL values between subcomponents or between the extension
and the application.  Some examples:



In the FTS3 extension, the 
MATCH operator (which does the full-text search)
needs to communicate details of matching entries 
to the snippet(), offsets(),
and matchinfo() functions so that those functions can convert the details
of the match into useful output.


In order for an application to 
add new extensions to FTS5, such as new tokenizers, 
the application needs a pointer to the "fts5_api" object.


In the CARRAY extension, the application needs to tell the 
extension the
location of a C-language array that contains the data for the table-valued
function that the extension implements.



The traditional way of communicating this information was to transform a
C-language pointer into a BLOB or a 64-bit integer, then move that BLOB or
integer through SQLite using the usual interfaces like
sqlite3_bind_blob(), sqlite3_result_blob(), sqlite3_value_blob() or
the integer equivalents.


c2: Pointer Passing Interfaces
c3: 2.1. Upping The Threat Level
c5: bindptr.html#upping_the_threat_level
c4:
Passing around pointers as if they were integers or BLOBs is easy,
effective, and works well in an environment where the application
components are all friendly toward one another.  However, passing pointers
as integers and BLOBs allows hostile SQL text to forge invalid pointers that
can carry out mischief.


For example, the first argument to the snippet() function is supposed to
be a special column of the FTS3 table that contains a pointer to an fts3cursor
object that contains information about the current full text search match.
That pointer was formerly passed as a BLOB.  
For example, if the FTS3 table is named "t1" and has a column named "cx",
one might write:

SELECT snippet(t1) FROM t1 WHERE cx MATCH $pattern;



But if a hacker is able to run arbitrary SQL, he might run a slightly
different query, like this:

SELECT hex(t1) FROM t1 WHERE cx MATCH $pattern;



Because the pointer is passed in the t1 column of the t1
table as a BLOB (in older versions of SQLite), such a query would have 
shown the value of the
pointer in hex.  The attacker could then modify that pointer to try to
get the snippet() function to modify memory in some other part of 
the application address space instead of the fts3cursor object it 
was supposed to be operating on:

SELECT snippet(x'6092310100000000') FROM t1 WHERE cx MATCH $pattern;



Historically, this was not considered a threat.  The argument was that if
a hostile agent is able to inject arbitrary SQL text into the application,
then that agent is already in full control of the application, so
letting the hostile agent forge a pointer does not give the agent
any new capability.


For most cases, it is true that potential attackers have no way of injecting
arbitrary SQL, and so most uses of SQLite are immune to the attack above.
But there are some notable exceptions.  To wit:



The WebSQL interface
to webkit allowed any webpage to run arbitrary SQL in the browser
for Chrome and Safari.  That arbitrary SQL was supposed to be run inside
a sandbox where it could do no harm even if exploited, but that sandbox
turned out to be less secure than people supposed.  In the spring of 2017, 
one team of hackers was able to root an iMac using a long sequence of 
exploits, one of which involved corrupting the pointers passed as BLOB 
values to the snippet() FTS3 function of an SQLite database running via
the WebSQL interface inside of Safari.


On Android, we are told, there are many services that will blindly 
run arbitrary SQL that is passed to them by untrustworthy apps
that have been downloaded from dodgy corners of the internet.
Android services are suppose to be more guarded about running SQL
from unvetted sources.  This author does not have any specific examples
to the contrary, but he has heard rumors that they exist.  Even if
all Android services are more careful and properly vet all the SQL
they run, it would be difficult to audit them
all in order to verify that they are safe.  Hence, security-minded people
are keen to ensure that no exploits are possible by passing arbitrary
SQL text.


The Fossil version control system (designed
and written for the purpose of supporting SQLite development) allows
mildly trusted users to enter arbitrary SQL for generating trouble-ticket
reports.  That SQL is sanitized using the
sqlite3_set_authorizer() interface, and no exploits have ever been
found.  But this is an example of potentially hostile agents being able to
inject arbitrary SQL into the system.


c2: Pointer Passing Interfaces
c3: 2.2. Preventing Forged Pointers
c5: bindptr.html#preventing_forged_pointers
c4:
The first attempt at closing security gaps in pointer passing was to
prevent pointer values from being forged.  This was accomplished by
having the sender attach a subtype to each pointer using
sqlite3_result_subtype() and having the receiver verify that subtype
using sqlite3_value_subtype() and reject pointers that had an incorrect
subtype.  Since there is no way to attach a subtype to a result using
pure SQL, this prevents pointers from being forged using SQL.  The only
way to send a pointer is to use C code.  If an attacker can set a subtype,
then he is also able to forge a pointer without the help of SQLite.


Using subtypes to identify valid pointers prevented the WebSQL exploit.
But it turned out to be an incomplete solution.


c2: Pointer Passing Interfaces
c3: 2.3. Pointer Leaks
c5: bindptr.html#pointer_leaks
c4:
The use of subtypes on pointers prevented pointer forgery using
pure SQL.  But subtypes do nothing to prevent an attacker from reading
the values of pointers.  In other words, subtypes on pointer values
prevent attacks using SQL statements like this:

SELECT snippet(x'6092310100000000') FROM t1 WHERE cx MATCH $pattern;



The BLOB argument to snippet() does not have the correct subtype, so the
snippet function ignores it, makes no changes to any data structures,
and harmlessly returns NULL.


But the use of subtypes does nothing to prevent the value of a
pointer from being read using SQL code like this:

SELECT hex(t1) FROM t1 WHERE cx MATCH $pattern;



What harm could come of that, you ask?  The SQLite developers (including
this author) wondered the same thing.  But then security researchers
pointed out that knowledge of pointers can help attackers to circumvent
address-space randomization defenses.  This is called a "pointer leak".
A pointer leak is not itself a vulnerability, but it can aid an attacker
in effectively exploiting other vulnerabilities.


c2: Pointer Passing Interfaces
c3: 3. The New Pointer-Passing Interfaces
c5: bindptr.html#the_new_pointer_passing_interfaces
c4:
Allowing extension components to pass private information to one another
securely and without introducing pointer leaks requires new interfaces:


sqlite3_bind_pointer(S,I,P,T,D) &rarr;
Bind pointer P of type T to the I-th parameter of prepared statement S.
D is an optional destructor function for P.
sqlite3_result_pointer(C,P,T,D) &rarr;
Return pointer P of type T as the argument of function C.
D is an optional destructor function for P.
sqlite3_value_pointer(V,T) &rarr;
Return the pointer of type T associated with value V, or if V has no
associated pointer, or if the pointer on V is of a type different from
T, then return NULL.



To SQL, the values created by sqlite3_bind_pointer() and 
sqlite3_result_pointer() are indistinguishable from NULL.  An
SQL statement that tries to use the hex() function to read the
value of a pointer will get an SQL NULL answer.  The only way to
discover whether or not a value has an associated pointer is to
use the sqlite3_value_pointer() interface with the appropriate
type string T. 


Pointer values read by sqlite3_value_pointer()
cannot be generated by pure SQL.  Hence, it is not possible for SQL to
forge pointers.


Pointer values generated by sqlite3_bind_pointer() and
sqlite3_result_pointer() cannot be read by pure SQL.
Hence, it is not possible for SQL to leak the value of pointers.


In this way the new pointer-passing interface seems to solve all of the
security problems associated with passing pointer values from one
extension to another in SQLite.


c2: Pointer Passing Interfaces
c3: 3.1. Pointer Types
c5: bindptr.html#pointer_types
c4:
The "pointer type" in the last parameter to sqlite3_bind_pointer(),
sqlite3_result_pointer(), and sqlite3_value_pointer() is used to prevent
pointers intended for one extension from being redirected to a different
extension.  For example, without the use of pointer types, an attacker 
could still get access to pointer information in a system that included 
both the FTS3 and the CARRAY extension using SQL like this:

SELECT ca.value FROM t1, carray(t1,10) AS ca WHERE cx MATCH $pattern



In the statement above, the FTS3 cursor pointer generated by the
MATCH operator is send into the carray() table-valued function instead
of its intended recipient snippet().  The carray() function treats the
pointer as a pointer to an array of integers and returns each integer
one by one, thus leaking the content of the FTS3 cursor object.  Since
the FTS3 cursor object contains pointers to other objects, the statement
above would be a pointer leak.


Except, the statement above does not work, thanks to pointer types.
The pointer generated by the MATCH operator has a type of "fts3cursor"
but the carray() function expects to receives a pointer of type "carray".
Because the pointer type on the sqlite3_result_pointer() does not match
the pointer type on the sqlite3_value_pointer() call, 
sqlite3_value_pointer() returns NULL in carray() and thus signals
the CARRAY extension that it has been passed an invalid pointer.


c2: Pointer Passing Interfaces
c3: 3.1.1. Pointer types are static strings
c5: bindptr.html#pointer_types_are_static_strings
c4:
Pointer types are static strings, which ideally should be string literals
embedded directly in the SQLite API call, not parameters passed in from
other functions.  Consideration was given to using integer values as
the pointer type, but static strings provides a much larger name space
which reduces the chance of accidental type-name collisions between
unrelated extensions.


By "static string", we mean a zero-terminated array of bytes that is
fixed and unchanging for the life of the program.  In other words, the
pointer type string should be a string constant.
In contrast, a "dynamic string" is a zero-terminated array of bytes
that is held in memory allocated
from the heap, and which must be freed to avoid a memory leak.
Do not use dynamic strings as the pointer type string.


Multiple commentators have expressed a desire to use dynamic strings
for the pointer type, and to have SQLite take ownership of the type strings
and to automatically free the type string
when it has finished using it.  That design is rejected for the
following reasons:



The pointer type is not intended to be flexible and dynamic.  The
pointer type is intended to be a design-time constant.  Applications
should not synthesize pointer type strings at run-time.  Providing
support for dynamic pointer type strings would lead developers
to misuse the pointer-passing interfaces by creating run-time
synthesized pointer type strings.  Requiring the pointer type strings
to be static encourages developers to do the right thing by choosing
fixed pointer type names at design-time and encoding those names
as constant strings.


All string values at the SQL level in SQLite are dynamic strings.  
Requiring type strings to be static makes it difficult to 
create an application-defined SQL function that
can synthesize a pointer of an arbitrary type.  We do not want users
to create such SQL functions, since such functions would compromise the
security of the system.  Thus, the requirement to use static strings
helps to defend that the integrity of the pointer-passing interfaces against
ill-designed SQL functions.  The static string requirement is not
a perfect defense, since a sophisticated programmer can code around
it, and a novice program can simply take the memory leak.  But by
stating that the pointer type string must be static, we hope to encourage
developers who might otherwise use a dynamic string for the pointer type
to think more carefully about the problem and avoid introducing
security issues.


Having SQLite take ownership of the type strings would impose a performance
cost on all applications, even applications that do not use the 
pointer-passing interfaces.  SQLite passes values around as instances
of the sqlite3_value object.  That object has a destructor, which because
of the fact that sqlite3_value objects are used for nearly everything, is
invoked frequently.  If the destructor needs to check to see if there is
a pointer type string that needs to be freed, that is a few extra CPU
cycles that need to be burned on each call to the destructor.  Those
cycles add up.  We would be willing to bear the cost of the extra CPU
cycles if pointer-passing was a commonly used programming paradigm, but
pointer-passing is rare, and so it seems unwise to impose a run-time cost
on billions and billions of applications that do not use pointer passing
just for convenience of a few applications that do.



If you feel that you need dynamic pointer type strings in your application,
that is a strong indicator that you are misusing the pointer-passing interface.
Your intended use may be unsafe.
Please rethink your design.  Determine if you really need to be passing
pointers through SQL in the first place.  Or perhaps find a different
mechanism other than the pointer-passing interfaces described by this
article.


c2: Pointer Passing Interfaces
c3: 3.2. Destructor Functions
c5: bindptr.html#destructor_functions
c4:
The last parameter to the sqlite3_bind_pointer() and
sqlite3_result_pointer() routines is a pointer to a procedure
used to dispose of the P pointer once SQLite has finished with it.
This pointer can be NULL, in which case no destructor is called.


When the D parameter is not NULL, that means that ownership of the
pointer is being transferred to SQLite.  SQLite will take responsibility
for freeing resources associated with the pointer when it has finished
using the pointer.  If the D parameter is NULL, that means that ownership
of the pointer remains with the caller and the caller is responsible for
disposing of the pointer.


Note that the destructor function D is for the pointer value P, not for
the type string T.  The type string T should be a static string with an
infinite lifetime.


If ownership of the pointer is passed into SQLite by providing a
non-NULL D parameter to sqlite3_bind_pointer() or sqlite3_result_pointer()
then the ownership remains with SQLite until the object is destroyed.
There is no way to transfer ownership out of SQLite and back into the
application again.


c2: Pointer Passing Interfaces
c3: 4. Restrictions On The Use of Pointer Values
c5: bindptr.html#restrictions_on_the_use_of_pointer_values
c4:
The pointers that piggy-back on SQL NULL values using the
sqlite3_bind_pointer(), sqlite3_result_pointer(), and
sqlite3_value_pointer() interface are transient and ephemeral.
The pointers are never written into the database.  The pointers
will not survive sorting.  The latter fact is why there is no
sqlite3_column_pointer() interface, since it is impossible to
predict whether or not the query planner will insert a sort operation
prior to returning a value from a query, so it would be impossible to
know if a pointer value inserted into a query by
sqlite3_bind_pointer() or sqlite3_result_pointer() would survive
through to the result set.


Pointer values must flow directly from their producer into their
consumer, with no intermediate operators or functions.  Any transformation
of a pointer value destroys the pointer and transforms the value into
an ordinary SQL NULL.


Both the pointer and the pointer type parameter to the
sqlite3_bind_pointer() and sqlite3_result_pointer() interfaces
are "owned" by the caller.  In other words, the caller is responsible
for ensuring that both values remain valid until after the last access
via sqlite3_value_pointer().


c2: Pointer Passing Interfaces
c3: 5. Summary
c5: bindptr.html#summary
c4:
Key take-aways from this essay:



The internet is an increasingly hostile place.  These day, developers
should assume that attackers will find a way to execute arbitrary SQL 
in an application.
Applications should be designed to prevent the execution of arbitrary
SQL from escalating into a more severe exploit.


A few SQLite extensions benefit from passing pointers:

The FTS3 MATCH operator passes pointers into snippet(),
    offsets(), and matchinfo().
The carray table-valued function needs to accept a pointer to
    an array of C-language values from the application.
The remember() extension
    needs a pointer to a C-language integer variable in which to remember
    the value it passes.
Applications need to receive a pointer to the "fts5_api" object in order
    to add extensions, such as custom tokenizers, to the FTS5 extension.



Pointers should never be exchanged by encoding them as some other
SQL datatype, such as integers or BLOBs.  Instead, use the interfaces
designed to facilitate secure pointer passing:
sqlite3_bind_pointer(), sqlite3_result_pointer(), and
sqlite3_value_pointer().


The use of pointer-passing is an advanced technique that should be used
infrequently and cautiously.  Pointer-passing should not be
used haphazardly or carelessly.  Pointer-passing is a sharp tool 
that can leave deep scars if misused.


The "pointer type" string which is the last parameter to each of the
pointer-passing interfaces should be a distinct, application-specific
string literal that appears directly in the API call.  The pointer type
should not be a parameter passed in from a higher-level function.


c2: SQLite Archive Files
c3: 1. Introduction
c5: sqlar.html#introduction
c4:
An "SQLite Archive" is a file container similar to a 
ZIP archive or
Tarball but
based on an SQLite database.


An SQLite Archive is an ordinary SQLite database file that contains the
following table as part of its schema:

CREATE TABLE sqlar(
  name TEXT PRIMARY KEY,  -- name of the file
  mode INT,               -- access permissions
  mtime INT,              -- last modification time
  sz INT,                 -- original file size
  data BLOB               -- compressed content
);



Each row of the SQLAR table holds the content of a single file.
The filename (the full pathname relative to the root of the archive)
is in the "name" field.
The "mode" field is an integer which is the unix-style access permissions
for the file.  "mtime" is the modification time of the file in seconds
since 1970.  "sz" is the original uncompressed size of the file.
The "data" field contains the file content.  The content is usually
compressed using Deflate, though not always.  If the
"sz" field is equal to the size of the "data" field, then the content
is stored uncompressed.


c2: SQLite Archive Files
c3: 1.1. Database As Container Object
c5: sqlar.html#database_as_container_object
c4:
An SQLite Archive is one example of a more general
idea that an SQLite database can behave as a container object holding
lots of smaller data components.


With client/server databases like PostgreSQL or Oracle, users and
developers tend to think of the database as a service or a "node", not
as an object.  This is because the database content is spread out across
multiple files on the server, or possibly across multiple servers in a
service cluster.  One cannot point to a single file or even a single
directory and say "this is the database".


SQLite, in contrast, stores all content in a single file on disk.
That single file is something you can point to and say
"this is the database".  It behaves as an object.
An SQLite database file can be copied, renamed, sent as an
email attachment, passed as the argument a POST HTTP request,
or otherwise treated as other data object such as an image,
document, or media file.


Studies show that many applications already use
SQLite as a container object.  For example,
Kennedy
(no relation to the SQLite developer) reports that 14% of
Android applications never write to their SQLite databases.  It is
believed that these applications are downloading entire databases
from the cloud and then using the information locally as needed.  In other
words, the applications are using SQLite not so much as a database but as
a queryable wire-transfer format.


c2: SQLite Archive Files
c3: 1.2. Applications Using SQLite Archives
c5: sqlar.html#applications_using_sqlite_archives
c4:
The Fossil Distributed Version Control system
provides users with the option to download check-ins as either Tarballs,
ZIP Archives, or SQLite Archives.


c2: SQLite Archive Files
c3: 2. Advantages Of SQLite Archives
c5: sqlar.html#advantages_of_sqlite_archives
c4:
An SQLite Archive is flexible.
ZIP Archives and Tarballs are limited to storing only files.  An
SQLite Archive stores files plus whatever other tabular
and/or relational data seems useful to the application.


An SQLite Archive is transactional.
Updates are atomic and durable, even if there are crashes
or power losses in the middle of the update.
Readers see a consistent and unchanging version of the content even
is some other process is simultaneously updating the archive.


An SQLite Archive can be updated incrementally.
Individual files can be added or removed or replaced without having
to rewrite the entire archive.


An SQLite Archive can be queried using a high-level query language (SQL).
Some examples:

 What is the total size of all files in the archive whose names
     end in ".h" or ".cpp"?
 What percentage of the files are compressed by less than 25%?
 How many executable files are in the archive?

Questions like these (and countless others) can be answered without
having to uncompress or extract any content.


Applications that already use SQLite for other purposes can easily
add support for SQLite Archives using a small extension
(https://sqlite.org/src/file/ext/misc/sqlar.c) to handle the compression
and decompression of content.  Even this tiny extension can be omitted
if the files in the archive are uncompressed.  In contrast, supporting
ZIP Archives and/or Tarballs requires either separate libraries or
lots of extra custom code, or sometimes both.


An SQLite Archive can work around firewall-imposed censorship.
For example, certain file types that are considered "dangerous" 
(examples: DLLs) will be
blocked by Gmail
and probably many other email services and firewall, even if those
files are wrapped inside a ZIP Archive or Tarball.
But these firewall usually do not care about SQLite Archives and
so content can be put inside an SQLite Archive to evade censorship.


c2: SQLite Archive Files
c3: 3. Disadvantages Of SQLite Archives
c5: sqlar.html#disadvantages_of_sqlite_archives
c4:
The SQLite Archive is a relatively new format.  It was first described in
in 2014.  ZIP Archives and Tarballs, on the other hand, have been around
for decades and are well-entrenched as standard formats.  Most programmers
know what a ZIP Archive or Tarball is, but if you say "SQLite Archive" you
are more likely to get a reply of "What?"  Tooling to process ZIP Archives
and Tarballs is more likely to be installed on stock computers.


Since an SQLite database is a more general format (it is designed to do
much more than simply store a bunch of files) it is not as compact as either
the ZIP Archive or Tarball formats.  An SQLite Archive is usually about 1%
larger than the equivalent ZIP Archive.  Tarballs are compressed as a single
unit rather than compressing each file separately as is done by both
SQLite and ZIP Archives.  For these reason, Tarballs tend to be smaller
than either ZIP or SQLite Archives.

As an example, the following table show the relative sizes for an
SQLite Archive, a ZIP Archive, and a Tarball of the 1,743 files
in the SQLite 3.22.0 source tree:

  SQLite Archive10,754,048
  ZIP Archive (using Info-ZIP 3.0)10,662,365
  ZIP Archive (using zipfile)10,390,215
  Tarball&#x2007;9,781,109



An SQLite Archive supports only the Deflate compression
method.  Tarballs and ZIP Archive support a wider assortment of
compression methods.


c2: SQLite Archive Files
c3: 4. Managing An SQLite Archive From The Command-Line
c5: sqlar.html#managing_an_sqlite_archive_from_the_command_line
c4:
The recommended way of creating, updating, listing, and extracting
an SQLite Archive is to use the sqlite3.exe command-line shell 
for SQLite version 3.23.0 (2018-04-02) or later.  This CLI
supports the -A command-line option that allows easy management
of SQLite Archives.
The CLI for SQLite version 3.22.0 (2018-01-22) has the
.archive command for managing SQLite Archives, but that requires
interacting with the shell.


To list all of the files in an SQLite Archive named "example.sqlar"
using one of these commands:

sqlite3 example.sqlar -At
sqlite3 example.sqlar -Atv



To extract all files from an SQLite Archive named "example.sqlar":

sqlite3 example.sqlar -Ax



To create a new SQLite Archive named "alltxt.sqlar" containing all *.txt
files in the current directory:

sqlite3 alltxt.sqlar -Ac *.txt



To add or update files in an existing SQLite Archive:

sqlite3 example.sqlar -Au *.md



For usage hints and a summary of all options, simply give the CLI
the -A option with no additional arguments:

sqlite3 -A



All of these commands work the same way if the filename argument is
is a ZIP Archive instead of an SQLite database.


c2: SQLite Archive Files
c3: 4.1. Other command-line tools
c5: sqlar.html#other_command_line_tools
c4:
Just as there is the "zip" program to manage ZIP Archives, and the
"tar" program to manage Tarballs, the 
"sqlar" program exists to manage SQL Archives.
The "sqlar" program is able to create a new SQLite Archive, list the
content of an existing archive, add or remove files from the archive,
and/or extract files from the archive.
A separate "sqlarfs" program is able to mount the SQLite Archive as
a Fuse Filesystem.


c2: SQLite Archive Files
c3: 5. Managing SQLite Archives From Application Code
c5: sqlar.html#managing_sqlite_archives_from_application_code
c4:
Applications can easily read or write SQLite Archives by linking against
SQLite and including the 
ext/misc/sqlar.c extension
to handle the compression and decompression.  The sqlar.c extension
creates two new SQL functions.


sqlar_compress(X)

The sqlar_compress(X) function attempts to compress a copy of the
string or blob X using the Default algorithm and
returns the result as a blob.  If the input X is incompressible, then
a copy of X is returned.  This routine is used when inserting content
into an SQLite Archive.
sqlar_uncompress(Y,SZ)

The sqlar_uncompress(Y,SZ) function will undo the compression accomplished
by sqlar_compress(X).  The Y parameter is the compressed content (the output
from a prior call to sqlar_compress()) and SZ is the original uncompressed
size of the input X that generated Y.  If SZ is less than or equal to the
size of Y, that indicates that no compression occurred, and so
sqlar_uncompress(Y,SZ) returns a copy of Y.  Otherwise, sqlar_uncompress(Y,SZ)
runs the Inflate algorithm on Y to uncompress it and restore it to its
original form and returns the uncompressed content.
This routine is used when extracting content from an SQLite Archive.



Using the two routines above, it is simple for applications to insert
new records into or extract existing records from an SQLite Archive.
Insert a new into an SQLite Archive using code like this:

INSERT INTO sqlar(name,mode,mtime,sz,data)
 VALUES ($name,$mode,strftime('%s',$mtime),
         length($content),sqlar_compress($content));



Extract an entry from the SQLite Archive using code like this:

SELECT name, mode, datetime(mtime,'unixepoch'), sqlar_uncompress(data,sz)
  FROM sqlar
 WHERE ...;



The code above is for the general case.  For the special case of an
SQLite Archive that only stores uncompressed or uncompressible content
(this might come up, for example, in an SQLite Archive that stores only
JPEG, GIF, and/or PNG images) then the content can be inserted into
and extracted from the database without using the sqlar_compress()
and sqlar_uncompress() functions, and the sqlar.c extension is not
required.


c2: LoC Recommended Storage Format
c3: 1. Overview
c5: locrsf.html#overview
c4:
SQLite is a
Recommended Storage Format
for datasets according to the
US Library of Congress.
Further information:




https://www.loc.gov/preservation/digital/formats/fdd/fdd000461.shtml#local

https://www.loc.gov/preservation/resources/rfs/data.html



As of this writing (2018-05-29) the only other recommended storage formats
for datasets are XML, JSON, and CSV.


c2: LoC Recommended Storage Format
c3: 1.1. What Is A Recommended Storage Format?
c5: locrsf.html#what_is_a_recommended_storage_format_
c4:
Recommended storage formats are formats which, in the opinion of the
preservationists at the Library of Congress, maximizes the chance of
survival and continued accessibility of digital content.
When selecting recommended storage formats, the following criteria are
considered (quoting from the LOC website):


Disclosure.
Degree to which complete specifications and tools for validating 
technical integrity exist and are accessible to those creating and
sustaining digital content.
A spectrum of disclosure levels can be observed for digital formats.
What is most significant is not approval by a recognized standards body,
but the existence of complete documentation.
Adoption.
Degree to which the format is already used by the primary creators,
disseminators, or users of information resources. 
This includes use as a master format, for delivery to end users, 
and as a means of interchange between systems.
Transparency.
Degree to which the digital representation is open to direct analysis 
with basic tools, such as human readability using a text-only editor.
Self-documentation.
Self-documenting digital objects contain basic descriptive, 
technical, and other administrative metadata.
External Dependencies.
Degree to which a particular format depends on particular hardware,
operating system, or software for rendering or use and the predicted
complexity of dealing with those dependencies in future technical 
environments.
Impact of Patents.
Degree to which the ability of archival institutions to sustain content 
in a format will be inhibited by patents.
Technical Protection Mechanisms.
Implementation of mechanisms such as encryption that prevent the 
preservation of content by a trusted repository.


c2: Measuring and Reducing CPU Usage in SQLite
c3: 1. Overview
c5: cpu.html#overview
c4:
The graph below shows the number of CPU cycles used by SQLite on a
standard workload, for versions of SQLite going back about 10 years.
Recent versions of SQLite use about one third as many the CPU cycles
compared to older versions.


This article describes how the SQLite developers measure CPU usage,
what those measurements actually mean, and the techniques used by
SQLite developers on their continuing quest to further reduce the
CPU usage of the SQLite library.






Measured using cachegrind on Ubuntu 16.04 on x64 with gcc 5.4.0 and -Os.


c2: Measuring and Reducing CPU Usage in SQLite
c3: 2. Measuring Performance
c5: cpu.html#measuring_performance
c4:
In brief, the CPU performance of SQLite is measured as follows:


 Compile SQLite in an as-delivered configuration, without any special
     telemetry or debugging options.
 Link SQLite against a test program that runs approximately 30,000
     SQL statements representing a typical workload.
 Count the number of CPU cycles consumed using
     cachegrind.


c2: Measuring and Reducing CPU Usage in SQLite
c3: 2.1. Compile Options
c5: cpu.html#compile_options
c4:
For performance measurement, SQLite is compiled in approximately the same
way as it would be for use in production systems.  The compile-time configuration
is "approximate" in the sense that every production use of SQLite is 
different. Compile-time options used by one system are not necessarily
the same as those used by others.  The key point is that options that 
significantly impact the generated machine code are avoided.  For example,
the -DSQLITE_DEBUG option is omitted because that option inserts thousands
of assert() statements in the middle of performance critical sections of the
SQLite library.  The -pg option (on GCC) is omitted because it causes the
compiler to emit extra probabilistic performance measuring code which interferes
with actual performance measurements.


For performance measurements,
the -Os option is used (optimize for size) rather than -O2 because the
-O2 option creates so much code movement that it is difficult to associate
specific CPU instructions to C source code lines.


c2: Measuring and Reducing CPU Usage in SQLite
c3: 2.2. Workload
c5: cpu.html#workload
c4:
The "typical" workload is generated by the
speedtest1.c
program in the canonical SQLite source tree.  This program strives to
exercise the SQLite library in a way that is typical of real-world
applications.  Of course, every application is different, and so
no test program can exactly mirror the behavior of all applications.


The speedtest1.c program is updated from time to time as the SQLite
developers' understanding of what constitutes "typical" usage evolves.


The 
speed-check.sh shell
script, also in the canonical source tree, is used to run the speedtest1.c
program.  To replicate the performance measurements, collect the following
files into a single directory:

 the "speed-check.sh" script,
 the "speedtest1.c" test program, and
 the SQLite amalgamation source files "sqlite3.c" and
     "sqlite3.h"


Then run "sh speed-check.sh trunk".


c2: Measuring and Reducing CPU Usage in SQLite
c3: 2.3. Performance Measurement
c5: cpu.html#performance_measurement
c4:
Cachegrind is used to
measure performance because it gives answers that are repeatable to 
7 or more significant digits.  In comparison, actual (wall-clock)
run times are scarcely repeatable beyond one significant digit.


c2: Measuring and Reducing CPU Usage in SQLite
c3: 2.4. Microoptimizations
c5: cpu.html#microoptimizations
c4:
The high repeatability of cachegrind allows the SQLite developers to
implement and measure "microoptimizations".  A microoptimization is
a change to the code that results in a very small performance increase.
Typical micro-optimizations reduce the number of CPU cycles by 0.1% or
0.05% or even less.  Such improvements are impossible to measure with
real-world timings.  But hundreds or thousands of microoptimizations
add up, resulting in measurable real-world performance gains.


c2: Measuring and Reducing CPU Usage in SQLite
c3: 3. Performance Measurement Workflow
c5: cpu.html#performance_measurement_workflow
c4:
As SQLite developers edit the SQLite source code, they run the
speed-check.sh
shell script to track the performance impact of changes.  This
script compiles the speedtest1.c program, runs it under cachegrind,
processes the cachegrind output using the
cg_anno.tcl TCL
script, then saves the results in a series of text files.
Typical output from the speed-check.sh script looks like this:


==8683== 
==8683== I   refs:      1,060,925,768
==8683== I1  misses:       23,731,246
==8683== LLi misses:            5,176
==8683== I1  miss rate:          2.24%
==8683== LLi miss rate:          0.00%
==8683== 
==8683== D   refs:        557,686,925  (361,828,925 rd   + 195,858,000 wr)
==8683== D1  misses:        5,067,063  (  3,544,278 rd   +   1,522,785 wr)
==8683== LLd misses:           57,958  (     16,067 rd   +      41,891 wr)
==8683== D1  miss rate:           0.9% (        1.0%     +         0.8%  )
==8683== LLd miss rate:           0.0% (        0.0%     +         0.0%  )
==8683== 
==8683== LL refs:          28,798,309  ( 27,275,524 rd   +   1,522,785 wr)
==8683== LL misses:            63,134  (     21,243 rd   +      41,891 wr)
==8683== LL miss rate:            0.0% (        0.0%     +         0.0%  )
   text	   data	    bss	    dec	    hex	filename
 523044	   8240	   1976	 533260	  8230c	sqlite3.o
 220507 1007870 7769352 sqlite3.c


The important parts of the output (the parts that the developers pay
the most attention to) are shown in red.
Basically, the developers want to know the size of the compiled SQLite
library and how many CPU cycles were needed to run the performance test.

The output from the 
cg_anno.tcl script
shows the number of CPU cycles spent on each line of code.
The report is approximately 80,000 lines long.  The following is a brief
snippet taken from the middle of the report to show what it looks like:


         .  SQLITE_PRIVATE int sqlite3BtreeNext(BtCursor *pCur, int *pRes){
         .    MemPage *pPage;
         .    assert( cursorOwnsBtShared(pCur) );
         .    assert( pRes!=0 );
         .    assert( *pRes==0 || *pRes==1 );
         .    assert( pCur-&gt;skipNext==0 || pCur-&gt;eState!=CURSOR_VALID );
   369,648    pCur-&gt;info.nSize = 0;
   369,648    pCur-&gt;curFlags &amp;= ~(BTCF_ValidNKey|BTCF_ValidOvfl);
   369,648    *pRes = 0;
   739,296    if( pCur-&gt;eState!=CURSOR_VALID ) return btreeNext(pCur, pRes);
 1,473,580    pPage = pCur-&gt;apPage&#91;pCur-&gt;iPage&#93;;
 1,841,975    if( (++pCur-&gt;aiIdx&#91;pCur-&gt;iPage&#93;)&gt;=pPage-&gt;nCell ){
     4,340      pCur-&gt;aiIdx&#91;pCur-&gt;iPage&#93;--;
     5,593      return btreeNext(pCur, pRes);
         .    }
   728,110    if( pPage-&gt;leaf ){
         .      return SQLITE_OK;
         .    }else{
     3,117      return moveToLeftmost(pCur);
         .    }
   721,876  }



The numbers on the left are the CPU cycle counts for that line of code,
of course.


The cg_anno.tcl script removes extraneous details from the default 
cachegrind annotation
output so that before-and-after reports can be compared using a 
side-by-side diff to view specific details of how a
micro-optimization attempt affected performance.


c2: Measuring and Reducing CPU Usage in SQLite
c3: 4. Limitations
c5: cpu.html#limitations
c4:
The use of the standardized speedtest1.c workload and cachegrind has
enabled significant performance improvement.
However, it is important to recognize the limitations of this approach:



Performance measurements are done with a single compiler (gcc 5.4.0),
optimization setting (-Os), and
on a single platform (Ubuntu 16.04 LTS on x64).  The performance of
other compilers and processors may vary.


The speedtest1.c workload that is being measured tries to be representative
of a wide range of typical uses of SQLite.  But every application is
different.  The speedtest1.c workload might not be a good proxy for the
kinds of activities performed by some applications.  The SQLite developers
are constantly working to improve the speedtest1.c program, to make it
a better proxy for actual SQLite usage.  Community feedback is welcomed.


The cycle counts provided by cachegrind are a good proxy for actual
performance, but they are not 100% accurate.


Only CPU cycle counts are being measured here. 
CPU cycle counts are a good proxy for energy consumption,
but do not necessary correlate well with real-world timings.
Time spent doing I/O is not reflected in the CPU cycle counts,
and I/O time predominates in many SQLite usage scenarios.


c2: Query Planning
c3: Overview
c5: queryplanner.html#overview
c4:
The best feature of SQL (in all its implementations, not just SQLite)
is that it is a declarative language, not a procedural
language.  When programming in SQL you tell the system what you
want to compute, not how to compute it.  The task of figuring out
the how is delegated to the query planner subsystem within 
the SQL database engine.

For any given SQL statement, there might be hundreds or thousands or
even millions of different algorithms of performing the operation.  All
of these algorithms will get the correct answer, though some will run
faster than others.
The query planner is an 
AI that 
tries to pick the fastest and most efficient algorithm for each SQL
statement.



Most of the time, the query planner in SQLite does a good job.
However, the query planner needs indices to
work with.  
These indices must normally be added by programmers.
Rarely, the query planner AI will make a suboptimal algorithm
choice.
In those cases, programmers may want to provide additional
hints to help the query planner do a better job.



This document provides background information about how the
SQLite query planner and query engine work.
Programmers can use this information to help create better
indexes, and provide hints to help the query planner when
needed.



Additional information is provided in the
SQLite query planner and 
next generation query planner documents.


c2: Query Planning
c3: 1.1. Tables Without Indices
c5: queryplanner.html#_tables_without_indices
c4:
Most tables in SQLite consist of zero or more rows with a unique integer
key (the rowid or INTEGER PRIMARY KEY) followed by content.  
(The exception is WITHOUT ROWID tables.)
The rows
are logically stored in order of increasing rowid.  As an example, this
article uses a table named "FruitsForSale" which relates various fruits 
to the state
where they are grown and their unit price at market.  The schema is this:



CREATE TABLE FruitsForSale(
  Fruit TEXT,
  State TEXT,
  Price REAL
);




With some (arbitrary) data, such a table might be logically stored on disk
as shown in figure 1:





Figure 1: Logical Layout Of Table "FruitsForSale"




In this example, the rowids are not
consecutive but they are ordered.  SQLite usually creates rowids beginning
with one and increasing by one with each added row.  But if rows are 
deleted, gaps can appear in the sequence.  And the application can control
the rowid assigned if desired, so that rows are not necessarily inserted 
at the bottom.  But regardless of what happens, the rowids are always 
unique and in strictly ascending order.



Suppose you want to look up the price of peaches.  The query would
be as follows:



SELECT price FROM fruitsforsale WHERE fruit='Peach';




To satisfy this query, SQLite reads every row out of the
table, checks to see if the "fruit" column has the value of "Peach" and if
so, outputs the "price" column from that row.  The process is illustrated
by figure 2 below.
This is algorithm is called a full table scan 
since the entire content of the
table must be read and examined in order to find the one row of interest.
With a table of only 7 rows, a full table scan is acceptable, 
but if the table contained 7 million rows, a full table scan might read 
megabytes of content in order to find a single 8-byte number.  
For that reason, one normally tries to avoid full table scans.





Figure 2: Full Table Scan


c2: Query Planning
c3: 1.2. Lookup By Rowid
c5: queryplanner.html#_lookup_by_rowid
c4:
One technique for avoiding a full table scan is to do lookups by
rowid (or by the equivalent INTEGER PRIMARY KEY).   To lookup the
price of peaches, one would query for the entry with a rowid of 4:



SELECT price FROM fruitsforsale WHERE rowid=4;




Since the information is stored in the table in rowid order, SQLite
can find the correct row using a binary search.
If the table contains N elements, the time required to look up the
desired row is proportional to logN rather than being proportional
to N as in a full table scan.  If the table contains 10 million elements,
that means the query will be on the order of N/logN or about 1 million
times faster.





Figure 3: Lookup By Rowid


c2: Query Planning
c3: 1.3. Lookup By Index
c5: queryplanner.html#_lookup_by_index
c4:
The problem with looking up information by rowid is that you probably
do not care what the price of "item 4" is - you want to know the price
of peaches.  And so a rowid lookup is not helpful.



To make the original query more efficient, we can add an index on the
"fruit" column of the "fruitsforsale" table like this:



CREATE INDEX Idx1 ON fruitsforsale(fruit);




An index is another table similar to the original "fruitsforsale" table
but with the content (the fruit column in this case) stored in front of the
rowid and with all rows in content order.
Figure 4 gives a logical view of the Idx1 index.
The "fruit" column is the primary key used to order the elements of the
table and the "rowid" is the secondary key used to break the tie when
two or more rows have the same "fruit".  In the example, the rowid
has to be used as a tie-breaker for the "Orange" rows.
Notice that since the rowid
is always unique over all elements of the original table, the composite key
of "fruit" followed by "rowid" will be unique over all elements of the index.





Figure 4: An Index On The Fruit Column




This new index can be used to implement a faster algorithm for the
original "Price of Peaches" query.



SELECT price FROM fruitsforsale WHERE fruit='Peach';




The query starts by doing a binary search on the Idx1 index for entries
that have fruit='Peach'.  SQLite can do this binary search on the Idx1 index
but not on the original FruitsForSale table because the rows in Idx1 are sorted
by the "fruit" column.  Having found a row in the Idx1 index that has
fruit='Peach', the database engine can extract the rowid for that row.
Then the database engines does a second binary search
on the original FruitsForSale table to find the
original row that contains fruit='Peach'.  
From the row in the FruitsForSale table,
SQLite can then extract the value of the price column.
This procedure is illustrated by figure 5.





Figure 5: Indexed Lookup For The Price Of Peaches




SQLite has to do two binary searches to find the price of peaches using
the method show above.  But for a table with a large number of rows, this
is still much faster than doing a full table scan.


c2: Query Planning
c3: 1.4. Multiple Result Rows
c5: queryplanner.html#_multiple_result_rows
c4:
In the previous query the fruit='Peach' constraint narrowed the result
down to a single row.  But the same technique works even if multiple
rows are obtained.  Suppose we looked up the price of Oranges instead of
Peaches:



SELECT price FROM fruitsforsale WHERE fruit='Orange'




Figure 6: Indexed Lookup For The Price Of Oranges




In this case, SQLite still does a single binary search to find the first
entry of the index where fruit='Orange'.  Then it extracts the rowid from
the index and uses that rowid to lookup the original table entry via
binary search and output the price from the original table.  But instead
of quitting, the database engine then advances to the next row of index
to repeat the process for next fruit='Orange' entry.  Advancing to the
next row of an index (or table) is much less costly than doing a binary
search since the next row is often located on the same database page as
the current row.  In fact, the cost of advancing to the next row is so
cheap in comparison to a binary search that we usually ignore it.  So
our estimate for the total cost of this query is 3 binary searches.
If the number of rows of output is K and the number of rows in the table
is N, then in general the cost of doing the query is proportional
to (K+1)*logN.


c2: Query Planning
c3: 1.5. Multiple AND-Connected WHERE-Clause Terms
c5: queryplanner.html#_multiple_and_connected_where_clause_terms
c4:
Next, suppose that you want to look up the price of not just any orange,
but specifically California-grown oranges.  The appropriate query would
be as follows:



SELECT price FROM fruitsforsale WHERE fruit='Orange' AND state='CA'




Figure 7: Indexed Lookup Of California Oranges




One approach to this query is to use the fruit='Orange' term of the WHERE
clause to find all rows dealing with oranges, then filter those rows
by rejecting any that are from states other than California.  This
process is shown by figure 7 above.  This is a perfectly
reasonable approach in most cases.  Yes, the database engine did have
to do an extra binary search for the Florida orange row that was
later rejected, so it was not as efficient as we might hope, though
for many applications it is efficient enough.  



Suppose that in addition to the index on "fruit" there was also
an index on "state".



CREATE INDEX Idx2 ON fruitsforsale(state);




Figure 8: Index On The State Column




The "state" index works just like the "fruit" index in that it is a
new table with an extra column in front of the rowid and sorted by
that extra column as the primary key.  The only difference is that
in Idx2, the first column is "state" instead of "fruit" as it is with
Idx1.  In our example data set, there is more redundancy in the "state"
column and so they are more duplicate entries.  The ties are still
resolved using the rowid.



Using the new Idx2 index on "state", SQLite has another option for
lookup up the price of California oranges:  it can look up every row
that contains fruit from California and filter out those rows that
are not oranges.





Figure 9: Indexed Lookup Of California Oranges




Using Idx2 instead of Idx1 causes SQLite to examine a different set of
rows, but it gets the same answer in the end (which is very important -
remember that indices should never change the answer, only help SQLite to
get to the answer more quickly) and it does the same amount of work.
So the Idx2 index did not help performance in this case.



The last two queries take the same amount of time, in our example.
So which index, Idx1 or Idx2, will SQLite choose?  If the
ANALYZE command has been run on the database, so that SQLite has
had an opportunity to gather statistics about the available indices,
then SQLite will know that the Idx1 index usually narrows the search
down to a single item (our example of fruit='Orange' is the exception
to this rule) whereas the Idx2 index will normally only narrow the 
search down to two rows.  So, if all else is equal, SQLite will
choose Idx1 with the hope of narrowing the search to as small
a number of rows as possible.  This choice is only possible because
of the statistics provided by ANALYZE.  If ANALYZE has not been
run then the choice of which index to use is arbitrary.


c2: Query Planning
c3: 1.6. Multi-Column Indices
c5: queryplanner.html#_multi_column_indices
c4:
To get the maximum performance out of a query with multiple AND-connected
terms in the WHERE clause, you really want a multi-column index with
columns for each of the AND terms.  In this case we create a new index
on the "fruit" and "state" columns of FruitsForSale:



CREATE INDEX Idx3 ON FruitsForSale(fruit, state);




Figure 1: A Two-Column Index




A multi-column index follows the same pattern as a single-column index;
the indexed columns are added in front of the rowid.  The only difference
is that now multiple columns are added.  The left-most column is the
primary key used for ordering the rows in the index.  The second column is
used to break ties in the left-most column.  If there were a third column,
it would be used to break ties for the first two columns.  And so forth for
all columns in the index.  Because rowid is guaranteed
to be unique, every row of the index will be unique even if all of the
content columns for two rows are the same.  That case does not happen
in our sample data, but there is one case (fruit='Orange') where there
is a tie on the first column which must be broken by the second column.



Given the new multi-column Idx3 index, it is now possible for SQLite
to find the price of California oranges using only 2 binary searches:



SELECT price FROM fruitsforsale WHERE fruit='Orange' AND state='CA'




Figure 11: Lookup Using A Two-Column Index




With the Idx3 index on both columns that are constrained by the WHERE clause,
SQLite can do a single binary search against Idx3 to find the one rowid
for California oranges, then do a single binary search to find the price
for that item in the original table.  There are no dead-ends and no
wasted binary searches.  This is a more efficient query.



Note that Idx3 contains all the same information as the original 
Idx1.  And so if we have Idx3, we do not really need Idx1
any more.  The "price of peaches" query can be satisfied using Idx3
by simply ignoring the "state" column of Idx3:



SELECT price FROM fruitsforsale WHERE fruit='Peach'




Figure 12: Single-Column Lookup On A Multi-Column Index




Hence, a good rule of thumb is that your database schema should never
contain two indices where one index is a prefix of the other.  Drop the
index with fewer columns.  SQLite will still be able to do efficient
lookups with the longer index.


c2: Query Planning
c3: 1.7. Covering Indices
c5: queryplanner.html#_covering_indices
c4:
The "price of California oranges" query was made more efficient through
the use of a two-column index.  But SQLite can do even better with a
three-column index that also includes the "price" column:



CREATE INDEX Idx4 ON FruitsForSale(fruit, state, price);




Figure 13: A Covering Index




This new index contains all the columns of the original FruitsForSale table that
are used by the query - both the search terms and the output.  We call
this a "covering index".  Because all of the information needed is in
the covering index, SQLite never needs to consult the original table
in order to find the price.



SELECT price FROM fruitsforsale WHERE fruit='Orange' AND state='CA';




Figure 14: Query Using A Covering Index




Hence, by adding extra "output" columns onto the end of an index, one
can avoid having to reference the original table and thereby
cut the number of binary searches for a query in half.  This is a
constant-factor improvement in performance (roughly a doubling of
the speed).  But on the other hand, it is also just a refinement;
A two-fold performance increase is not nearly as dramatic as the
one-million-fold increase seen when the table was first indexed.
And for most queries, the difference between 1 microsecond and
2 microseconds is unlikely to be noticed.


c2: Query Planning
c3: 1.8. OR-Connected Terms In The WHERE Clause
c5: queryplanner.html#_or_connected_terms_in_the_where_clause
c4:
Multi-column indices only work if the constraint terms in the WHERE
clause of the query are connected by AND.
So Idx3 and Idx4 are helpful when the search is for items that
are both Oranges and grown in California, but neither index would
be that useful if we wanted all items that were either oranges
or are grown in California.



SELECT price FROM FruitsForSale WHERE fruit='Orange' OR state='CA';




When confronted with OR-connected terms in a WHERE clause, SQLite 
examines each OR term separately and tries to use an index to
find the rowids associated with each term.
It then takes the union of the resulting rowid sets to find
the end result.  The following figure illustrates this process:





Figure 15: Query With OR Constraints




The diagram above implies that SQLite computes all of the rowids first
and then combines them with a union operation before starting to do
rowid lookups on the original table.  In reality, the rowid lookups
are interspersed with rowid computations.  SQLite uses one index at
a time to find rowids while remembering which rowids it has seen
before so as to avoid duplicates.  That is just an implementation
detail, though.  The diagram, while not 100% accurate, provides a good
overview of what is happening.



In order for the OR-by-UNION technique shown above to be useful, there
must be an index available that helps resolve every OR-connected term
in the WHERE clause.  If even a single OR-connected term is not indexed,
then a full table scan would have to be done in order to find the rowids
generated by the one term, and if SQLite has to do a full table scan, it
might as well do it on the original table and get all of the results in
a single pass without having to mess with union operations and follow-on
binary searches.



One can see how the OR-by-UNION technique could also be leveraged to
use multiple indices on queries where the WHERE clause has terms connected
by AND, by using an intersect operator in place of union.  Many SQL
database engines will do just that.  But the performance gain over using
just a single index is slight and so SQLite does not implement that technique
at this time.  However, a future version SQLite might be enhanced to support
AND-by-INTERSECT.


c2: Query Planning
c3: 2. Sorting
c5: queryplanner.html#_sorting
c4:
SQLite (like all other SQL database engines) can also use indices to
satisfy the ORDER BY clauses in a query, in addition to expediting
lookup.  In other words, indices can be used to speed up sorting as
well as searching.



When no appropriate indices are available, a query with an ORDER BY
clause must be sorted as a separate step.  Consider this query:



SELECT * FROM fruitsforsale ORDER BY fruit;




SQLite processes this by gathering all the output of query and then
running that output through a sorter.





Figure 16: Sorting Without An Index




If the number of output rows is K, then the time needed to sort is
proportional to KlogK.  If K is small, the sorting time is usually
not a factor, but in a query such as the above where K==N, the time
needed to sort can be much greater than the time needed to do a
full table scan.  Furthermore, the entire output is accumulated in
temporary storage (which might be either in main memory or on disk,
depending on various compile-time and run-time settings)
which can mean that a lot of temporary storage is required to complete
the query.


c2: Query Planning
c3: 2.1. Sorting By Rowid
c5: queryplanner.html#_sorting_by_rowid
c4:
Because sorting can be expensive, SQLite works hard to convert ORDER BY
clauses into no-ops.  If SQLite determines that output will
naturally appear in the order specified, then no sorting is done.
So, for example, if you request the output in rowid order, no sorting
will be done:



SELECT * FROM fruitsforsale ORDER BY rowid;




Figure 17: Sorting By Rowid




You can also request a reverse-order sort like this:



SELECT * FROM fruitsforsale ORDER BY rowid DESC;




SQLite will still omit the sorting step.  But in order for output to
appear in the correct order, SQLite will do the table scan starting at
the end and working toward the beginning, rather than starting at the
beginning and working toward the end as shown in 
figure 17.


c2: Query Planning
c3: 2.2. Sorting By Index
c5: queryplanner.html#_sorting_by_index
c4:
Of course, ordering the output of a query by rowid is seldom useful.
Usually one wants to order the output by some other column.



If an index is available on the ORDER BY column, that index can be used
for sorting.  Consider the request for all items sorted by "fruit":



SELECT * FROM fruitsforsale ORDER BY fruit;






Figure 18: Sorting With An Index




The Idx1 index is scanned from top to bottom (or from bottom to top if
"ORDER BY fruit DESC" is used) in order to find the rowids for each item
in order by fruit.  Then for each rowid, a binary search is done to lookup
and output that row.  In this way, the output appears in the requested order
without the need to gather the entire output and sort it using a separate step.



But does this really save time?  The number of steps in the 
original indexless sort is proportional to NlogN since
that is how much time it takes to sort N rows.  But when we use Idx1 as
shown here, we have to do N rowid lookups which take logN time each, so
the total time of NlogN is the same!



SQLite uses a cost-based query planner.  When there are two or more ways
of solving the same query, SQLite tries to estimate the total amount of
time needed to run the query using each plan, and then uses the plan with
the lowest estimated cost.  A cost is computed mostly from the estimated
time, and so this case could go either way depending on the table size and
what WHERE clause constraints were available, and so forth.  But generally
speaking, the indexed sort would probably be chosen, if for no other
reason, because it does not need to accumulate the entire result set in
temporary storage before sorting and thus uses much less temporary storage.


c2: Query Planning
c3: 2.3. Sorting By Covering Index
c5: queryplanner.html#_sorting_by_covering_index
c4:
If a covering index can be used for a query, then the multiple rowid lookups
can be avoided and the cost of the query drops dramatically.





Figure 19: Sorting With A Covering Index




With a covering index, SQLite can simply walk the index from one end to the
other and deliver the output in time proportional to N and without having
allocate a large buffer to hold the result set.


c2: Query Planning
c3: 3. Searching And Sorting At The Same Time
c5: queryplanner.html#_searching_and_sorting_at_the_same_time
c4:
The previous discussion has treated searching and sorting as separate
topics.  But in practice, it is often the case that one wants to search
and sort at the same time.  Fortunately, it is possible to do this
using a single index.


c2: Query Planning
c3: 3.1. Searching And Sorting With A Multi-Column Index
c5: queryplanner.html#_searching_and_sorting_with_a_multi_column_index
c4:
Suppose we want to find the prices of all kinds of oranges sorted in
order of the state where they are grown.  The query is this:



SELECT price FROM fruitforsale WHERE fruit='Orange' ORDER BY state




The query contains both a search restriction in the WHERE clause
and a sort order in the ORDER BY clause.  Both the search and the sort
can be accomplished at the same time using the two-column index Idx3.





Figure 20: Search And Sort By Multi-Column Index




The query does a binary search on the index to find the subset of rows
that have fruit='Orange'.  (Because the fruit column is the left-most column
of the index and the rows of the index are in sorted order, all such 
rows will be adjacent.)  Then it scans the matching index rows from top to
bottom to get the rowids for the original table, and for each rowid does
a binary search on the original table to find the price.



You will notice that there is no "sort" box anywhere in the above diagram.
The ORDER BY clause of the query has become a no-op.  No sorting has to be
done here because the output order is by the state column and the state
column also happens to be the first column after the fruit column in the
index.  So, if we scan entries of the index that have the same value for
the fruit column from top to bottom, those index entries are guaranteed to
be ordered by the state column.


c2: Query Planning
c3: 3.2. Searching And Sorting With A Covering Index
c5: queryplanner.html#_searching_and_sorting_with_a_covering_index
c4:
A covering index can also be used to search and sort at the same time.
Consider the following:



SELECT * FROM fruitforsale WHERE fruit='Orange' ORDER BY state




Figure 21: Search And Sort By Covering Index




As before, SQLite does single binary search
for the range of rows in the covering
index that satisfy the WHERE clause, the scans that range from top to 
bottom to get the desired results.  
The rows that satisfy the WHERE clause are guaranteed to be adjacent
since the WHERE clause is an equality constraint on the left-most
column of the index.  And by scanning the matching index rows from
top to bottom, the output is guaranteed to be ordered by state since the
state column is the very next column to the right of the fruit column.
And so the resulting query is very efficient.



SQLite can pull a similar trick for a descending ORDER BY:



SELECT * FROM fruitforsale WHERE fruit='Orange' ORDER BY state DESC




The same basic algorithm is followed, except this time the matching rows
of the index are scanned from bottom to top instead of from top to bottom,
so that the states will appear in descending order.


c2: Query Planning
c3: 3.3. Partial Sorting Using An Index (a.k.a. Block Sorting)
c5: queryplanner.html#_partial_sorting_using_an_index_a_k_a_block_sorting_
c4:
Sometimes only part of an ORDER BY clause can be satisfied using indexes.
Consider, for example, the following query:



SELECT * FROM fruitforsale ORDER BY fruit, price




If the covering index is used for the scan, the "fruit" column will appear
naturally in the correct order, but when there are two or more rows with
the same fruit, the price might be out of order.  When this occurs, SQLite
does many small sorts, one sort for each distinct value of fruit, rather
than one large sort.  Figure 22 below illustrates the concept.





Figure 22: Partial Sort By Index




In the example, instead of a single sort of 7 elements, there
are 5 sorts of one-element each and 1 sort of 2 elements for the
case of fruit=='Orange'.


The advantages of doing many smaller sorts instead of a single large sort
are:

Multiple small sorts collectively use fewer CPU cycles than a single
    large sort.
Each small sort is run independently, meaning that much less information
    needs to be kept in temporary storage at any one time.
Those columns of the ORDER BY that are already in the correct order
    due to indexes can be omitted from the sort key, further reducing
    storage requirements and CPU time.
Output rows can be returned to the application as each small sort
    completes, and well before the table scan is complete.
If a LIMIT clause is present, it might be possible to avoid scanning
    the entire table.


Because of these advantages, SQLite always tries to do a partial sort using an
index even if a complete sort by index is not possible.


c2: Query Planning
c3: 4. WITHOUT ROWID tables
c5: queryplanner.html#_without_rowid_tables
c4:
The basic principals described above apply to both ordinary rowid tables
and WITHOUT ROWID tables.
The only difference is that the rowid column that serves as the key for
tables and that appears as the right-most term in indexes is replaced by
the PRIMARY KEY.


c2: Defense Against Dark Arts
c3: 1. SQLite Always Validates Its Inputs
c5: security.html#sqlite_always_validates_its_inputs
c4:
SQLite should never crash, overflow a buffer, leak memory,
or exhibit any other harmful behavior, even when presented with
maliciously malformed SQL inputs or database files.  SQLite should
always detect erroneous inputs and raise an error, not crash or
corrupt memory.
Any malfunction caused by an SQL input or database file
is considered a serious bug and will be promptly addressed when
brought to the attention of the SQLite developers.  SQLite is
extensively fuzz-tested to help ensure that it is resistant
to these kinds of errors.


Nevertheless, bugs happen.
If you are writing an application that sends untrusted SQL inputs
or database files to SQLite, there are additional steps you can take
to help reduce the attack surface and
prevent zero-day exploits caused by undetected bugs.


c2: Defense Against Dark Arts
c3: 1.1. Untrusted SQL Inputs
c5: security.html#untrusted_sql_inputs
c4:
Applications that accept untrusted SQL inputs should take the following
precautions:



Set the SQLITE_DBCONFIG_DEFENSIVE flag.
This prevents ordinary SQL statements from deliberately corrupting the 
database file.  SQLite should be proof against attacks that involve both
malicious SQL inputs and a maliciously corrupted database file at the
same time.  Nevertheless, denying a script-only attacker access to 
corrupt database inputs provides an extra layer of defense.


Reduce the limits that SQLite imposes on inputs.  This can help prevent
denial of service attacks and other kinds of mischief that can occur
as a result of unusually large inputs.  You can do this either at compile-time
using -DSQLITE_MAX_... options, or at run-time using the
sqlite3_limit() interface.  Most applications can reduce limits
dramatically without impacting functionality.  The table below
provides some suggestions, though exact values will vary depending
on the application:


Limit SettingDefault ValueHigh-security Value
LIMIT_LENGTH1,000,000,0001,000,000
LIMIT_SQL_LENGTH1,000,000,000100,000
LIMIT_COLUMN2,000100
LIMIT_EXPR_DEPTH1,00010
LIMIT_COMPOUND_SELECT5003
LIMIT_VDBE_OP250,000,00025,000
LIMIT_FUNCTION_ARG1278
LIMIT_ATTACH100
LIMIT_LIKE_PATTERN_LENGTH50,00050
LIMIT_VARIABLE_NUMBER99910
LIMIT_TRIGGER_DEPTH1,00010



Consider using the sqlite3_set_authorizer() interface to limit
the scope of SQL that will be processed.  For example, an application
that does not need to change the database schema might add an
sqlite3_set_authorizer() callback that causes any CREATE or DROP
statement to fail.


The SQL language is very powerful, and so it is always possible for
malicious SQL inputs (or erroneous SQL inputs caused by an application
bug) to submit SQL that runs for a very long time.  To prevent this
from becoming a denial-of-service attack, consider using the
sqlite3_progress_handler() interface to invoke a callback periodically
as each SQL statement runs, and have that callback return non-zero to
abort the statement if the statement runs for too long.  Alternatively,
set a timer in a separate thread and invoke sqlite3_interrupt() when
the timer goes off to prevent the SQL statement from running forever.


Limit the maximum amount of memory that SQLite will allocate using
the sqlite3_hard_heap_limit64() interface.  This helps prevent
denial-of-service attacks.  To find out how much heap space an
application actually needs, run the it against typical inputs and
then measure the maximum instantaneous memory usage with the 
sqlite3_memory_highwater() interface.  Set the hard heap limit
to the maximum observed instantaneous memory usage plus some margin.


For embedded systems, consider compiling SQLite with the
-DSQLITE_ENABLE_MEMSYS5 option and then providing SQLite with
a fixed chunk of memory to use as its heap via the
sqlite3_config(SQLITE_CONFIG_HEAP) interface. This will
prevent malicious SQL from executing a denial-of-service attack
by using an excessive amount of memory. If (say) 5 MB of memory
is provided for SQLite to use, once that much has been consumed,
SQLite will start returning SQLITE_NOMEM errors rather than
soaking up memory needed by other parts of the application.
This also sandboxes SQLite's memory so that a write-after-free
error in some other part of the application will not cause
problems for SQLite, or vice versa.



To control memory usage in the printf() SQL function, compile
with "-DSQLITE_PRINTF_PRECISION_LIMIT=1000" or some similar small value.
This #define limits the width and precision for %-substitutions in the
printf() function, and thus prevents a hostile SQL statement from
consuming large amounts of RAM via constructs such as
"printf('%1000000000s','hi')".


c2: Defense Against Dark Arts
c3: 1.2. Untrusted SQLite Database Files
c5: security.html#untrusted_sqlite_database_files
c4:
Applications that read or write SQLite database files of uncertain
provenance should take precautions enumerated below.

Even if the application does not deliberately accept database files 
from untrusted sources, beware of attacks in which a local 
database file is altered.  For best security, any database file which 
might have ever been writable by an agent in a different security domain
should be treated as suspect.



If the application includes any custom SQL functions or 
custom virtual tables that have side effects or that might leak
privileged information, then the application should use one or more
of the techniques below to prevent a maliciously crafted database
schema from surreptitiously running those SQL functions and/or
virtual tables for nefarious purposes:

 Invoke sqlite3_db_config(db,SQLITE_DBCONFIG_TRUSTED_SCHEMA,0,0)
     on each database connection as soon as it is opened.
 Run the PRAGMA trusted_schema=OFF statement on each database connection
     as soon as it is opened.
 Compile SQLite using the -DSQLITE_TRUSTED_SCHEMA=0 compile-time option.
 Disable the surreptitious use of custom SQL functions and virtual tables
     by setting the SQLITE_DIRECTONLY flag on all custom SQL functions and
     the SQLITE_VTAB_DIRECTONLY flag on all custom virtual tables.



If the application does not use triggers or views, consider disabling the
unused capabilities with:

sqlite3_db_config(db,SQLITE_DBCONFIG_ENABLE_TRIGGER,0,0);
sqlite3_db_config(db,SQLITE_DBCONFIG_ENABLE_VIEW,0,0);





For reading database files that are unusually high-risk, such as database
files that are received from remote machines, and possibly from anonymous
contributors, the following extra precautions
might be justified.  These added defenses come with performance costs,
however, and so may not be appropriate in every situation:



Run PRAGMA integrity_check or PRAGMA quick_check on the database
as the first SQL statement after opening the database files and
prior to running any other SQL statements.  Reject and refuse to
process any database file containing errors.


Enable the PRAGMA cell_size_check=ON setting.



Do not enable memory-mapped I/O.
In other words, make sure that PRAGMA mmap_size=0.


c2: Defense Against Dark Arts
c3: 2. Summary
c5: security.html#summary
c4:
The precautions above are not required in order to use SQLite safely
with potentially hostile inputs.
However, they do provide an extra layer of defense against zero-day
exploits and are encouraged for applications that pass data from
untrusted sources into SQLite.


c2: The Lemon LALR(1) Parser Generator
c3: 1. Overview
c5: lemon.html#overview
c4:
The SQL language parser for SQLite is generated using a code-generator
program called "Lemon".  The Lemon program reads a grammar of the input
language and emits C-code to implement a parser for that language.


c2: The Lemon LALR(1) Parser Generator
c3: 1.1. Lemon Source Files And Documentation
c5: lemon.html#lemon_source_files_and_documentation
c4:
Lemon does not have its own source repository.  Rather, Lemon consists
of a few files in the SQLite source tree:



     lemon.html &rarr;
     The original detailed usage documentation and programmers reference
     for Lemon.

     lemon.c &rarr; The source code
     for the utility program that reads a grammar file and generates 
     corresponding parser C-code.

     lempar.c &rarr; A template
     for the generated parser C-code.  The "lemon" utility program reads this
     template and inserts additional code in order to generate a parser.


c2: The Lemon LALR(1) Parser Generator
c3: 2. Advantages of Lemon
c5: lemon.html#advantages_of_lemon
c4:
Lemon generates an LALR(1) parser.  Its operation is similar to the
more familiar tools Yacc and
Bison, but Lemon adds important
improvements, including:



     The grammar syntax is less error prone - using symbolic names for
     semantic values rather that the "$1"-style positional notation
     of Yacc.

     In Lemon, the tokenizer calls the parser.  Yacc operates the other
     way around, with the parser calling the tokenizer.  The Lemon
     approach is reentrant and threadsafe, whereas Yacc uses global 
     variables and is therefore neither.  Reentrancy is especially
     important for SQLite since some SQL statements make recursive calls
     to the parser.  For example, when parsing a CREATE TABLE statement,
     SQLite invokes the parser recursively to generate an INSERT statement
     to make a new entry in the sqlite_schema table.

     Lemon has the concept of a non-terminal destructor that can be
     used to reclaim memory or other resources following a syntax error
     or other aborted parse.


c2: The Lemon LALR(1) Parser Generator
c3: 2.1. Use of Lemon Within SQLite
c5: lemon.html#use_of_lemon_within_sqlite
c4:
Lemon is used in two places in SQLite.

The primary use of Lemon is to create the SQL language parser.
A grammar file (parse.y) is
compiled by Lemon into parse.c and parse.h.  The parse.c file is
incorporated into the amalgamation without further modification.

Lemon is also used to generate the parser for the query pattern
expressions in the FTS5 extension.  In this case, the input grammar
file is fts5parse.y.


c2: The Lemon LALR(1) Parser Generator
c3: 2.2. Lemon Customizations Especially For SQLite
c5: lemon.html#lemon_customizations_especially_for_sqlite
c4:
One of the advantages of hosting code generator tools as part of
the project is that the tools can be optimized to serve specific needs of
the overall project.  Lemon has benefited from this effect. Over the years,
the Lemon parser generator has been extended and enhanced to provide
new capabilities and improved performance to SQLite.  A few of the
specific enhancements to Lemon that are specifically designed for use
by SQLite include:



Lemon has the concept of a "fallback" token.
The SQL language contains a large number of keywords and these keywords
have the potential to collide with identifier names.
Lemon has the ability to designate some keywords has being able to
"fallback" to an identifier.  If the keyword appears in the input token
stream in a context that would otherwise be a syntax error, the token
is automatically transformed into its fallback before the syntax error
is raised.  This feature allows the parser to be very forgiving of
reserved words used as identifiers, which is a problem that comes up
frequently in the SQL language.


In support of the 100% MC/DC testing goal for SQLite, 
the parser code generated by Lemon has no unreachable branches,
and contains extra (compile-time selected) instrumentation useful
for measuring test coverage.


Lemon supports conditional compilation of grammar file rules, so that
a different parser can be generated depending on compile-time options.


As a performance optimization, reduce actions in the Lemon input grammar
are allowed to contain comments of the form "/*A-overwrites-Z*/" to indicate
that the semantic value "A" on the right-hand side of the rule is allowed
to directly overwrite the semantic value "Z" on the left-hand side.
This simple optimization reduces the number of stack operations in the
push-down automaton used to parse the input grammar, and thus improve
performance of the parser.  It also makes the generated code a little smaller.


The parsing of SQL statements is a significant consumer of CPU cycles 
in any SQL database engine.  On-going efforts to optimize SQLite have caused
the developers to spend a lot of time tweaking Lemon to generate faster
parsers.  These efforts have benefited all users of the Lemon parser generator,
not just SQLite.  But if Lemon had been a separately maintained tool, it
would have been more difficult to make coordinated changes to both SQLite
and Lemon, and as a result not as much optimization would have been
accomplished.  Hence, the fact that the parser generator tool is included
in the source tree for SQLite has turned out to be a net benefit for both
the tool itself and for SQLite.


c2: The Lemon LALR(1) Parser Generator
c3: 3. History Of Lemon
c5: lemon.html#history_of_lemon
c4:
Lemon was originally written by D. Richard Hipp (also the creator of SQLite)
while he was in graduate school at Duke University between 1987 and 1992.
The original creation date of Lemon has been lost, but was probably sometime
around 1990.  Lemon generates an LALR(1) parser.  There was a companion 
LL(1) parser generator tool named "Lime", but the source code for Lime
has been lost.

The Lemon source code was originally written as separate source files,
and only later merged into a single "lemon.c" source file.

The author of Lemon and SQLite (Hipp) reports that his C programming
skills were greatly enhanced by studying John Ousterhout's original
source code to Tcl.  Hipp discovered and studied Tcl in 1993.  Lemon
was written before then, and SQLite afterwards.  There is a clear
difference in the coding styles of these two products, with SQLite seeming
to be cleaner, more readable, and easier to maintain.


c2: The Error And Warning Log
c3: Overview
c5: errlog.html#overview
c4:
SQLite can be configured to invoke a callback function containing
an error code and a terse error message whenever anomalies occur.
This mechanism is very helpful in tracking obscure problems that
occur rarely and in the field.  Application developers are encouraged
to take advantage of the error logging facility of SQLite in their
products, as it is very low CPU and memory cost but can be a
huge aid for debugging.


c2: The Error And Warning Log
c3: 1. Setting Up The Error Logging Callback
c5: errlog.html#setting_up_the_error_logging_callback
c4:
There can only be a single error logging callback per process.
The error logging callback is registered at start-time using C-code
similar to the following:


sqlite3_config(SQLITE_CONFIG_LOG, errorLogCallback, pData);


The error logger callback function might look something like this:


void errorLogCallback(void *pArg, int iErrCode, const char *zMsg){
  fprintf(stderr, "(%d) %s\n", iErrCode, zMsg);
}


The example above illustrates the signature of the error logger callback.
However, in an embedded application, one usually does not print
messages on stderr.  Instead, one might store the messages in a
preallocated circular buffer where they can be accessed when diagnostic
information is needed during debugging.  Or perhaps the messages can be
sent to Syslog.  Somehow, the
messages need to be stored where they are accessible to developers,
not displayed to end users.

Do not misunderstand: There is nothing technically wrong with displaying 
the error logger messages to end users.  The messages do not contain
sensitive or private information that must be protected from unauthorized
viewing.  Rather the messages are technical in nature and are not useful
or meaningful to the typical end user.  The messages coming from the
error logger are intended for database geeks.  Display them accordingly.


c2: The Error And Warning Log
c3: 2. Interface Details
c5: errlog.html#interface_details
c4:
The third argument to the sqlite3_config(SQLITE_CONFIG_LOG,...) 
interface (the "pData" argument in the example above) is a pointer to arbitrary
data.  SQLite passes this pointer through to the first argument of the
error logger callback.  The pointer can be used to pass application-specific 
setup or state information, if desired.  Or it can simply be a NULL 
pointer which is ignored by the callback.

The second argument to the error logger callback is an integer
extended error code.  The third argument to the error logger is the
text of the error message.  The error message text is stored in a fixed-length
stack buffer in the calling function and so will only be valid for the
duration of the error logger callback function.  The error logger should
make a copy of this message into persistent storage if retention of the
message is needed.

The error logger callback should be treated like a signal handler.
The application should save off or otherwise process the error, then return
as soon as possible.  No other SQLite APIs should be invoked, directly or
indirectly, from the error logger.  SQLite is not reentrant through
the error logger callback.  In particular, the error logger callback
is invoked when a memory allocation fails, so it is generally a bad idea
to try to allocate memory inside the error logger.  Do not even think
about trying to store the error message in another SQLite database.

Applications can use the sqlite3_log(E,F,..) API to send new messages
to the log, if desired, but this is discouraged.  The sqlite3_log()
interface is intended for use by extensions only, not by applications.


c2: The Error And Warning Log
c3: 3. Variety of Error Messages
c5: errlog.html#variety_of_error_messages
c4:
The error messages that might be sent to the error logger and their
exact format is subject to changes from one release to the next.  So
applications should not depend on any particular error message text formats or
error codes.  Things do not change capriciously, but they do sometimes
changes.

The following is a partial list of the kinds of messages that might
appear in the error logger callback.



Any time there is an error either compiling an SQL statement 
(using sqlite3_prepare_v2() or its siblings) or running an SQL
statement (using sqlite3_step()) that error is logged.



When a schema change occurs that requires a prepared statement to be reparsed
and reprepared, that event is logged with the error code SQLITE_SCHEMA.
The reparse and reprepare is normally automatic (assuming that
sqlite3_prepare_v2() has been used to prepared the statements originally,
which is recommended) and so these logging events are normally the only
way to know that reprepares are taking place.


SQLITE_NOTICE messages are logged whenever a database has to be recovered
because the previous writer crashed without completing its transaction.
The error code is SQLITE_NOTICE_RECOVER_ROLLBACK when recovering a
rollback journal and SQLITE_NOTICE_RECOVER_WAL when recovering a 
write-ahead log.



SQLITE_WARNING messages are logged when database files are renamed or
aliased in ways that can lead to database corruption.
(See 1 and 2 for
additional information.)



Out of memory (OOM) error conditions generate error logging events
with the SQLITE_NOMEM error code and a message that says how many bytes
of memory were requested by the failed allocation.


I/O errors in the OS-interface generate error logging events.
The message to these events gives the line number in the source code where
the error originated and the filename associated with the event when
there is a corresponding file. 

When database corruption is detected, an SQLITE_CORRUPT error
logger callback is invoked.  As with I/O errors, the error message text
contains the line number in the original source code where the error
was first detected.


An error logger callback is invoked on SQLITE_MISUSE errors.
This is useful in detecting application design issues when return codes
are not consistently checked in the application code.


SQLite strives to keep error logger traffic low and only send messages
to the error logger when there really is something wrong.  Applications
might further cull the error message traffic 
by deliberately ignore certain classes of error
messages that they do not care about.  For example, an application that
makes frequent database schema changes might want to ignore all
SQLITE_SCHEMA errors.


c2: The Error And Warning Log
c3: 4. Summary
c5: errlog.html#summary
c4:
The use of the error logger callback is highly recommended.
The debugging information that the error logger provides has proven
very useful in tracking down obscure problems that occurs with applications
after they get into the field.  The error logger callback has also 
proven useful in catching errors occasional errors that the application
misses because of inconsistent checking of API return codes.
Developers are encouraged to implement an error logger callback early
in the development cycle in order to spot unexpected behavior quickly,
and to leave the error logger callback turned on through deployment.
If the error logger never finds a problem, then no harm is done.  
But failure to set up an appropriate error logger might compromise
diagnostic capabilities later on.


c2: The SQLITE_DBPAGE Virtual Table
c3: 1. Overview
c5: dbpage.html#overview
c4:
The SQLITE_DBPAGE extension implements an eponymous-only virtual table that
provides direct access to the underlying database file by interacting
with the pager.  SQLITE_DBPAGE is capable of both reading and writing any
page of the database.  Because interaction is through the pager layer, all
changes are transactional.



Warning: writing to the SQLITE_DBPAGE virtual table can very easily
cause unrecoverably database corruption.  Do not allow untrusted components
to access the SQLITE_DBPAGE table.  Use appropriate care while using the
SQLITE_DBPAGE table.  Back up important data prior to experimenting with the
SQLITE_DBPAGE table.  Writes to the SQLITE_DBPAGE virtual table are
disabled when the SQLITE_DBCONFIG_DEFENSIVE flag is set.


The SQLITE_DBPAGE extension is included in the amalgamation though 
it is disabled
by default.  Use the SQLITE_ENABLE_DBPAGE_VTAB compile-time option to enable
the SQLITE_DBPAGE extension.  The SQLITE_DBPAGE extension makes use of
unpublished internal interfaces and is not run-time loadable.  The only way
to add SQLITE_DBPAGE to an application is to compile it in using the
SQLITE_ENABLE_DBPAGE_VTAB compile-time option.



The SQLITE_DBPAGE extension is enabled in default builds
of the command-line shell.


c2: The SQLITE_DBPAGE Virtual Table
c3: 2. Usage
c5: dbpage.html#usage
c4:
The SQLITE_DBPAGE virtual table read/write table that provides direct
access to the underlying disk file on a page-by-page basis.  The
virtual table appears to have a schema like this:

CREATE TABLE sqlite_dbpage(
  pgno INTEGER PRIMARY KEY,
  data BLOB
);



An SQLite database file is divided into pages.
The first page is 1, the second page is 2, and so forth.
There is no page 0.
Every page is the same size.
The size of every page is a power of 2 between 512 and 65536.
See the file format documentation for further details.


The SQLITE_DBPAGE table allows an application to view or replace the
raw binary content of each page of the database file.
No attempt is made to interpret the content of the page.
Content is returned byte-for-byte as it appears on disk.


The SQLITE_DBPAGE table has one row for each page in the database file.
SQLITE_DBPAGE allows pages to be read or to be overwritten.
However the size of the database file cannot be changed.  It is not
possible to change the number of rows in the SQLITE_DBPAGE table by
running DELETE or INSERT operations against that table.


c2: The SQLITE_DBPAGE Virtual Table
c3: 2.1. Using SQLITE_DBPAGE On ATTACH-ed Databases
c5: dbpage.html#using_sqlite_dbpage_on_attach_ed_databases
c4:
The SQLITE_DBPAGE table schema shown above is incomplete.  There is
a third hidden column named "schema" that determines which
ATTACH-ed database should be read or written.  Because
the "schema" column is hidden, it can be used as a parameter when
SQLITE_DBPAGE is invoked as a table-valued function.


For example, suppose an additional database is attached to the 
database connection using a statement like this:

ATTACH 'auxdata1.db' AS aux1;



Then to read the first page of that database file, one merely runs:

SELECT data FROM sqlite_dbpage('aux1') WHERE pgno=1;



If the "schema" is omitted, it defaults to the primary database
(usually called 'main', unless renamed using SQLITE_DBCONFIG_MAINDBNAME).
Hence, the following two queries are normally equivalent:

SELECT data FROM sqlite_dbpage('main') WHERE pgno=1;
SELECT data FROM sqlite_dbpage WHERE pgno=1;



The SQLITE_DBPAGE table can participate in a join just like any other
table.  Hence, to see the content of the first page to all connected
database files, one might run a statement like this:

SELECT dbpage.data, dblist.name
  FROM pragma_database_list AS dblist
  JOIN sqlite_dbpage(dblist.name) AS dbpage
 WHERE dbpage.pgno=1;


c2: Row Values
c3: 1. Definitions
c5: rowvalue.html#definitions
c4:
A "value" is a single number, string, BLOB or NULL.
Sometimes the qualified name "scalar value" is used to emphasize that
only a single quantity is involved.

A "row value" is an ordered list of two or more scalar values.
In other words, a "row value" is a vector.

The "size" of a row value is the number of scalar values the row value contains.
The size of a row value is always at least 2.
A row value with a single column is just a scalar value.
A row value with no columns is a syntax error.


c2: Row Values
c3: 2. Syntax
c5: rowvalue.html#syntax
c4:
SQLite allows row values to be expressed in two ways:

A parenthesized, comma-separated list of scalar values.
A subquery expression with two or more result columns.


SQLite can use row values in two contexts:

Two row values of the same size 
can be compared using operators &lt;, &lt;=, &gt;, &gt;=,
=, &lt;&gt;, IS, IS NOT, IN, NOT IN, BETWEEN, or CASE.
In an UPDATE statement, a list of column names can be set to a row value of
the same size.


The syntax for row values and the circumstances in which row values
can be used are illustrated in examples below.


c2: Row Values
c3: 2.1. Row Value Comparisons
c5: rowvalue.html#row_value_comparisons
c4:
Two row values are compared by looking at the constituent scalar
values from left to right.
A NULL means of "unknown".  
The overall result of comparison is NULL if it is possible to make the
result either true or false by substituting alternative values in place
of the constituent NULLs.
The following query demonstrates some row value comparisons:

SELECT
  (1,2,3) = (1,2,3),          -- 1
  (1,2,3) = (1,NULL,3),       -- NULL
  (1,2,3) = (1,NULL,4),       -- 0
  (1,2,3) &lt; (2,3,4),          -- 1
  (1,2,3) &lt; (1,2,4),          -- 1
  (1,2,3) &lt; (1,3,NULL),       -- 1
  (1,2,3) &lt; (1,2,NULL),       -- NULL
  (1,3,5) &lt; (1,2,NULL),       -- 0
  (1,2,NULL) IS (1,2,NULL);   -- 1


The result of "(1,2,3)=(1,NULL,3)" is NULL because the result might be
true if we replaced NULL&rarr;2 or false if we replaced NULL&rarr;9.
The result of "(1,2,3)=(1,NULL,4)" is not NULL because there is no
substitutions of the constituent NULL that will make the expression true,
since 3 will never equal 4 in the third column.

Any of the row values in the previous example could be replace by a
subquery that returns three columns and the same answer would result.
For example:

CREATE TABLE t1(a,b,c);
INSERT INTO t1(a,b,c) VALUES(1,2,3);
SELECT (1,2,3)=(SELECT * FROM t1); -- 1


For a row-value IN operator, the left-hand side (hereafter "LHS") can be either
a parenthesized list of values or a subquery with multiple columns.  But the
right-hand side (hereafter "RHS") must be a subquery expression.

CREATE TABLE t2(x,y,z);
INSERT INTO t2(x,y,z) VALUES(1,2,3),(2,3,4),(1,NULL,5);
SELECT
   (1,2,3) IN (SELECT * FROM t2),  -- 1
   (7,8,9) IN (SELECT * FROM t2),  -- 0
   (1,3,5) IN (SELECT * FROM t2);  -- NULL


c2: Row Values
c3: 2.2. Row Values In UPDATE Statements
c5: rowvalue.html#row_values_in_update_statements
c4:
Row values can also be used in the SET clause of an UPDATE statement.
The LHS must be a list of column names.  The RHS can be any row value.
For example:

UPDATE tab3 
   SET (a,b,c) = (SELECT x,y,z
                    FROM tab4
                   WHERE tab4.w=tab3.d)
 WHERE tab3.e BETWEEN 55 AND 66;


c2: Row Values
c3: 3.1. Scrolling Window Queries
c5: rowvalue.html#scrolling_window_queries
c4:
Suppose an application wants to display a list of contacts
in alphabetical order by lastname, firstname, in a scrolling window
that can only show 7 contacts at a time.  Initialize the scrolling
window to the first 7 entries is easy:

SELECT * FROM contacts
 ORDER BY lastname, firstname
 LIMIT 7;


When the user scrolls down, the application needs to find the
second set of 7 entries.  One way to do this is to use the OFFSET clause:

SELECT * FROM contacts
 ORDER BY lastname, firstname
 LIMIT 7 OFFSET 7;


OFFSET gives the correct answer.  However, OFFSET requires time
proportional to the offset value.  What really happens
with "LIMIT x OFFSET y" is that SQLite computes the query as
"LIMIT x+y" and discards the first y values without returning them
to the application.  So as the window scrolls down toward
the bottom of a long list, and the y value becomes larger and larger,
successive offset computations take more and more time.

A more efficient approach is to remember the last entry currently
displayed and then use a row value comparison in the WHERE
clause:

SELECT * FROM contacts
 WHERE (lastname,firstname) &gt; (?1,?2)
 ORDER BY lastname, firstname
 LIMIT 7;


If the lastname and firstname on the bottom row of the previous
screen are bound to ?1 and ?2, then the query above computes the next
7 rows.  And, assuming there is an appropriate index, it does so
very efficiently &mdash; much more efficiently than OFFSET.


c2: Row Values
c3: 3.2. Comparison of dates stored as separate fields
c5: rowvalue.html#comparison_of_dates_stored_as_separate_fields
c4:
The usual way of storing a date in a database table is as a single
field, as either a unix timestamp, a julian day number, or an ISO-8601
dates string.  But some application store dates as three separate
fields for the year, month, and day.  

CREATE TABLE info(
  year INT,          -- 4 digit year
  month INT,         -- 1 through 12
  day INT,           -- 1 through 31
  other_stuff BLOB   -- blah blah blah
);


When dates are stored this way, row value comparisons provide a
convenient way to compare dates:

SELECT * FROM info
 WHERE (year,month,day) BETWEEN (2015,9,12) AND (2016,9,12);


c2: Row Values
c3: 3.3. Search against multi-column keys
c5: rowvalue.html#search_against_multi_column_keys
c4:
Suppose we want to know the order number, product number, and quantity
for any item in which the product number and quantity match the product
number and quantity of any item in order number 365:

SELECT ordid, prodid, qty
  FROM item
 WHERE (prodid, qty) IN (SELECT prodid, qty
                           FROM item
                          WHERE ordid = 365);


The query above could be rewritten as a join and without the use
of row values:

SELECT t1.ordid, t1.prodid, t1.qty
  FROM item AS t1, item AS t2
 WHERE t1.prodid=t2.prodid
   AND t1.qty=t2.qty
   AND t2.ordid=365;


Because the same query could be written without the use of row values,
row values do not provide new capabilities.  However, many developers say
that the row value format is easier to read, write, and debug.

Even in the JOIN form, the query can be made clearer through the use of
row values:

SELECT t1.ordid, t1.prodid, t1.qty
  FROM item AS t1, item AS t2
 WHERE (t1.prodid,t1.qty) = (t2.prodid,t2.qty)
   AND t2.ordid=365;


This later query generates exactly the same bytecode as the previous
scalar formulation, but using syntax that it cleaner and
easier to read.


c2: Row Values
c3: 3.4. Update multiple columns of a table based on a query
c5: rowvalue.html#update_multiple_columns_of_a_table_based_on_a_query
c4:
The row-value notation is useful for updating two or more columns
of a table from the result of a single query.
An example of this is in the full-text search feature of the
Fossil version control system.

In the Fossil full-text search system,
documents that participate in the full-text search (wiki pages, tickets,
check-ins, documentation files, etc) are tracked by a table called
"ftsdocs" (full text search documents).
As new documents are added to the repository, they are not indexed right
away.  Indexing is deferred until there is a search request.  The
ftsdocs table contains an "idxed" field which is true if the document
has been indexed and false if not.

When a search request occurs and pending documents are indexed for the
first time, the ftsdocs table must be updated by setting the idxed column
to true and also filling in several other columns with information pertinent
to the search.  That other information is obtained from a join.  The
query is this:

UPDATE ftsdocs SET
  idxed=1,
  name=NULL,
  (label,url,mtime) = 
      (SELECT printf('Check-in &#91;%%.16s&#93; on %%s',blob.uuid,
                     datetime(event.mtime)),
              printf('/timeline?y=ci&amp;c=%%.20s',blob.uuid),
              event.mtime
         FROM event, blob
        WHERE event.objid=ftsdocs.rid
          AND blob.rid=ftsdocs.rid)
WHERE ftsdocs.type='c' AND NOT ftsdocs.idxed


(See the 
source code
for further detail.  Other examples
here and
here.)

Five out of nine columns in the ftsdocs table are updated.  Two of
the modified columns, "idxed" and "name", can be updated independently of
the query.  But the three columns "label", "url", and "mtime" all require
a join query against the "event" and "blob" tables.  Without row values,
the equivalent UPDATE would require that the join be repeated three times, 
once for each column to be updated.


c2: Row Values
c3: 3.5. Clarity of presentation
c5: rowvalue.html#clarity_of_presentation
c4:
Sometimes the use of row values just makes the SQL easier to read
and write.  Consider the following two UPDATE statements:

UPDATE tab1 SET (a,b)=(b,a);
UPDATE tab1 SET a=b, b=a;


Both UPDATE statements do exactly the same thing.  (They generate
identical bytecode.)  But the first form, the row value form, seems
to make it clearer that the intent of the statement is to swap the
values in columns A and B.

Or consider these identical queries:

SELECT * FROM tab1 WHERE a=?1 AND b=?2;
SELECT * FROM tab1 WHERE (a,b)=(?1,?2);


Once again, the SQL statements generate identical bytecode and thus
do exactly the same job in exactly the same way.  But the second form
is made easier for humans to read by grouping the query parameters together
into a single row value rather than scattering them across the WHERE
clause.


c2: Row Values
c3: 4. Backwards Compatibility
c5: rowvalue.html#backwards_compatibility
c4:
Row values were added to SQLite
version 3.15.0 (2016-10-14).  Attempts to use row values in
prior versions of SQLite will generate syntax errors.


c2: The UINT Collating Sequence
c3: 1. Overview
c5: uintcseq.html#overview
c4:
The UINT collating sequences is a loadable extension for
SQLite that implements a new collating sequence that compares text
containing unsigned integers in numeric order.

 The UINT collating sequence is not a standard part of SQLite.
It must be loaded as a separate extension.  The source code to
UINT is in the uint.c source file
in the ext/misc/ folder of the
SQLite source tree.

 The UINT collating sequence is not included in standard builds of
the SQLite library, but it is loaded by default in the CLI.  This
is typical of the CLI which loads various extensions above and beyond
what are available in the standard SQLite library.

 The UINT collating sequence works just like the default
BINARY collating sequence for text, except that embedded strings
of digits compare in numeric order.



Leading zeros are handled properly, in the sense that
they do not mess of the maginitude comparison of embedded
strings of digits.  "x00123y" is equal to "x123y".


Only unsigned integers are recognized.  Plus and minus
signs are ignored.  Decimal points and exponential notation
are ignored.


Embedded integers can be of arbitrary length.  Comparison
is not limited to integers that can be expressed as a
64-bit machine integer.


c2: The UINT Collating Sequence
c3: 2. Example:
c5: uintcseq.html#example_
c4:
COLLATE binaryCOLLATE uint

'0000123457'
'123456'
'abc0000000010xyz'
'abc0010xyy'
'abc10xzz'
'abc674xyz'
'abc87xyz'
'abc9xyz'

'123456'
'0000123457'
'abc9xyz'
'abc0010xyy'
'abc0000000010xyz'
'abc10xzz'
'abc87xyz'
'abc674xyz'


c2: Quirks, Caveats, and Gotchas In SQLite
c3: 1. Overview
c5: quirks.html#overview
c4:
The SQL language is a "standard".
Even so, no two SQL database engines work exactly alike.
Every SQL implementation has it own peculiarities and oddities,
and SQLite is no exception to this rule.


This document strives to highlight the principal differences
between SQLite and other SQL implementations, as an aid to developers
that are porting to or from SQLite or who are trying to build a
system that works across multiple database engines.


If you are an SQLite user whose has stumbled over some quirk of
SQLite that is not mentioned here, please send us an email so that
we can document the problem.


c2: Quirks, Caveats, and Gotchas In SQLite
c3: 2. SQLite Is Embedded, Not Client-Server
c5: quirks.html#sqlite_is_embedded_not_client_server
c4:
Whenever comparing SQLite to other SQL database engines like
SQL Server, PostgreSQL, MySQL, or Oracle, it is important first of all
to realize that SQLite is not intended as a replacement or competitor to
any of those systems.  SQLite is serverless.  There is no separate
server process that manages the database.  An application interacts
with the database engine using function calls, not by sending messages
to a separate process or thread.


The fact that SQLite is embedded and serverless instead of being
client/server is a feature, not a bug.


Client/server databases like MySQL, PostgreSQL, SQL Server, Oracle, and
others are an important component of modern systems.  
These systems solve an important problem.  
But SQLite solves a different problem.
Both SQLite and client/server databases have their role.
Developers who are comparing SQLite against other SQL database engines need
to clearly understand this distinction.


See the Appropriate Uses For SQLite document for additional information.


c2: Quirks, Caveats, and Gotchas In SQLite
c3: 3. Flexible Typing
c5: quirks.html#flexible_typing
c4:
SQLite is very flexible with regard to datatypes.


Some commentators say that SQLite is "weakly typed" and that other
SQL databases are "strongly typed".  We consider these terms to be
inaccurate and pejorative.  We prefer to say that SQLite is "flexibly typed"
and that other SQL databases are "rigidly typed".


See the Datatypes in SQLite Version 3 document for a detailed
discussion of the type system in SQLite.


The key point is that SQLite is very forgiving of the type of data that
you put into the database.  For example, if a column has a datatype of
"INTEGER" and the application inserts a text string into that column,
SQLite will first try to convert the text string into an integer, just like
every other SQL database engine.  Thus, if one inserts '1234' into
an INTEGER column, that value is converted into an integer 1234 and stored.
But, if you insert a non-numeric string like 'wxyz' into an INTEGER
column, unlike other SQL databases, SQLite does not throw an error.  Instead,
SQLite stores the actual string value in the column.


Similarly, SQLite allows you to store a 2000-character string into a
column of type VARCHAR(50).  Other SQL implementations would either throw
an error or truncate the string.  SQLite stores the entire 2000-character
string with no loss of information and without complaint.


Where this ends up causing problems is when developers do some initial
coding work using SQLite and get their application working, but then try
to convert to another database like PostgreSQL or SQL Server for deployment.
If the application is initially taking advantage of SQLite's flexible typing,
then it will fail when moved to another database that uses a more rigid
and unforgiving type enforcement policy.


Flexible typing is considered a feature of SQLite, not a bug.
Nevertheless, we recognize that this feature does sometimes cause
confusion and pain for developers who are acustomed to working with
other databases that are more judgmental with regard to data types.
In retrospect, perhaps it would have been better if SQLite had merely
implemented an ANY datatype so that developers could explicitly state
when they wanted to use flexible typing, rather than making flexible
typing the default.
But that is not something that can be changed now without breaking
the millions of applications and trillions of database files that 
already use SQLite's flexible typing feature.


c2: Quirks, Caveats, and Gotchas In SQLite
c3: 3.1. No Separate BOOLEAN Datatype
c5: quirks.html#no_separate_boolean_datatype
c4:
Unlike most other SQL implementations,
SQLite does not have a separate BOOLEAN data type.
Instead, TRUE and FALSE are (normally) represented as integers 1 and 0,
respectively.
This does not seem to cause many problems, as we seldom get complaints
about it.  But it is important to recognize.


Beginning with SQLite version 3.23.0 (2018-04-02), SQLite also
recognizes TRUE and FALSE keywords as aliases for integer values 1 and 0,
respectively.
This provides better compatibility with other SQL implementations.
But to retain backwards compatibility, if there are columns named TRUE or
FALSE, then the keywords are treated as identifiers referencing those
columns, rather than BOOLEAN literals.


c2: Quirks, Caveats, and Gotchas In SQLite
c3: 3.2. No Separate DATETIME Datatype
c5: quirks.html#no_separate_datetime_datatype
c4:
SQLite has no DATETIME datatype.
Instead, dates and times can be stored in any of these ways:


 As a TEXT string in the ISO-8601 format.  Example: '2018-04-02 12:13:46'.
 As an INTEGER number of seconds since 1970 (also known as "unix time").
 As a REAL value that is the fractional 
     Julian day number.



The built-in date and time functions of SQLite understand date/times in
all of the formats above, and can freely change between them.
Which format you use, is entirely up to your application.


c2: Quirks, Caveats, and Gotchas In SQLite
c3: 3.3. The datatype is optional
c5: quirks.html#the_datatype_is_optional
c4:
Because SQLite is flexible and forgiving with regard to datatypes,
table columns can be created that have no specified datatype at all.
For example:
CREATE TABLE t1(a,b,c,d);

The table "t1" has four columns "a", "b", "c", and "d" that have
no particular datatype assigned.  You can store anything you want in
any of those columns.


c2: Quirks, Caveats, and Gotchas In SQLite
c3: 4. Foreign Key Enforcement Is Off By Default
c5: quirks.html#foreign_key_enforcement_is_off_by_default
c4:
SQLite has parsed foreign key constraints for time out of mind,
but added the ability to actually enforce those constraints much later,
with version 3.6.19 (2009-10-14).  By the time foreign key constraint
enforcement was added, there were already countless millions of databases
in circulation that contained foreign key constraints, some of which
were not correct.  To avoid breaking those legacy databases, foreign key
constraint enforcement is turned off by default in SQLite.

Applications can activate foreign key enforcement at run-time using
the PRAGMA foreign_keys statement.  Or, foreign key enforcement can
be activated at compile-time using the
-DSQLITE_DEFAULT_FOREIGN_KEYS=1 compile-time option.


c2: Quirks, Caveats, and Gotchas In SQLite
c3: 5. PRIMARY KEYs Can Sometimes Contain NULLs
c5: quirks.html#primary_keys_can_sometimes_contain_nulls
c4:
Usually (the exceptions are INTEGER PRIMARY KEY tables and
WITHOUT ROWID tables) a PRIMARY KEY in an SQLite table is really
the same as a UNIQUE constraint.  Due to an historical oversight,
the column values of such a PRIMARY KEY are allowed to be NULL.
This is a bug, but by the time the problem was discovered there
where so many databases in circulation that depended on the bug that
the decision was made to support the bugging behavior moving forward.

The value of an INTEGER PRIMARY KEY column must always be a 
non-NULL integer.  The PRIMARY KEY columns of a WITHOUT ROWID
table are also required to be non-NULL.


c2: Quirks, Caveats, and Gotchas In SQLite
c3: 6. Aggregate Queries Can Contain Non-Aggregate Result Columns That Are Not In The GROUP BY Clause
c4:
In most SQL implementations, output columns of an aggregate query
may only reference aggregate functions or columns named in the
GROUP BY clause.  It does not make good sense to reference an ordinary
column in an aggregate query because each output row might be composed
from two or more rows in the input table(s).


SQLite does not enforce this restriction.
The output columns from an aggregate query can be arbitrary expressions
that include columns not found in GROUP BY clause.
This feature has two uses:



With SQLite (but not any other SQL implementation that we know of) if
an aggregate query contains a single min() or max() function, then the
values of columns used in the output are taken from the row where
the min() or max() value was achieved.  If two or more rows have the
same min() or max() value, then the columns values will be chosen arbitrarily
from one of those rows.

For example to find the highest paid employee:
SELECT max(salary), first_name, last_name FROM employee;


In the query above, the values for the first_name and last_name columns
will correspond to the row that satisfied the max(salary) condition.


If a query contains no aggregate functions at all, then a GROUP BY
clause can be added as a substitute for the DISTINCT ON clause.  In other words,
output rows are filtered so that only one row is shown for each distinct
set of values in the GROUP BY clause.  If two or more output rows would
have otherwise had the same set of values for the GROUP BY columns, then
one of the rows is chosen arbitrarily.  (SQLite supports DISTINCT but not
DISTINCT ON, whose functionality is provided instead by GROUP BY.)

c5:
quirks.html#aggregate_queries_can_contain_non_aggregate_result_columns_that_are_not_in_the_group_by_clause


c2: Quirks, Caveats, and Gotchas In SQLite
c3: 7. Does Not Do Full Unicode Case Folding By Default
c5: quirks.html#does_not_do_full_unicode_case_folding_by_default
c4:
SQLite does not know about the upper-case/lower-case distinction
for all unicode characters.  SQL functions like
upper() and lower() only work on ASCII characters.  There are two
reasons for this:

 Though stable now, when SQLite was first designed, the rules for
     unicode case folding where still in flux.  That means that the
     behavior might have changed with each new unicode release, disrupting
     applications and corrupting indexes in the process.
 The tables necessary to do full and proper unicode case folding are
     larger than the whole SQLite library.


Full unicode case folding is supported in SQLite if it is compiled
with the -DSQLITE_ENABLE_ICU option and linked against the
International Components for Unicode
library.


c2: Quirks, Caveats, and Gotchas In SQLite
c3: 8. Double-quoted String Literals Are Accepted
c5: quirks.html#double_quoted_string_literals_are_accepted
c4:
The SQL standard requires double-quotes around identifiers
and single-quotes around string literals.  For example:

 "this is a legal SQL column name"
 'this is an SQL string literal'


SQLite accepts both of the above.  But, in an effort to be compatible
with MySQL 3.x (which was one of the most widely used RDBMSes
when SQLite was first being designed) SQLite will also interpret
a double-quotes string as
string literal if it does not match any valid identifier.

This misfeature means that a misspelled double-quoted
identifier will be interpreted as a string literal, rather than generating
an error.
It also lures developers who are new to the SQL language into the
bad habit of using double-quoted string literals when they
really need to learn to use the correct single-quoted string literal form.

In hindsight, we should not have tried to make SQLite accept MySQL 3.x
syntax, and should have never allowed double-quoted string literals.
However, there are countless applications that make use of
double-quoted string literals and so we continue to support
that capability to avoid breaking legacy.

As of SQLite 3.27.0 (2019-02-07) the use of a double-quoted
string literal causes a warning message to be sent to the error log.

As of SQLite 3.29.0 (2019-07-10) the use of double-quoted
string literals can be disabled at run-time using the
SQLITE_DBCONFIG_DQS_DDL and SQLITE_DBCONFIG_DQS_DML actions
to sqlite3_db_config().  The default settings can be altered
at compile-time using the -DSQLITE_DQS=N compile-time
option.  Application developers are encouraged to compile using
-DSQLITE_DQS=0 in order to disable the double-quoted string literal
misfeature by default.  If that is not possible, then disable
double-quoted string literals for individual database connections
using C-code like this:

sqlite3_db_config(db, SQLITE_DBCONFIG_DQS_DDL, 0, (void*)0);
sqlite3_db_config(db, SQLITE_DBCONFIG_DQS_DML, 0, (void*)0);

Or, if double-quoted string literals are disabled by default, but need
to be selectively enabled for some historical database connections,
that can be done using the same C-code as shown above except with the
third parameter changed from 0 to 1.


c2: Quirks, Caveats, and Gotchas In SQLite
c3: 9. Keywords Can Often Be Used As Identifiers
c5: quirks.html#keywords_can_often_be_used_as_identifiers
c4:
The SQL language is rich in keywords.
Most SQL implementations do not allow keywords to be used as identifiers
(the names of table or columns) unless they are enclosed in double-quotes.
But SQLite is more flexible.  Many keywords can be used as identifiers without
needing to be quoted, as long as those keywords are used in a context where
it is clear that they are intended to be an identifier.

For example, the following statement is valid in SQLite:
CREATE TABLE union(true INT, with BOOLEAN);


The same SQL statement will fail on every other SQL implementation that
we know of due to the use of keywords "union", "true", and "with" as
identifiers.

The ability to use keywords as identifiers promotes backwards compatibility.
As new keywords are added, legacy schemas that just happen to use those
keywords as table or column names continue to work.  However, the ability
to use a keyword as an identifier sometimes leads to surprising outcomes.
For example:
CREATE TRIGGER AFTER INSERT ON tableX BEGIN
  INSERT INTO tableY(b) VALUES(new.a);
END;

The trigger created by the previous statement is named "AFTER"
and it is a "BEFORE" trigger.  The "AFTER" token is used as an identifier
instead of as a keyword, as that is the only way to parse the statement.
Another example:
CREATE TABLE tableZ(INTEGER PRIMARY KEY);

The tableZ table has a single column named "INTEGER".  That column
has no datatype specified, but it is the PRIMARY KEY.
The column is not the INTEGER PRIMARY KEY for the table because
it has no datatype.  The "INTEGER" token is used as an identifier for the
column name, not as a datatype keyword.


c2: Quirks, Caveats, and Gotchas In SQLite
c3: 10. Dubious SQL Is Allowed Without Any Error Or Warning
c5: quirks.html#dubious_sql_is_allowed_without_any_error_or_warning
c4:
The original implementation of SQLite sought to follow
Postel's Law which
states in part "Be liberal in what you accept".
This used to be considered good design - that a system would accept
dodgy inputs and try to do the best it could without complaining too much.
But lately, people have come to realize that it is sometimes better to
be strict in what you accept, so as to more easily find errors in the
input.


c2: Quirks, Caveats, and Gotchas In SQLite
c3: 11. AUTOINCREMENT Does Not Work The Same As MySQL
c5: quirks.html#autoincrement_does_not_work_the_same_as_mysql
c4:
The AUTOINCREMENT feature in SQLite works differently than
it does in MySQL.  This often causes confusion for people who
initially learned SQL on MySQL and then start using SQLite, and
expect the two systems to work identically.

See the SQLite AUTOINCREMENT documentation for
detailed instructions on what AUTOINCREMENT does and does not do
in SQLite.


c2: Quirks, Caveats, and Gotchas In SQLite
c3: 12. NUL Characters Are Allowed In Text Strings
c5: quirks.html#nul_characters_are_allowed_in_text_strings
c4:
NUL characters (ASCII code 0x00 and Unicode \u0000) may appear in
the middle of strings in SQLite.  This can lead to unexpected behavior.
See the "NUL characters in strings" document for further information.


c2: The Spellfix1 Virtual Table
c3: 1. Overview
c5: spellfix1.html#overview
c4:
This spellfix1 virtual table can be used to search
a large vocabulary for close matches.  For example, spellfix1
can be used to suggest corrections to misspelled words.  Or,
it could be used with FTS4 to do full-text search using potentially
misspelled words.

The implementation for the spellfix1 virtual table is held in the
SQLite source tree in the miscellaneous extensions folder and in
particular in the file 
ext/misc/spellfix1.c.
The spellfix1 virtual table is not included in the SQLite amalgamation
and is not a part of any standard SQLite build.  It is a loadable extension.

Once the spellfix1 extension is loaded, an instance of the spellfix1 
virtual table is created like this:


CREATE VIRTUAL TABLE demo USING spellfix1;


The "spellfix1" term is the name of the spellfix module and must be 
entered as shown.  The "demo" term is the
name of the virtual table you will be creating and can be altered
to suit the needs of your application.  The virtual table is initially
empty.  In order for the virtual table to be useful, you will need to
populate it with your vocabulary.  Suppose you
have a list of words in a table named "big_vocabulary".  Then do this:


INSERT INTO demo(word) SELECT word FROM big_vocabulary;


If you intend to use this virtual table in cooperation with an FTS4
table (for spelling correction of search terms) then you might extract
the vocabulary using an fts4aux table:


INSERT INTO demo(word) SELECT term FROM search_aux WHERE col='*';


You can also provide the virtual table with a "rank" for each word.
The "rank" is an estimate of how common the word is.  Larger numbers
mean the word is more common.  If you omit the rank when populating
the table, then a rank of 1 is assumed.  But if you have rank 
information, you can supply it and the virtual table will show a
slight preference for selecting more commonly used terms.  To
populate the rank from an fts4aux table "search_aux" do something
like this:


INSERT INTO demo(word,rank)
   SELECT term, documents FROM search_aux WHERE col='*';


To query the virtual table, include a MATCH operator in the WHERE
clause.  For example:


SELECT word FROM demo WHERE word MATCH 'kennasaw';


Using a dataset of American place names (derived from
http://geonames.usgs.gov/domestic/download_data.htm) the query above
returns 20 results beginning with:


kennesaw
kenosha
kenesaw
kenaga
keanak


If you append the character '*' to the end of the pattern, then
a prefix search is performed.  For example:


SELECT word FROM demo WHERE word MATCH 'kennes*';


Yields 20 results beginning with:


kennesaw
kennestone
kenneson
kenneys
keanes
keenes


c2: The Spellfix1 Virtual Table
c3: 2. Search Refinements
c5: spellfix1.html#search_refinements
c4:
By default, the spellfix1 table returns no more than 20 results.
(It might return less than 20 if there were fewer good matches.)
You can change the upper bound on the number of returned rows by
adding a "top=N" term to the WHERE clause of your query, where N
is the new maximum.  For example, to see the 5 best matches:


SELECT word FROM demo WHERE word MATCH 'kennes*' AND top=5;


Each entry in the spellfix1 virtual table is associated with
a particular language, identified by the integer "langid" column.
The default langid is 0 and if no other actions are taken, the
entire vocabulary is a part of the 0 language.  But if your application
needs to operate in multiple languages, then you can specify different
vocabulary items for each language by specifying the langid field
when populating the table.  For example:


INSERT INTO demo(word,langid) SELECT word, 0 FROM en_vocabulary;
INSERT INTO demo(word,langid) SELECT word, 1 FROM de_vocabulary;
INSERT INTO demo(word,langid) SELECT word, 2 FROM fr_vocabulary;
INSERT INTO demo(word,langid) SELECT word, 3 FROM ru_vocabulary;
INSERT INTO demo(word,langid) SELECT word, 4 FROM cn_vocabulary;


After the virtual table has been populated with items from multiple
languages, specify the language of interest using a "langid=N" term
in the WHERE clause of the query:


SELECT word FROM demo WHERE word MATCH 'hildes*' AND langid=1;


Note that if you do not include the "langid=N" term in the WHERE clause,
the search will be against language 0 (English in the example above.)
All spellfix1 searches are against a single language id.  There is no
way to search all languages at once.


c2: The Spellfix1 Virtual Table
c3: 3. Virtual Table Details
c5: spellfix1.html#virtual_table_details
c4:
Each row in the spellfix1 virtual table has a unique rowid 
with seven columns plus five extra hidden columns.
The columns are as follows:


rowid
A unique integer number associated with each
vocabulary item in the table.  This can be used
as a foreign key on other tables in the database.

word
The text of the word that matches the pattern.
Both word and pattern can contain unicode characters
and can be mixed case.

rank
This is the rank of the word, as specified in the
original INSERT statement.


distance
This is an edit distance or Levenshtein distance going
from the pattern to the word.

langid
This is the language-id of the word.  All queries are
against a single language-id, which defaults to 0.
For any given query this value is the same on all rows.

score
The score is a combination of rank and distance.  The
idea is that a lower score is better.  The virtual table
attempts to find words with the lowest score and 
by default (unless overridden by ORDER BY) returns
results in order of increasing score.

matchlen
In a prefix search, the matchlen is the number of characters in
the string that match against the prefix.  For a non-prefix search,
this is the same as length(word).

phonehash
This column shows the phonetic hash prefix that was used to restrict
the search.  For any given query, this column should be the same for
every row.  This information is available for diagnostic purposes and
is not normally considered useful in real applications.

top
(HIDDEN)  For any query, this value is the same on all
rows.  It is an integer which is the maximum number of
rows that will be output.  The actually number of rows
output might be less than this number, but it will never
be greater.  The default value for top is 20, but that
can be changed for each query by including a term of
the form "top=N" in the WHERE clause of the query.

scope
(HIDDEN)  For any query, this value is the same on all
rows.  The scope is a measure of how widely the virtual
table looks for matching words.  Smaller values of
scope cause a broader search.  The scope is normally
chosen automatically and is capped at 4.  Applications
can change the scope by including a term of the form
"scope=N" in the WHERE clause of the query.  Increasing
the scope will make the query run faster, but will reduce
the possible corrections.

srchcnt
(HIDDEN)  For any query, this value is the same on all
rows.  This value is an integer which is the number of
words examined using the edit-distance algorithm to
find the top matches that are ultimately displayed.  This
value is for diagnostic use only.

soundslike
(HIDDEN)  When inserting vocabulary entries, this field
can be set to a spelling that matches what the word
sounds like.  See the DEALING WITH UNUSUAL AND DIFFICULT
SPELLINGS section below for details.

command
(HIDDEN)  The value of the "command" column is always NULL.  However,
applications can insert special strings into the "command" column in order
to provoke certain behaviors in the spellfix1 virtual table.
For example, inserting the string 'reset' into the "command" column
will cause the virtual table to reread its edit distance weights
(if there are any).


c2: The Spellfix1 Virtual Table
c3: 4. Algorithm
c5: spellfix1.html#algorithm
c4:
The spellfix1 virtual table creates a single
shadow table named "%_vocab" (where the % is replaced by the name of
the virtual table; Ex: "demo_vocab" for the "demo" virtual table).  
the shadow table contains the following columns:


id
The unique id (INTEGER PRIMARY KEY)

rank
The rank of word.

langid
The language id for this entry.

word
The original UTF8 text of the vocabulary word

k1
The word transliterated into lower-case ASCII.  
There is a standard table of mappings from non-ASCII
characters into ASCII.  Examples: "æ" -> "ae",
"þ" -> "th", "ß" -> "ss", "á" -> "a", ...  The
accessory function spellfix1_translit(X) will do
the non-ASCII to ASCII mapping.  The built-in lower(X)
function will convert to lower-case.  Thus:
k1 = lower(spellfix1_translit(word)).

If the word is already all lower-case ASCII, then the k1 column
will contain a NULL.  This reduces the storage requirements for
the %_vocab table and helps spellfix to run a little faster.
Therefore, it is advantageous to populate as much of the spellfix
table as possible using lower-case ASCII vocabulary.

k2
This field holds a phonetic code derived from coalesce(k1,word).
Letters that have similar sounds are mapped into the same symbol.
For example, all vowels and vowel clusters become the
single symbol "A".  And the letters "p", "b", "f", and
"v" all become "B".  All nasal sounds are represented
as "N".  And so forth.  The mapping is based on
ideas found in Soundex, Metaphone, and other
long-standing phonetic matching systems.  This key can
be generated by the function spellfix1_phonehash(X).  
Hence: k2 = spellfix1_phonehash(coalesce(k1,word))


There is also a function for computing the Wagner edit distance or the
Levenshtein distance between a pattern and a word.  This function
is exposed as spellfix1_editdist(X,Y).  The edit distance function
returns the "cost" of converting X into Y.  Some transformations
cost more than others.  Changing one vowel into a different vowel,
for example is relatively cheap, as is doubling a constant, or
omitting the second character of a double-constant.  Other transformations
or more expensive.  The idea is that the edit distance function returns
a low cost for words that are similar and a higher cost for words
that are further apart.  In this implementation, the maximum cost
of any single-character edit (delete, insert, or substitute) is 100,
with lower costs for some edits (such as transforming vowels).

The "score" for a comparison is the edit distance between the pattern
and the word, adjusted down by the base-2 logarithm of the word rank.
For example, a match with distance 100 but rank 1000 would have a
score of 122 (= 100 - log2(1000) + 32) whereas a match with distance
100 with a rank of 1 would have a score of 131 (100 - log2(1) + 32).
(NB:  The constant 32 is added to each score to keep it from going
negative in case the edit distance is zero.)  In this way, frequently
used words get a slightly lower cost which tends to move them toward
the top of the list of alternative spellings.

A straightforward implementation of a spelling corrector would be
to compare the search term against every word in the vocabulary
and select the 20 with the lowest scores.  However, there will 
typically be hundreds of thousands or millions of words in the
vocabulary, and so this approach is not fast enough.

Suppose the term that is being spell-corrected is X.  To limit
the search space, X is converted to a k2-like key using the
equivalent of:


   key = spellfix1_phonehash(lower(spellfix1_translit(X)))


This key is then limited to "scope" characters.  The default scope
value is 4, but an alternative scope can be specified using the
"scope=N" term in the WHERE clause.  After the key has been truncated,
the edit distance is run against every term in the vocabulary that
has a k2 value that begins with the abbreviated key.

For example, suppose the input word is "Paskagula".  The phonetic 
key is "BACACALA" which is then truncated to 4 characters "BACA".
The edit distance is then run on the 4980 entries (out of
272,597 entries total) of the vocabulary whose k2 values begin with
BACA, yielding "Pascagoula" as the best match.

Only terms of the vocabulary with a matching langid are searched.
Hence, the same table can contain entries from multiple languages
and only the requested language will be used.  The default langid
is 0.


c2: The Spellfix1 Virtual Table
c3: 5. Configurable Edit Distance
c5: spellfix1.html#configurable_edit_distance
c4:
The built-in Wagner edit-distance function with fixed weights can be
replaced by the editdist3() edit-distance function
with application-defined weights and support for unicode, by specifying
the "edit_cost_table=TABLENAME" parameter to the spellfix1 module
when the virtual table is created.
For example:


CREATE VIRTUAL TABLE demo2 USING spellfix1(edit_cost_table=APPCOST);


The editdist3() edit-distance function can also be selected or 
deselected at run-time by inserting an appropriate string into the
"command" column of the virtual table:


INSERT INTO demo2(command) VALUES('edit_cost_table=APPCOST');



In the examples above, the APPCOST table would be interrogated to find
the edit distance coefficients.  It is the presence of the "edit_cost_table="
parameter to the spellfix1 module name that causes editdist3() to be used
in place of the built-in edit distance function.  If APPCOST is an empty
string, then the built-in Wagner edit-distance function is used.

The edit distance coefficients are normally read from the APPCOST table
once and there after stored in memory.  Hence, run-time changes to the
APPCOST table will not normally affect the edit distance results.
However, inserting the special string 'reset' into the "command" column of the
virtual table causes the edit distance coefficients to be reread the
APPCOST table.  Hence, applications should run a SQL statement similar
to the following when changes to the APPCOST table occur:


INSERT INTO demo2(command) VALUES("reset");


c2: The Spellfix1 Virtual Table
c3: 6. Dealing With Unusual And Difficult Spellings
c5: spellfix1.html#dealing_with_unusual_and_difficult_spellings
c4:
The algorithm above works quite well for most cases, but there are
exceptions.  These exceptions can be dealt with by making additional
entries in the virtual table using the "soundslike" column.

For example, many words of Greek origin begin with letters "ps" where
the "p" is silent.  Ex:  psalm, pseudonym, psoriasis, psyche.  In
another example, many Scottish surnames can be spelled with an
initial "Mac" or "Mc".  Thus, "MacKay" and "McKay" are both pronounced
the same.

Accommodation can be made for words that are not spelled as they
sound by making additional entries into the virtual table for the
same word, but adding an alternative spelling in the "soundslike"
column.  For example, the canonical entry for "psalm" would be this:


  INSERT INTO demo(word) VALUES('psalm');


To enhance the ability to correct the spelling of "salm" into
"psalm", make an addition entry like this:


  INSERT INTO demo(word,soundslike) VALUES('psalm','salm');


It is ok to make multiple entries for the same word as long as
each entry has a different soundslike value.  Note that if no
soundslike value is specified, the soundslike defaults to the word
itself.

Listed below are some cases where it might make sense to add additional
soundslike entries.  The specific entries will depend on the application
and the target language.


Silent "p" in words beginning with "ps":  psalm, psyche
Silent "p" in words beginning with "pn":  pneumonia, pneumatic
Silent "p" in words beginning with "pt":  pterodactyl, ptolemaic
Silent "d" in words beginning with "dj":  djinn, Djikarta
Silent "k" in words beginning with "kn":  knight, Knuthson
Silent "g" in words beginning with "gn":  gnarly, gnome, gnat
"Mac" versus "Mc" beginning Scottish surnames
"Tch" sounds in Slavic words:  Tchaikovsky vs. Chaykovsky
The letter "j" pronounced like "h" in Spanish:  LaJolla
Words beginning with "wr" versus "r":  write vs. rite
Miscellaneous problem words such as "debt", "tsetse",
      "Nguyen", "Van Nuyes".


c2: The Spellfix1 Virtual Table
c3: 7. Auxiliary Functions
c5: spellfix1.html#auxiliary_functions
c4:
The source code module that implements the spellfix1 virtual table also
implements several SQL functions that might be useful to applications
that employ spellfix1 or for testing or diagnostic work while developing
applications that use spellfix1.  The following auxiliary functions are
available:


editdist3(P,W)editdist3(P,W,L)editdist3(T)
These routines provide direct access to the version of the Wagner
edit-distance function that allows for application-defined weights
on edit operations.  The first two forms of this function compare
pattern P against word W and return the edit distance.  In the first
function, the langid is assumed to be 0 and in the second, the
langid is given by the L parameter.  The third form of this function
reloads edit distance coefficients from the table named by T.

spellfix1_editdist(P,W)
This routine provides access to the built-in Wagner edit-distance
function that uses default, fixed costs.  The value returned is
the edit distance needed to transform W into P.

spellfix1_phonehash(X)
This routine constructs a phonetic hash of the pure ascii input word X
and returns that hash.  This routine is used internally by spellfix1 in
order to transform the K1 column of the shadow table into the K2
column.

spellfix1_scriptcode(X)
Given an input string X, this routine attempts to determine the dominant
script of that input and returns the ISO-15924 numeric code for that
script.  The current implementation understands the following scripts:

 215 - Latin
 220 - Cyrillic
 200 - Greek

Additional language codes might be added in future releases.

spellfix1_translit(X)
This routine transliterates unicode text into pure ascii, returning
the pure ascii representation of the input text X.  This is the function
that is used internally to transform vocabulary words into the K1
column of the shadow table.


c2: The Spellfix1 Virtual Table
c3: 8. The editdist3 function
c5: spellfix1.html#the_editdist3_function
c4:
The editdist3 algorithm is a function that computes the minimum edit 
distance (a.k.a. the Levenshtein distance) between two input strings.
The editdist3 algorithm is a configurable alternative to the default
edit distance function of spellfix1.
Features of editdist3 include:


It works with unicode (UTF8) text.

A table of insertion, deletion, and substitution costs can be 
       provided by the application.

Multi-character insertions, deletions, and substitutions can be
       enumerated in the cost table.


c2: The Spellfix1 Virtual Table
c3: 9. The editdist3 COST table
c5: spellfix1.html#the_editdist3_cost_table
c4:
To program the costs of editdist3, create a table such as the following:


CREATE TABLE editcost(
  iLang INT,   -- The language ID
  cFrom TEXT,  -- Convert text from this
  cTo   TEXT,  -- Convert text into this
  iCost INT    -- The cost of doing the conversion
);


The cost table can be named anything you want - it does not have to be
called "editcost".  And the table can contain additional columns.
The only requirement is that the
table must contain the four columns show above, with exactly the names shown.

The iLang column is a non-negative integer that identifies a set of costs
appropriate for a particular language.  The editdist3 function will only use
a single iLang value for any given edit-distance computation.  The default
value is 0.  It is recommended that applications that only need to use a
single language always use iLang==0 for all entries.

The iCost column is the numeric cost of transforming cFrom into cTo.  This
value should be a non-negative integer, and should probably be less than 100.
The default single-character insertion and deletion costs are 100 and the
default single-character to single-character substitution cost is 150.  A
cost of 10000 or more is considered "infinite" and causes the rule to be
ignored.

The cFrom and cTo columns show edit transformation strings.  Either or both
columns may contain more than one character.  Or either column (but not both)
may hold an empty string.  When cFrom is empty, that is the cost of inserting
cTo.  When cTo is empty, that is the cost of deleting cFrom.

In the spellfix1 algorithm, cFrom is the text as the user entered it and
cTo is the correctly spelled text as it exists in the database.  The goal
of the editdist3 algorithm is to determine how close the user-entered text is
to the dictionary text.

There are three special-case entries in the cost table:


cFromcToMeaning
'''?'The default insertion cost
'?'''The default deletion cost
'?''?'The default substitution cost


If any of the special-case entries shows above are omitted, then the
value of 100 is used for insertion and deletion and 150 is used for
substitution.  To disable the default insertion, deletion, and/or substitution
set their respective cost to 10000 or more.

Other entries in the cost table specific transforms for particular 
characters.
The cost of specific transforms should be less than the default costs, or else
the default costs will take precedence and the specific transforms will never 
be used.

Some example, cost table entries:


INSERT INTO editcost(iLang, cFrom, cTo, iCost)
VALUES(0, 'a', 'ä', 5);


The rule above says that the letter "a" in user input can be matched against
the letter "ä" in the dictionary with a penalty of 5.


INSERT INTO editcost(iLang, cFrom, cTo, iCost)
VALUES(0, 'ss', 'ß', 8);


The number of characters in cFrom and cTo do not need to be the same.  The
rule above says that "ss" on user input will match "ß" with a penalty of 8.


c2: The Spellfix1 Virtual Table
c3: 10. Experimenting with the editcost3() function
c5: spellfix1.html#experimenting_with_the_editcost3_function
c4:
The spellfix1 virtual table
uses editdist3 if the "edit_cost_table=TABLE" option
is specified as an argument when the spellfix1 virtual table is created.  
But editdist3 can also be tested directly using the built-in "editdist3()"
SQL function.  The editdist3() SQL function has 3 forms:


 editdist3('TABLENAME');
 editdist3('string1', 'string2');
 editdist3('string1', 'string2', langid);


The first form loads the edit distance coefficients from a table called
'TABLENAME'.  Any prior coefficients are discarded.  So when experimenting
with weights and the weight table changes, simply rerun the single-argument
form of editdist3() to reload revised coefficients.  Note that the 
edit distance
weights used by the editdist3() SQL function are independent from the
weights used by the spellfix1 virtual table.

The second and third forms return the computed edit distance between strings
'string1' and "string2'.  In the second form, a language id of 0 is used.
The language id is specified in the third form.


c2: The DBSTAT Virtual Table
c3: 1. Overview
c5: dbstat.html#overview
c4:
The DBSTAT virtual table is a read-only eponymous virtual table that returns
information about the amount of disk space used to store the content
of an SQLite database.
Example use cases for the
DBSTAT virtual table include the sqlite3_analyzer.exe
utility program and the
table size pie-chart in
the Fossil-implemented version control system
for SQLite.



The DBSTAT virtual table is available on all 
database connections when SQLite is built using the
SQLITE_ENABLE_DBSTAT_VTAB compile-time option.


The DBSTAT virtual table is an eponymous virtual table, meaning
that is not necessary to run CREATE VIRTUAL TABLE to create an
instance of the dbstat virtual table before using it.  The "dbstat"
module name can be used as if it were a table name to query the
dbstat virtual table directly.  For example:

SELECT * FROM dbstat;



If a named virtual table that uses the dbstat module is desired,
then the recommended way to create an instance of the dbstat
virtual table is as follows:

CREATE VIRTUAL TABLE temp.stat USING dbstat(main);



Note the "temp." qualifier before the virtual table name ("stat").  This
qualifier causes the virtual table to be temporary - to only exist for
the duration of the current database connection.  This is the
recommended approach.


The "main" argument to dbstat is default schema
for which information is to be provided.  The default is "main", and
so the use of "main" in the example above is redundant.  For any
particular query, the schema can be changed by specifying the
alternative schema as a function argument to the virtual table
name in the FROM clause of the query.  (See further discussion of
table-valued functions in the FROM clause for more details.)


The schema for the DBSTAT virtual table looks like this:
CREATE TABLE dbstat(
  name       TEXT,        -- Name of table or index
  path       TEXT,        -- Path to page from root
  pageno     INTEGER,     -- Page number, or page count
  pagetype   TEXT,        -- 'internal', 'leaf', 'overflow', or NULL
  ncell      INTEGER,     -- Cells on page (0 for overflow pages)
  payload    INTEGER,     -- Bytes of payload on this page or btree
  unused     INTEGER,     -- Bytes of unused space on this page or btree
  mx_payload INTEGER,     -- Largest payload size of all cells on this row
  pgoffset   INTEGER,     -- Byte offset of the page in the database file
  pgsize     INTEGER,     -- Size of the page, in bytes
  schema     TEXT HIDDEN, -- Database schema being analyzed
  aggregate  BOOL HIDDEN  -- True to enable aggregate mode
);



The DBSTAT table only reports on the content of btrees within the database file.
Freelist pages, pointer-map pages, and the lock page are omitted from
the analysis.


By default, there is a single row in the DBSTAT table for each
btree page the database file.  Each row provides
information about the space utilization of that one page of the
database.  However, if the hidden column "aggregate" is TRUE, then
results are aggregated and there is a single row in the DBSTAT table
for each btree in the database, providing information about space
utilization across the entire btree.


c2: The DBSTAT Virtual Table
c3: 2. The "path" column of the dbstat virtual table
c5: dbstat.html#the_path_column_of_the_dbstat_virtual_table
c4:
The "path" column describes the path taken from the 
root node of the btree structure to each page.  The
"path" of the root node itself is '/'.
The "path" is NULL when "aggregate" is TRUE.

The "path" for the left-most child page of the root of
a btree page is '/000/'. (Btrees store content ordered from left to right
so the pages to the left have smaller keys than the pages to the right.)
The next to left-most child of the root page is '/001', and so on,
each sibling page identified by a 3-digit hex  value.
The children of the 451st left-most sibling have paths such
as '/1c2/000/, '/1c2/001/' etc.

Overflow pages are specified by appending a '+' character and a 
six-digit hexadecimal value to the path to the cell they are linked
from. For example, the three overflow pages in a chain linked from 
the left-most cell of the 450th child of the root page are identified
by the paths:

'/1c2/000+000000'         // First page in overflow chain
'/1c2/000+000001'         // Second page in overflow chain
'/1c2/000+000002'         // Third page in overflow chain



If the paths are sorted using the BINARY collation sequence, then
the overflow pages associated with a cell will appear earlier in the
sort-order than its child page:

'/1c2/000/'               // Left-most child of 451st child of root


c2: The DBSTAT Virtual Table
c3: 3. Aggregated Data
c5: dbstat.html#aggregated_data
c4:
Beginning with SQLite version 3.31.0 (2020-01-22), the DBSTAT table
has a new hidden column named "aggregate", which if constrained to be
TRUE will cause DBSTAT to generate one row per btree in the database,
rather than one row per page.  When running in aggregated mode, the 
"path", "pagetype", and "pgoffset" columns are always NULL and the
"pageno" column holds the number of pages in the entire btree, rather
than the number of the page that corresponds to the row.


The following table shows the meanings of the (non-hidden) columns of
DBSTAT in both normal and aggregated mode:




Column
Normal meaning
Aggregate-mode meaning


name

The name of the table or index that is implemented by
the btree of the current row

path
See description above
Always NULL

pageno
The page number of the database page for the current row
The total number of pages in the btree for the current row

pagetype
'leaf' or 'interior'
Always NULL

ncell
Number of cells on the current page or btree

payload
Bytes of useful payload on the current page or btree

unused
Unused bytes of on the current page or btree

mx_payload
The largest payload found anywhere in the current page
or btree.

pgoffset
Byte offset to the start of the page
Always NULL

pgsize
Total storage space used by the current page or btree.


c2: The DBSTAT Virtual Table
c3: 4. Example uses of the dbstat virtual table
c5: dbstat.html#example_uses_of_the_dbstat_virtual_table
c4:
To find the total number of pages used to store table "xyz" in schema "aux1",
use either of the following two queries (the first is the traditional way,
and the second shows the use of the aggregated feature):

SELECT count(*) FROM dbstat('aux1') WHERE name='xyz';
SELECT pageno FROM dbstat('aux1',1) WHERE name='xyz';



To see how efficiently the content of a table is stored on disk,
compute the amount of space used to hold actual content divided
by the total amount of disk space used.  The closer this number
is to 100%, the more efficient the packing.  (In this example, the
'xyz' table is assumed to be in the 'main' schema.  Again, there
are two different versions that show the use of DBSTAT both without
and with the new aggregated feature, respectively.)

SELECT sum(pgsize-unused)*100.0/sum(pgsize) FROM dbstat WHERE name='xyz';
SELECT (pgsize-unused)*100.0/pgsize FROM dbstat
 WHERE name='xyz' AND aggregate=TRUE;



To find the average fan-out for a table, run:

SELECT avg(ncell) FROM dbstat WHERE name='xyz' AND pagetype='internal';




Modern filesystems operate faster when disk accesses are sequential.
Hence, SQLite will run faster if the content of the database file
is on sequential pages.  To find out what fraction of the pages in
a database are sequential (and thus obtain a measurement that might
be useful in determining when to VACUUM), run a query like the following:

CREATE TEMP TABLE s(rowid INTEGER PRIMARY KEY, pageno INT);
INSERT INTO s(pageno) SELECT pageno FROM dbstat ORDER BY path;
SELECT sum(s1.pageno+1==s2.pageno)*1.0/count(*)
  FROM s AS s1, s AS s2
 WHERE s1.rowid+1=s2.rowid;
DROP TABLE s;


c2: The Checksum VFS Shim
c3: 1. Overview
c5: cksumvfs.html#overview
c4:
The checksum VFS extension is a VFS shim that adds an 8-byte
checksum to the end of every page in an SQLite database.  The checksum
is added as each page is written and verified as each page is read.
The checksum is intended to help detect database corruption caused by
random bit-flips in the mass storage device.

The checksum VFS extension requires 
SQLite version 3.32.0 (2020-05-22) or later.  It will not
work with earlier versions of SQLite.


c2: The Checksum VFS Shim
c3: 2. Compiling
c5: cksumvfs.html#compiling
c4:
The checksum VFS module is a loadable extension.  It is not
included in the amalgamation.  It must be added to SQLite
either at compile-time or at run-time.  The source code to
the checksum VFS module is in the
ext/misc/cksumvfs.c
source file in the
SQLite source tree.

To build the checksum VFS module into a run-time loadable
extension, use commands similar to the following:


(linux) &rarr; gcc -fPIC -shared cksumvfs.c -o cksumvfs.so
(mac) &rarr; clang -fPIC -dynamiclib cksumvfs.c -o cksumvfs.dylib
(windows) &rarr; cl cksumvfs.c -link -dll -out:cksumvfs.dll


You may want to add additional compiler options, of course,
according to the needs of your project.

To statically link this extension with your product,
compile it like any other C-language module but add the
"-DSQLITE_CKSUMVFS_STATIC" option so that this module knows that
it is being statically linked rather than dynamically linked.


c2: The Checksum VFS Shim
c3: 3. Loading
c5: cksumvfs.html#loading
c4:
To load this extension as a shared library, you first have to
bring up a dummy SQLite database connection to use as the argument
to the sqlite3_load_extension() API call.  Then you invoke the
sqlite3_load_extension() API and shutdown the dummy database
connection.  All subsequent database connections that are opened
will include this extension.  For example:

sqlite3 *db;
sqlite3_open(":memory:", &db);
sqlite3_load_extension(db, "./cksumvfs");
sqlite3_close(db);


If this extension is compiled with -DSQLITE_CKSUMVFS_STATIC and
statically linked against the application, initialize it using
a single API call as follows:

sqlite3_cksumvfs_init();


Cksumvfs is a VFS shim. When loaded, "cksmvfs" becomes the new
default VFS, and it uses the prior default VFS as the next VFS
down in the stack.  This is normally what you want.  However, in
complex situations where multiple VFS shims are being loaded,
it might be important to ensure that cksumvfs is loaded in the
correct order so that it sequences itself into the default VFS
Shim stack in the right order.


c2: The Checksum VFS Shim
c3: 4. Usage
c5: cksumvfs.html#usage
c4:
Open database connections using the sqlite3_open() or 
sqlite3_open_v2() interfaces, as normal.  Ordinary database files
(without a checksum) will operate normally.  Databases with 
checksums will return an SQLITE_IOERR_DATA error if a page is
encountered that contains an invalid checksum.

Checksumming only works on databases that have a reserve bytes
value of exactly 8.  The default value for reserve-bytes is 0.
Hence, newly created database files will omit the checksum by
default.  To create a database that includes a checksum, change
the reserve-bytes value to 8 by running code similar to this:

int n = 8;
sqlite3_file_control(db, 0, SQLITE_FCNTL_RESERVED_BYTES, &n);


If you do this immediately after creating a new database file,
before anything else has been written into the file, then that
might be all that you need to do.  Otherwise, the API call
above should be followed by:

sqlite3_exec(db, "VACUUM", 0, 0, 0);


It never hurts to run the VACUUM, even if you don't need it.
If the database is in WAL mode, you should shutdown and
reopen all database connections before continuing.

From the CLI, use the ".filectrl reserve_bytes 8" command, 
followed by "VACUUM;".

Note that SQLite allows the number of reserve-bytes to be
increased but not decreased.  So if a database file already
has a reserve-bytes value greater than 8, there is no way to
activate checksumming on that database, other than to dump
and restore the database file.  Note also that other extensions
might also make use of the reserve-bytes.  Checksumming will
be incompatible with those other extensions.


