Sindbad~EG File Manager
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<title>External File support</title>
<link rel="stylesheet" href="gettingStarted.css" type="text/css" />
<meta name="generator" content="DocBook XSL Stylesheets V1.73.2" />
<link rel="start" href="index.html" title="Berkeley DB Programmer's Reference Guide" />
<link rel="up" href="am_misc.html" title="Chapter 4. Access Method Wrapup" />
<link rel="prev" href="am_misc_diskspace.html" title="Disk space requirements" />
<link rel="next" href="am_misc_tune.html" title="Access method tuning" />
</head>
<body>
<div xmlns="" class="navheader">
<div class="libver">
<p>Library Version 18.1.40</p>
</div>
<table width="100%" summary="Navigation header">
<tr>
<th colspan="3" align="center">External File support</th>
</tr>
<tr>
<td width="20%" align="left"><a accesskey="p" href="am_misc_diskspace.html">Prev</a> </td>
<th width="60%" align="center">Chapter 4. Access Method Wrapup </th>
<td width="20%" align="right"> <a accesskey="n" href="am_misc_tune.html">Next</a></td>
</tr>
</table>
<hr />
</div>
<div class="sect1" lang="en" xml:lang="en">
<div class="titlepage">
<div>
<div>
<h2 class="title" style="clear: both"><a id="blobs"></a>External File support</h2>
</div>
</div>
</div>
<div class="toc">
<dl>
<dt>
<span class="sect2">
<a href="blobs.html#blob_threshold">The external file threshold</a>
</span>
</dt>
<dt>
<span class="sect2">
<a href="blobs.html#blob_create">Creating external files</a>
</span>
</dt>
<dt>
<span class="sect2">
<a href="blobs.html#blob_access">External file access</a>
</span>
</dt>
<dt>
<span class="sect2">
<a href="blobs.html#blob_storage">External file storage</a>
</span>
</dt>
<dt>
<span class="sect2">
<a href="blobs.html#blob_replication">External Files and Replication</a>
</span>
</dt>
</dl>
</div>
<p>
External file support is designed for
efficient storage of large objects. An object is considered to
be large if it is more than a third of the size of a page.
Without external file support, large objects must be broken up into
smaller pieces, and then reassembled and/or disassembled every
time the record is read or updated. Berkeley DB external file support
avoids this assembly/disassembly process by storing the large
object in a special directory set aside for the purpose. The
data itself is not kept in the database, nor is it placed into
the in-memory cache.
</p>
<p>
external files can only be stored using the data portion of a
key/data pair. They are supported only for Btree, Hash, and
Heap databases, and only so long as the database is not
configured for duplicate records, or
duplicate sorted records. In addition, the DBT that you use to
access the external file data cannot be configured as a partial DBT if
you want to access the data using the external file's streaming
interface (introduced below).
</p>
<p>
Note that if the environment is transactionally-protected,
then all access to the external file is also transactionally protected.
</p>
<div class="sect2" lang="en" xml:lang="en">
<div class="titlepage">
<div>
<div>
<h3 class="title"><a id="blob_threshold"></a>The external file threshold</h3>
</div>
</div>
</div>
<p>
The external file threshold is a positive integer, in bytes,
which indicates how large an object must be before it is
considered an external file. By default, the external file
threshold for any given database is <code class="literal">0</code>, which
means that no object will ever be considered an external file. This
means that the external file feature is not used by default for
Berkeley DB databases.
</p>
<p>
In order to use the external file feature, you must set the
external file threshold to a non-zero, positive integer value. You
do this for a given database using the <a href="../api_reference/C/set_blob_threshold.html" class="olink">DB->set_blob_threshold()</a>
method. Note that this value must be set before you create
the database. At any point after database creation time,
this method is ignored.
</p>
<p>
In addition, if you are using an environment, you can
change the default threshold for databases created in that
environment to something other than <code class="literal">0</code>
by using the <a href="../api_reference/C/envset_blob_threshold.html" class="olink">DB_ENV->set_blob_threshold()</a> method.
</p>
<p> You can retrieve the external file threshold set for a database
using the <a href="../api_reference/C/get_blob_threshold.html" class="olink">DB->get_blob_threshold()</a>. You can retrieve the
default external file threshold set for your environment using the
<a href="../api_reference/C/envget_blob_threshold.html" class="olink">DB_ENV->get_blob_threshold()</a>.
</p>
</div>
<div class="sect2" lang="en" xml:lang="en">
<div class="titlepage">
<div>
<div>
<h3 class="title"><a id="blob_create"></a>Creating external files</h3>
</div>
</div>
</div>
<p>
There are two ways to create an external file. Before you can use
either mechanism, you must set the external file threshold to a
non-zero positive integer value (see the previous section
for details). Once the external file threshold has been set, you
create an external file using one of the two following mechanisms:
</p>
<div class="itemizedlist">
<ul type="disc">
<li>
<p>
Configure the DBT used to access the external file data
(that is, the DBT used for the data portion of the
record) with the <a href="../api_reference/C/dbt.html#dbt_DB_DBT_BLOB" class="olink">DB_DBT_BLOB</a> flag. This causes
the data to be stored as an external file regardless of its
size, so long as the database otherwise supports
external files.
</p>
</li>
<li>
<p>
Alternatively, creating a data item with a size
greater than the external file threshold will cause that
data item to be automatically stored as an external file.
</p>
</li>
</ul>
</div>
</div>
<div class="sect2" lang="en" xml:lang="en">
<div class="titlepage">
<div>
<div>
<h3 class="title"><a id="blob_access"></a>External file access</h3>
</div>
</div>
</div>
<p>
external files may be accessed in the same way as other DBT
data, so long as the data itself will fit into memory.
More likely, you will find it necessary to use the external file
streaming API to read and write external file data. You open an
external file stream using the <a href="../api_reference/C/dbstream.html" class="olink">DBC->db_stream()</a> method, close it with the
<a href="../api_reference/C/dbstream_close.html" class="olink">DB_STREAM->close()</a> method, write to it using the the
<a href="../api_reference/C/dbstream_write.html" class="olink">DB_STREAM->write()</a> method, and read it using the
<a href="../api_reference/C/dbstream_read.html" class="olink">DB_STREAM->read()</a> method.
</p>
<p>
The following example code fragment can be found in
your DB distribution at
<code class="filename">.../db/examples/c/ex_external_file.c</code>.
</p>
<pre class="programlisting">...
/* Some necessary variable declarations */
DBC *dbc; /* Cursor handle */
DB_ENV *dbenv; /* Environment handle */
DB *dbp; /* Database handle */
DB_STREAM *dbs; /* Stream handle */
DB_TXN *txn; /* Transaction handle */
DBT data, key; /* DBT handles */
int ret;
db_off_t size;
...
/* Environment creation skipped for brevity's sake */
...
/* Enable external files and set the size threshold. */
if ((ret = dbenv->set_ext_file_threshold(dbenv, 1000, 0)) != 0) {
dbenv->err(dbenv, ret, "set_ext_file_threshold");
goto err;
}
...
/* Database and DBT creation skipped for brevity's sake */
...
/*
Access the external file using the DB_STREAM API.
*/
if ((ret = dbenv->txn_begin(dbenv, NULL, &txn, 0)) != 0){
dbenv->err(dbenv, ret, "txn");
goto err;
}
if ((ret = dbp->cursor(dbp, txn, &dbc, 0)) != 0) {
dbenv->err(dbenv, ret, "cursor");
goto err;
}
/*
* Set the cursor to an external file. Use DB_DBT_PARTIAL with
* dlen == 0 to avoid getting any external file data.
*/
data.flags = DB_DBT_USERMEM | DB_DBT_PARTIAL;
data.dlen = 0;
if ((ret = dbc->get(dbc, &key, &data, DB_FIRST)) != 0) {
dbenv->err(dbenv, ret, "Not an external file");
goto err;
}
data.flags = DB_DBT_USERMEM;
/* Create a stream on the external file the cursor points to. */
if ((ret = dbc->db_stream(dbc, &dbs, DB_STREAM_WRITE)) != 0) {
dbenv->err(dbenv, 0, "Creating stream.");
goto err;
}
/* Get the size of the external file. */
if ((ret = dbs->size(dbs, &size, 0)) != 0) {
dbenv->err(dbenv, 0, "Stream size.");
goto err;
}
/* Read from the external file. */
if ((ret = dbs->read(dbs, &data, 0, (u_int32_t)size, 0)) != 0) {
dbenv->err(dbenv, 0, "Stream read.");
goto err;
}
/* Write data to the external file, increasing its size. */
if ((ret = dbs->write(dbs, &data, size/2, 0)) != 0) {
dbenv->err(dbenv, 0, "Stream write.");
goto err;
}
/* Close the stream. */
if ((ret = dbs->close(dbs, 0)) != 0) {
dbenv->err(dbenv, 0, "Stream close.");
goto err;
}
dbs = NULL;
dbc->close(dbc);
dbc = NULL;
txn->commit(txn, 0);
txn = NULL;
free(data.data);
data.data = NULL;
...
/* Handle clean up skipped. */ </pre>
</div>
<div class="sect2" lang="en" xml:lang="en">
<div class="titlepage">
<div>
<div>
<h3 class="title"><a id="blob_storage"></a>External file storage</h3>
</div>
</div>
</div>
<p>
external files are not stored in the normal database files on
disk in the same way as is other data managed by DB.
Instead, they are stored as binary files in a special
directory set aside for the purpose.
</p>
<p>
If you are not using environments, this special external file
directory is created relative to the current working
directory from which your application is running. You can
modify this default location using the <a href="../api_reference/C/set_blob_dir.html" class="olink">DB->set_blob_dir()</a>
method, and retrieve the current external file directory using
<a href="../api_reference/C/get_blob_dir.html" class="olink">DB->get_blob_dir()</a>.
</p>
<p>
If you are using an environment, then by default the
external file directory is created within the environment's home
directory. You can change this default location using
<a href="../api_reference/C/envset_blob_dir.html" class="olink">DB_ENV->set_blob_dir()</a> and retrieve the current default
location using <a href="../api_reference/C/envget_blob_dir.html" class="olink">DB_ENV->get_blob_dir()</a>. (Note that
<a href="../api_reference/C/envget_blob_dir.html" class="olink">DB_ENV->get_blob_dir()</a> can successfully retrieve the external file
directory only if <a href="../api_reference/C/envset_blob_dir.html" class="olink">DB_ENV->set_blob_dir()</a> was previously
called.)
</p>
<p>
Note that because external files are stored outside of the
Berkeley DB database files, they are not confined by the
four gigabyte limit used for Berkeley DB key and data
items. The external file size limit is system dependent. It can be
the maximum value in bytes of a signed 32 bit integer (if
the Berkeley DB-defined type <code class="literal">db_off_t</code>
is four bytes in size), or a signed 64 bit integer (if
<code class="literal">db_off_t</code> is eight bytes in size).
</p>
</div>
<div class="sect2" lang="en" xml:lang="en">
<div class="titlepage">
<div>
<div>
<h3 class="title"><a id="blob_replication"></a>External Files and Replication</h3>
</div>
</div>
</div>
<p>
Replication supports external files without any special
requirements. However, enabling external files in a replicated
environment can result in long synchronization times between the
client and master sites. To avoid this, execute a transaction
checkpoint after updating or deleting one or more external file
records.
</p>
</div>
</div>
<div class="navfooter">
<hr />
<table width="100%" summary="Navigation footer">
<tr>
<td width="40%" align="left"><a accesskey="p" href="am_misc_diskspace.html">Prev</a> </td>
<td width="20%" align="center">
<a accesskey="u" href="am_misc.html">Up</a>
</td>
<td width="40%" align="right"> <a accesskey="n" href="am_misc_tune.html">Next</a></td>
</tr>
<tr>
<td width="40%" align="left" valign="top">Disk space
requirements </td>
<td width="20%" align="center">
<a accesskey="h" href="index.html">Home</a>
</td>
<td width="40%" align="right" valign="top"> Access method tuning</td>
</tr>
</table>
</div>
</body>
</html>
Sindbad File Manager Version 1.0, Coded By Sindbad EG ~ The Terrorists