aboutsummaryrefslogtreecommitdiffstats
path: root/camel/devel-docs
diff options
context:
space:
mode:
Diffstat (limited to 'camel/devel-docs')
-rw-r--r--camel/devel-docs/camel-index.txt407
1 files changed, 407 insertions, 0 deletions
diff --git a/camel/devel-docs/camel-index.txt b/camel/devel-docs/camel-index.txt
new file mode 100644
index 0000000000..0ebc250db0
--- /dev/null
+++ b/camel/devel-docs/camel-index.txt
@@ -0,0 +1,407 @@
+
+The history of text-indexing in evolution.
+
+CamelTextIndex was written to address several shortcomings in the
+existing libibex (referred to as libibex2), which had been written to
+address shortcomings in the original libibex.
+
+Mail indexing characteristics
+
+First, i'll cover some of the scenarios that a mail indexing system
+must cover. They are slightly different from other indexing systems,
+at least we wanted them to be.
+
+1. Indexing a few new messages, they may potentially reference most of
+ the alphabet in the index.
+2. Indexing a whole mailbox for the first time
+3. Unindexing anywhere from a few to all existing messages during expunge.
+4. Searching.
+
+Cases 1, 3, and 4 occur the most often, however 2 is the most noticeable
+at first use, or if the index must be reset. So the code needs to
+work fast in all cases, which generally leads to trade-offs being
+made. Each implementation aimed to address or ignore these
+requirements in different ways, with the final implementation probably
+having the best balance so far.
+
+The main issue is that the indexing be performed real time. We index
+as we add the messages. We index before we open the mailbox. We
+index as we remove messages. Because of this we need to approach
+things differently to many other indexing systems; most of which work
+with static data in an off-line mode. This allows them to index the
+whole body of content and use as much memory and cpu time as required.
+
+We probably need to look at doing offline, or at least delayed
+indexing in the future - but this introduces some coherency problems
+with vFolders and any body searches. However, having the indexing
+library a base part of Camel helps in implementing a mechanism to
+achieve this.
+
+Ibex the first
+
+The original ibex used a memory-based hash table to store words. This made
+the index very fast for both lookups and modifications. However any
+queries required a full load of the index into memory, and any updates
+required a full write of the index to disk. After about 5-10 000
+messages occasionaly adding to the index became appreciably slower as
+the whole index needed to be loaded into memory first. This obviously
+took a toll on memory as well.
+
+I wont cover the algorithms used, they were all pretty basic, the only
+real smarts were that index deletes were only flagged and that data
+not written to disk when the index saved.
+
+Evolution 1.x, ibex 2.
+
+In an attempt to rectify the incremental update performance of
+libibex, it was completely rewritten to use an on-disk block-based
+filesystem.
+
+Note: the first attempt used libdb - however performance was so slow
+and the indices were so large it was dropped in favour of a custom
+filesystem-like data file.
+
+The motivation was that a few extra disk lookups during
+retrieval wouldn't be noticeably slower, however it should be able to
+scale up to many more messages with lower memory overhead and slower
+startup time.
+
+The block filesystem contains 3 major components:
+
+1. A hash table that mapped message's to a word id sequence list.
+2. A hash table that mapped word's to a message id sequence list.
+3. A sequence filesystem that stored sequences of id's.
+
+The id's are 32 bit identifiers that are unique for each word or
+message. They are also designed to be reversible and static.
+That is, given the id, you can map it to the string identifier that it
+represents directly, without having to look it up in the hash table.
+
+Other features of this design is that the database file should be
+kept in sync at all times with the state of the index. The message to
+wordid tables are used to remove the messageid's from the word's it
+contains when the message is expunged, and so on.
+
+Indexing operation
+
+The indexing operation consists of the basic steps:
+
+1. Lookup the messageid from the message name, using the messageid table.
+2. Generate a list of words in the message
+3. For each word:
+4. Lookup the wordid and sequence information
+5. If the word doesn't exist, create a new word/wordid
+6. Add the messageid to the word sequence.
+7. Add the wordid to the message sequence.
+
+The initial implementation only used caching at the disk-block level.
+Unfortunately, the simple hash table design chosen (fixed sized base
+table with chained buckets) scaled very poorly above about 10 000
+messages. So this approach proved to be too i/o intensive for
+practical use, and several other caches were added to improve
+performance:
+
+1. Stage (1) above is done entirely in memory. At initial startup
+ the whole list of potential names is read into an in-memory hash
+ table.
+2. Stage (4) above is also done entirely in memory. Even a large
+ cache provided little benefit due to wide distribution of potential
+ words. This cache is only created when adding to the index.
+3. Stage (6) uses the table from stage (4) and concatenates upto
+ approximately one disk blocks worth of messageid's before writing
+ them out to the word sequence.
+4. Stage (7) concatenates all wordid's for a given message before
+ writing them out at once.
+
+As you can see, the added complexity meant we nearly have to cache as
+much as the original version! This also almost removed all of the
+startup-time benefit for incremental update of the index, as the table
+was not stored as compactly on disk as the original version.
+
+However, we only ever stored a subset of the index in memory, and only
+during updates, with some tricks to reduce memory usage for very rare
+words, so the overall memory use was still much lower.
+
+Removing a message
+
+Removing a message is fairly involved:
+
+1. Lookup the messageid and word sequence list from the messageid table.
+2. For each wordid in the sequence list
+3. Lookup the message sequence list directly from the wordid table.
+4. Scan each block in the sequence, and remove any instances of the
+ messageid.
+5. Remove the message to messageid mapping in the messageid table.
+
+Unfortunately caching helped very little here, particularly if many
+messages were removed. Also note that the file could never shrink as
+the data could be spread randomly over it. Removal is an extremely
+expensive an unbounded process. Deleting all of the messages in a
+mailbox is extremely i/o intensive, with blocks potentially being
+accessed dozens of times.
+
+Performing a query
+
+Performing a query is fast:
+
+1. Lookup the messageid sequence list from the wordid table.
+2. For each messageid
+3. Lookup the message name directly from the messageid table.
+
+Even without caching this performs at a very acceptable level.
+
+Summary
+
+This index performs reasonably well upto about 10 000 messages for a
+complete re-index. However with incremental updates it degrads much
+faster, only a few thousand messages added and it becomes tiresomely
+slow and i/o bound. The index becomes more fragmented with random
+updates and removals and heavily bogs down the system as you go much
+beyond those few thousand messages.
+
+The code is also very complicated and hard to follow. There are too
+many special cases, and it is buggy. Detected on-disk structure
+errors result in the index being reset, which although it shrinks the
+index, is very slow.
+
+The indices created are bulky, and never shrink. Because of the
+reverse index used for message removal, there is 50% redundant data at
+all times. Some overly tricky techniques (very much like ReiserFS's
+tail packing) are used to waste as little space as possible, with a
+great impact on performance.
+
+One other problem is that because the index is disk based, we
+use a file descriptor continuously. With some users having
+>100 folders, they quickly run out of process file descriptors and
+evolution fails. To get around this a cache of least recently used
+index files is used to flush away and free file descriptors so they
+can be re-used. This makes it hard to lock the files; this problem
+still exists with the next implementation.
+
+Anyway, a better solution is required.
+
+CamelIndex
+
+The first problem to address was the api. It was starting to age.
+Although adequate, the api wasn't terribly clean, reusable, or
+scalable. The first thing was to objectise the library, and since we
+needed to use it in Camel, the best way was to create a CamelObject.
+
+CamelIndex was born. A mostly abstract class that provides a simple
+common interface for accessing indices, including cursors and utility
+and maintenance functions.
+
+In addition, a number of the features in libibex2 were simplified or
+rewritten and abstracted into the re-usable classes that follow.
+
+By providing simple cursors, more complex queries were easier to write
+and can execute more efficiently; camel searching now does sub-string
+searches for all body queries, and still runs at a very healthy speed
+and uses less memory than before.
+
+CamelBlockFile
+
+This is basically the same block filesystem used in libibex2. It
+handles disk i/o based on blocks (CamelBlock), flushing modified
+blocks to disk, and caching of recently accessed blocks. It was
+enhanced slightly to allow blocks to be locked in memory.
+
+CamelKeyFile
+
+This is a simple reverse-linked list of sequences of keyid's.
+
+The main property of this file is that updates are only ever appended
+to the end of the file, which improves i/o characteristics markedly.
+
+When an existing keyid sequence is updated, it simply points back to
+the start of the previous one, and provides a pointer to the new
+entry. i.e. a simple linked list.
+
+CamelKeyTable
+
+This is taken from the libibex2 code for mapping keys, with few
+changes. It uses a CamelBlockFile for its i/o.
+
+The key table is a linked list of blocks (CamelKeyBlock) which contain
+key strings and and a data pointer and flags for each key. Each block
+is a packed array of string descriptors (CamelKeyKey's).
+
+A keyid (camel_key_t) is a 32 bit descriptor which identifies this key
+in a reversible way. In this case the bottom 10 bits are used to
+identify the index of the key within the key block, and the top 22
+bits are used to identify the key block itself. In this way, given
+the 32 bit key id, we can reference the block containing the key
+directly (with at most 1 access), and access the flags and key string
+using the key index.
+
+Keys can potentially be removed and their keyid's reused by simply
+re-packing the key block. This was used in libibex2, but not in
+CamelIndex.
+
+[diagram - camelkeyblock]
+
+CamelPartitionTable
+
+An implementation of a scalable, on-disk 'perfect' hash table. It
+uses the CamelBlockFile to handle its i/o. This is a completely new
+hash table implementation which was not present in libibex2.
+
+[FIXME: Reference the original paper the algorithm is based on.]
+
+A partition table consists of a list of mapping blocks
+(CamelPartitionMapBlock), which is a compact table that maps a range
+of hashid's to a partition block (CamelPartitionKeyBlock), which
+contains hashid's of that range.
+
+[diagram - camelpartitiontable]
+
+The partition block only maps the hashid to a keyid (see CamelKeyTable)
+which means it can store a lot of keys in each block.
+
+To add a new value to the partition table:
+
+1. Calculate the hash value of the key
+2. Find out which partition block the key will fit into, using the
+ partition table.
+3. If the partition block is full:
+4. If there is room in the next or previous block:
+5. Merge the 2 blocks together, and split at the half-way point
+6. Update the partition table hash indices to match the blocks
+7. Else
+8. Create a new block, and split the existing block across it
+9. Insert the new block into the partition table
+10. Else
+11. Just add the key to the end of the block.
+
+Steps 5 and 8 perform a sorting of the partition key entries by hashid
+to find the midpoint. It may be beneficial to store the hashid's
+sorted always, it would then not require a sort to split the blocks.
+This would also benefit key lookups by being able to use a binary
+search. However, the incremental sort may be more expensive.
+
+If the partition table itself fills up, then perform a similar
+splitting function on its blocks, and store it over multiple blocks.
+With a block size of 1024 bytes, we can fit 127 blocks pointers, each
+with 127 keys in it - around 16000 keys. So we only need 1024 bytes
+of memory for each 16000 on-disk keys (assuming full tables).
+
+Removal is basically the same, but if we end up with an empty block we
+just remove it from the partition table. CamelTextIndex doesn't
+actually use removal although it is implemented in
+CamelPartitionTable.
+
+Lookup is very simple. We basically follow steps 1 and 2, and then
+perform a linear search through the block to find a matching hash id.
+That is our key. This is assuming a perfect hash, additionally the
+code could use the keyid to lookup in a keytable to verify the key is
+indeed the right one. This would require having to support duplicate
+hashid's and would make block splitting slightly more complex, but
+only by a couple of lines of code. This is something that will
+probably have to be addressed in the future.
+
+Using a partition table means that we can tell with 1 disk access
+whether or not a key exists (assuming a perfect hash function), and 1
+more access to look up all of the details of the key since the keyid
+is reversible. Another feature is that the partition table is always
+self-balancing for any data processed in any order.
+
+Yet one more feature is that it is quite easy to order the writes to
+the partition table so that its structure is always consistent, even
+in the event of program failure. Although this has been disabled in
+the current code to take maximal advantage of the block cache.
+
+CamelTextIndex
+
+CamelTextIndex is the implementation of CamelIndex now used by camel
+for indexing mail. It shares some features with the second
+incarnation of libibex, but is generally simpler. It uses the
+previously described classes to implement the CamelIndex interface.
+
+Indexing operation
+
+Indexing operation is similar to libibex2, but without the requirement
+to maintain the reverse index.
+
+1. Lookup the messageid from the message name, using the messageid
+ partition table.
+2. Generate a list of words in the message
+3. For each word
+4. Lookup the wordid and sequence information.
+5. Append the messageid to the word sequence.
+
+In practice we also have a word cache which caches upto 32 messageid's
+for each word before it is written to the key file.
+
+Removing a message
+
+Removal is not immediate. This is one of the major performance
+improvements in CamelIndex.
+
+1. Lookup the messageid from the message name partition table
+2. Use the messageid to set a flag in the message key table to
+ indicate the message has been deleted.
+3. Remove the key hash from the partition table.
+
+This comes down to a maximum of 2 disk reads and 2 disk writes.
+libibex2 had unbounded maximums, depending on the number of words in a
+given message. The key file is not changed.
+
+Because data is not removed from the files at all, an additional
+optional step is required, that of compressing the indices.
+
+Performing a query
+
+Performing a query is much the same as with libibex2. We usually have
+slightly less disk i/o because of a more efficient and scalable hash
+table implementation, and improved locality of reference of the key
+table data.
+
+1. Lookup the messageid from the message name partition table
+2. Use the messageid to get the data pointer directly from the key
+ table.
+3. Iterate through the key file, reading blocks backwards through the
+ file.
+
+Compressing
+
+Although it could have benefited from it, libibex2 did not ever
+compress indices - the only way to compress an index was to remove it
+and have it be rebuilt.
+
+CamelIndex requires a compression stage as data is never removed from
+it otherwise. Because of the much greater locality of reference, the
+compression stage is actually much faster than an incremental removal
+of data inside the data files.
+
+Compressing comprises the following steps:
+
+1. Open a new temporary index, an index block file and an index key
+ file.
+2. For each message in the message partition table
+3. If the message is not marked deleted, add it to the new message
+ partition table, and recored the old messageid to new messageid
+ mapping.
+4. For each word in the word partition table
+5. For each messageid's in the word sequence list
+6. If the messageid maps to a new messageid, remap the messageid,
+ else discard it.
+7. Concatenate upto 256 messageid's in a row before writing to the
+ key file, to improve lookups.
+8. Create a new word in the new word key table
+9. Add the wordid and new sequence id to the word partition table.
+
+Note that at step 8 we could (should?) also check if the word has any
+messages associated with it, and discard the word from the new index.
+
+After compression, the name partition index only contains names which
+are not deleted, and the key file is compressed into larger blocks
+which takes up less space and is faster to retrieve.
+
+During index operations a number of statistics are taken which trigger
+an automatic compress when the file fragmentation or number of deleted
+messages exceed a threshold. So the index maintains itself, and does
+not need manual compression.
+
+
+
+