diff options
| -rw-r--r-- | devel-docs/query/virtual-folder-in-depth.txt | 306 |
1 files changed, 306 insertions, 0 deletions
diff --git a/devel-docs/query/virtual-folder-in-depth.txt b/devel-docs/query/virtual-folder-in-depth.txt new file mode 100644 index 0000000000..fc7b80ca07 --- /dev/null +++ b/devel-docs/query/virtual-folder-in-depth.txt @@ -0,0 +1,306 @@ +* introduction + +This document describes a different way of approaching mail +organization and how all things are possible in this brave new +world. This document does not describe physical storage issues nor +interface issues. + +Historically mail has been organized into folders. These folders +usually mapped to a single storage medium. The relationship between +mail organization and storage medium was one to one. There was one +mail organization for every storage medium. This scheme had its +limitations. + +Efforts at categorizations are only meaningful at the instance that +one categorized. To find any piece of data, regardless of how well +it was categorized, required some amount of searching. Therefore, any +attempts to nullify searching is doomed to fail. It's time to embrace +searching as a way of life. + +These are the terms and their definitions. The example rules used are +based on the syntax for VM (http://www.wonderworks.com/vm/) by Kyle +Jones whose ideas form the basis for this. I'm only adding the +existence of summary files to aid in scaling. I currently use VM and +it's virtual-folder rules for my daily mail purposes. To date, my only +complaints are speed (it has no caches) and for the unitiated, it's +not very user-friendly. + +Comments, questions, rants, etc. should be directed at Giao Nguyen +<grail@cafebabe.org> who will try to address issues in a timely +manner. + +* Definitions + +** store + +A location where mail can be found. This may be a file (Berkeley +mbox), directory (MH), IMAP server, POP3 server, Exchange server, +Lotus Notes server, a stack of Post-Its by your monitor fed through +some OCR system. + +** message + +An individual mail message. + +** vfolder + +A group of messages sharing some commonality. This is the result of a +query. The vfolder maybe contained in a store, but it is not necessary +that a store holds only one vfolder. There is always an implicit +vfolder rule which matches all messages. A store contains the vfolder +which is the result of the query (any). It's short for virtual folder +or maybe view folder. I dunno. + +** default-vfolder + +The vfolder defined by (any) applied to the store. This is not the +inbox. The inbox could easily be defined by a query. A default rule +for the inbox could be (new) but it doesn't have to be. Mine happens +to be (or (unread) (new)). + +** folder + +The classical mail folder approach: one message organization per +store. + +** query + +A search for messages. The result of this is a vfolder. There are two +kinds of queries: named queries and lambda queries. More on this +later. + +** summary file + +An external file that contains pointers to messages which are matches +for a named query. In addition to pointers, the summary file should +also contain signatures of the store for sanity checks. When the term +"index" is used as a verb, it means to build a summary file for a +given name-value pair. + +* Queries + +Named queries are analogous to classical mail folders. Because named +queries maybe reused, summary files are kept as caches to reduce +the overall cost of viewing a vfolder. Summary files are superior to +folders in that they allow for the same messages to appear in multiple +vfolders without message duplications. Duplications of messages +defeats attempts at tagging a message with additional user information +like annotations. Named queries will define folders. + +Lambda queries are similar to named queries except that they have no +name. These are created on the fly by the user to filter out or +include certain messages. + +All queries can be layered on top of each other. A lambda query can be +layered on a named query and a named query can be layered on a lambda +query. The possibilities are endless. + +The layerings can be done as boolean operations (and, or, not). Short +circuiting should be used. + +Examples: + +(and (author "Giao") + (unread)) + +The (unread) query should only be evaluated on the results of (author +"Giao"). + +(or (author "Giao") + (unread)) + +Both of these queries should be evaluated. Any matches are added to the +resulting vfolder. + +* Summary files + +Summary files are only meaningful when applied to the context of the +default-vfolder of a store. + +Summary files should be generated for queries of the form: + +(function "constant value") + +Summary files should never be generated for queries of the form: + +(function (function1)) + +(and (function "value") + (another-function "another value")) + +Given a query of the form: + +(and (function "value") + (another-function "another value")) + +The system should use one summary file for (function "value") and +another summary file for (another-function "another value"). I will +call the prior form the "plain form". + +It should be noted that the signature of the store should be based on +the assumption that new data may have been added to the store since +the application generated the summary file. Signatures generated on +the entirety of the store will most likely be meaningless for things +like POP/IMAP servers. + +* Incremental indexing + +When new messages are detected, all known queries should be evaluated +on the new messages. vfolders should be notified of new messages that +are positive matches for their queries. The indexes generated by this +process should be merged into the current indexes for the vfolder. + +* Can I have multiple stores? + +I don't see why not. Again, the inbox is a vfolder so you can get a +unified inbox consisting of all new mail sent to all your stores or +your can get inboxes for each store or any combination your heart +desire. You get your cake, eat it, and someone else cleans the dishes! + +* Why all this? + +Consider the dynamic nature of the following query: + +(and (author "Giao") + (sent-after (today-midnight))) + +today-midnight would be a function that is evaluated at run-time to +calculate the appropriate object. + +* Scenarios of usage and their solutions + +** Mesage alterations + +This is a fuzzy area that should be left to the UI to handle. Messages +are altered. Read status are altered when a new message is read for +example. How do we handle this if our query is for unread messages? +Upon viewing the state would change. + +One idea is to not evaluate the queries unless we're changing between +vfolder views. This assumes that one can only view a particular +vfolder at a time. For multi-vfolder viewing, a message change should +propagate through the vfolder system. Certain effects (as in our +example) would not be intuitive. + +It would not be a clean solution to make special cases but they may be +necessary where certain defined fields are ignored when they are +changed. Some combination of the above rules can be used. I don't +think it's an easy solution. + +** Message inclusion and exclusion + +Messages are included and excluded also with queries. The final query +will have the form of: + +(and (author "Giao") + (criteria value) + (not (criteria other-value))) + +Userland criterias may be a label of some sort. These may be userland +labels or Message-IDs. What are the performance issues involved in +this? With short circuiting, it's not a major problem. + +The criterias and values are determined by the UI. The vfolder +mechanism isn't concerned with such issues. + +Messages can be included and excluded at will. The idea is often +called "arbitrary inclusion/exclusion". This can be done by +Message-IDs or other fields. It's been noted that Message-IDs are not +unique. + +I propose that any given vfolder is allocated an inclusion label and an +exclusion label. These should be randomly generated. This should be +part of the vfolder description. It should be noted that the vfolder +description has not been drafted yet. + +The result is such that the rules for a given named query is: + +(and (user-query) + (label inclusion-label) + (not exclusion-label)) + +** Query scheduling + +Consider the following extremely dynamic queries: + +A: +(and (author "Giao") + (sent-after (today-midnight))) + +B: +(and (sent-after (today-midnight)) + (author "Giao")) + +C: +(or (author "Giao") + (sent-after (today-midnight))) + +Query A would be significantly faster because (author "Giao") is not +dynamic. A summary file could be generated for this query. Query B is +slow and can be optimized if there was a query compiler of some +sort. Query C demonstrates a query in which there is no good +optimization which can be applied. These come with a certain amount of +baggage. + +It seems then that for boolean 'and' operations, plain forms should be +moved forward and other queries should be moved such that they are +evaluated later. I would expect that the majority of queries would be +of the plain form. + +First is that the summary file is tied to the query and the store +where the query originates from. Second, a hashing function for +strings needs to be calculated for the query so that the query and the +summary file can be associated. This hashing function could be similar +to the hashing function described in Rob Pike's "The Practice of +Programming". (FIXME: Stick page number here) + +** Archives + +Many people are concerned that archives won't be preserved, archives +aren't supported, and many other archive related issues. This is the +short version. + +Archives are just that, archives. Archives are stores. Take your +vfolder, export it to a store. You are done. If you load up the store +again, then the default-vfolder of that store is the view of the +vfolder, except the query is different. + +The point to vfolder is not to do away with classical folder +representation but to move the queries to the front where it would +make data management easier for people who don't think in terms of +files but in terms of queries because ordinary people don't think in +terms of files. + +* Miscellany + +** Annotations + +There should be a scheme to add annotations to messages. Common mail +user agents have used a tag in the message header to mark messages as +read/unread for example. Extending on this we have the ability to add +our own data to a message to add meaning to it. If we have a good +scheme for doing this, new possibilities are opened. + +*** Keywords + +When sending a message, a message could have certain keywords attached +to it. While this can be done with the subject line, the subject line +has a tendency to be munged by other mail applications. One popular +example is the "[rR]e:" prefix. Using the subject line also breaks the +"contract" with other mail user agents. Using keywords in another +field in the message header allows the sender to assist the recipient +in organizing data automatically. Note that the sender can only +provide hints as the sender is unlikely to know the organization +schemes of the recipient. + +** Scope + +Let us assume that we have multiple stores. Does a query work on a +given store? Or does it work on all stores? Or is it configurable such +that a query can work on a user-selected list of stores? + +* Alternatives to the above + +Jim Meyer <purp@selequa.com> is putting some notes on where +annotations needs to be located. They'll be located here as well as +any contributions I may have to them.
\ No newline at end of file |