bluuuuuuurrrrrgh

svn path=/trunk/; revision=4692

1 files changed, 357 insertions, 0 deletions

diff --git a/mail/README.async b/mail/README.async
new file mode 100644
index 0000000000..a7b9d9d92d
--- /dev/null
+++ b/mail/README.async
@@ -0,0 +1,357 @@
+Asynchronous Mailer Information
+Peter Williams <peterw@helixcode.com>
+8/4/2000
+
+1. INTRODUCTION
+
+It's pretty clear that the Evolution mailer needs to be asynchronous in
+some manner. Blocking the UI completely on IMAP calls or large disk reads
+is unnacceptable, and it's really nice to be able to thread the message
+view in the background, or do other things while a mailbox is downloading.
+
+The problem in making Evolution asynchronous is Camel. Camel is not 
+asynchronous for a few reasons. All of its interfaces are synchronous --
+calls like camel_store_get_folder, camel_folder_get_message, etc. can
+take a very long time if they're being performed over a network or with
+a large mbox mailbox file. However, these functions have no mechanism
+for specifying that the operation is in progress but not complete, and
+no mechanism for signaling when to operation does complete.
+
+2. WHY I DIDN'T MAKE CAMEL ASYNCHRONOUS
+
+It seems like it would be a good idea, then, to rewrite Camel to be
+asynchonous. This presents several problems:
+
+	* Many interfaces must be rewritten to support "completed"
+	  callbacks, etc. Some of these interfaces are connected to
+	  synchronous CORBA calls.
+	* Everything must be rewritten to be asynchonous. This includes
+	  the CamelStore, CamelFolder, CamelMimeMessage, CamelProvider,
+	  every subclass thereof, and all the code that touches these.
+	  These include the files in camel/, mail/, filter/, and
+	  composer/. The change would be a complete redesign for any
+	  provider implementation.
+	* All the work on providers (IMAP, mh, mbox, nntp) up to this
+	  point would be wasted. While they were being rewritten
+	  evolution-mail would be useless.
+
+However, it is worth noting that the solution I chose is not optimal,
+and I think that it would be worthwhile to write a libcamel2 or some
+such thing that was designed from the ground up to work asynchronously.
+Starting fresh from such a design would work, but trying to move the
+existing code over would be more trouble than it's worth.
+
+3. WHY I MADE CAMEL THREADED
+
+If Camel was not going to be made asynchronous, really the only other
+choice was to make it multithreaded. [1] No one has been particularly
+excited by this plan, as debugging and writing MT-safe code is hard.
+But there wasn't much of a choice, and I believed that a simple thread
+wrapper could be written around Camel.
+
+The important thing to know is that while Camel is multithreaded, we
+DO NOT and CANNOT share objects between threads. Instead,
+evolution-mail sends a request to a dispatching thread, which performs
+the action or queues it to be performed. (See section 4 for details)
+
+The goal that I was working towards is that there should be no calls
+to camel made, ever, in the main thread. I didn't expect to and
+didn't do this, but that was the intent.
+
+[1]. Well, we could fork off another process, but they share so much
+data that this would be pretty impractical.
+
+4. IMPLEMENTATION
+
+a. CamelObject
+
+Threading presented a major problem regarding Camel. Camel is based
+on the GTK Object system, and uses signals to communicate events. This
+is okay in a nonthreaded application, but the GTK Object system is
+not thread-safe.
+
+Particularly, signals and object allocations use static data. Using
+either one inside Camel would guarantee that we'd be stuck with 
+random crashes forevermore. That's Bad (TM).
+
+There were two choices: make sure to limit our usage of GTK, or stop
+using the GTK Object system. I decided to do the latter, as the
+former would lead to a mess of "what GTK calls can we make" and
+GDK_THREADS_ENTER and accidentally calling UI functions and upgrades
+to GTK breaking everything.
+
+So I wrote a very very simple CamelObject system. It had three goals:
+
+	* Be really straightforward, just encapsulate the type 
+	  heirarchy without all that GtkArg silliness or anything.
+	* Be as compatible as possible with the GTK Object system
+	  to make porting easy
+	* Be threadsafe
+
+It supports:
+
+	* Type inheritance
+	* Events (signals)
+	* Type checking
+	* Normal refcounting
+	* No unref/destroy messiness
+	* Threadsafety
+	* Class functions
+
+The entire code to the object system is in camel/camel-object.c. It's
+a naive implementation and not full of features, but intentionally that
+way. The main differences between GTK Objects and Camel Objects are:
+
+	* s,gtk,camel,i of course
+	* Finalize is no longer a GtkObjectClass function. You specify
+	  a finalize function along with an init function when declaring
+	  a type, and it is called automatically and chained automatically.
+	* Declaring a type is a slightly different API
+	* The signal system is replaced with a not-so-clever event system.
+	  Every event is equivalent to a NONE__POINTER signal. The syntax
+	  is slightly different: a class "declares" an event and specifies
+	  a name and a "prep func", that is called before the event is
+	  triggered and can cancel it.
+	* There is only one CamelXXXClass in existence for every type.
+	  All objects share it.
+
+There is a shell script, tools/make-camel-object.sh that will do all of
+the common substitutions to make a file CamelObject-compatible. Usually
+all that needs to be done is move the implementation of the finalize
+event out of the class init, modify the get_type function, and replace
+signals with events.
+
+Pitfalls in the transition that I ran into were:
+
+	* gtk_object_ref -> camel_object_ref or you coredump
+	* some files return 'guint' instead of GtkType and must be changed
+	* Remove the #include <gtk/gtk.h>
+	* gtk_object_set_datas must be changed (This happened once; I 
+	  added a static hashtable)
+	* signals have to be fudged a bit to match the gpointer input
+	* the BAST_CASTARD option is on, meaning failed typecasts will
+	  return NULL, almost guaranteeing a segfault -- gets those
+	  bugs fixed double-quick!
+
+b. API -- mail_operation_spec
+
+I worked by creating a very specific definition of a "mail operation"
+and wrote an engine to queue and dispatch them.
+
+A mail operation is defined by a structure mail_operation_spec
+prototyped in mail-threads.h. It comes in three logical parts -- a 
+"setup" phase, executed in the main thread; a "do" phase, executed
+in the dispatch thread; and a "cleanup" phase, executed in the main
+thread. These three phases are guaranteed to be performed in order
+and atomically with respect to other mail operations.
+
+Each of these phases is represented by a function pointer in the
+mail_operation_spec structure. The function mail_operation_queue() is
+called and passed a pointer to a mail_operation_spec and a user_data-style
+pointer that fills in the operation's parameters. The "setup" callback
+is called immediately, though that may change.
+
+Each callback is passed three parameters: a pointer to the user_data,
+a pointer to the "operation data", and a pointer to a CamelException.
+The "operation data" is allocated automatically and freed when the operation
+completes. Internal data that needs to be shared between phases should
+be stored here. The size allocated is specified in the mail_operation_spec
+structure.
+
+Because all of the callbacks use Camel calls at some point, the 
+CamelException is provided as utility. The dispatcher will catch exceptions
+and display error dialogs, unlike the synchronous code which lets
+exceptions fall through the cracks fairly easily.
+
+I tried to implement all the operations following this convention. Basically
+I used this skeleton code for all the operations, just filling in the 
+specifics:
+
+===================================
+
+typedef struct operation_name_input_s {
+	parameters to operation
+} operation_name_input_t;
+
+typedef struct operation_name_data_s {
+	internal data to operation, if any
+	(if none, omit the structure and set opdata_size to 0)
+} operation_name_data_t;
+
+static gchar *describe_operation_name (gpointer in_data, gboolean gerund);
+static void setup_operation_name   (gpointer in_data, gpointer op_data, CamelException *ex);
+static void do_operation_name      (gpointer in_data, gpointer op_data, CamelException *ex);
+static void cleanup_operation_name (gpointer in_data, gpointer op_data, CamelException *ex);
+
+static gchar *describe_operation_name (gpointer in_data, gboolean gerund)
+{
+	if (gerund) {
+		return a g_strdup'ed string describing what we're doing
+	} else {
+		return a g_strdup'ed string describing what we're about to do
+	}
+}
+
+static void setup_operation_name (gpointer in_data, gpointer op_data, CamelException *ex)
+{
+	operation_name_input_t *input = (operation_name_input_t *) in_data;
+
+	verify that parameters are valid
+
+	initialize op_data
+
+	reference objects
+}
+
+static void do_operation_name (gpointer in_data, gpointer op_data, CamelException *ex)
+{
+	operation_name_input_t *input = (operation_name_input_t *) in_data;
+	operation_name_data_t *data = (operation_name_data_t *) op_data;
+
+	perform camel operations
+}
+
+static void cleanup_operation_name (gpointer in_data, gpointer op_data, CamelException *ex)
+{
+	operation_name_input_t *input = (operation_name_input_t *) in_data;
+	operation_name_data_t *data = (operation_name_data_t *) op_data;
+
+	perform UI updates
+
+	free allocations
+
+	dereference objects
+}
+
+static const mail_operation_spec op_operation_name = {
+	describe_operation_name,
+	sizoef (operation_name_data_t),
+	setup_operation_name,
+	do_operation_name,
+	cleanup_operation_name
+};
+
+void
+mail_do_operation_name (parameters)
+{
+	operation_name_input_t *input
+
+	input = g_new (operation_name_input_t, 1);
+
+	store parameters in input
+
+	mail_operation_queue (&op_operation_name, input, TRUE);
+}
+
+===========================================
+
+c. mail-ops.c
+
+Has been drawn and quartered. It has been split into:
+
+	* mail-callbacks.c: the UI callbacks
+	* mail-tools.c: useful sequences wrapping common Camel operations
+	* mail-ops.c: implementations of all the mail_operation_specs
+
+An important part of mail-ops.c are the global functions 
+mail_tool_camel_lock_{up,down}. These simulate a recursize mutex around
+camel. There are an extreme few, supposedly safe, calls to Camel made in
+the main thread. These functions should go around evey call to Camel or
+group thereof. I don't think they're necessary but it's nice to know
+they're there.
+
+If you look at mail-tools.c, you'll notice that all the Camel calls are
+protected with these functions. Remember that a mail tool is really 
+just another Camel call, so don't use them in the main thread either.
+
+All the mail operations are implemented in mail-ops.c EXCEPT:
+
+	* filter-driver.c: the filter_mail operation
+	* message-list.c: the regenerate_messagelist operation
+	* message-thread.c: the thread_messages operation
+
+d. Using the operations
+
+The mail operations as implemented are very specific to evolution-mail. I
+was thinking about leaving them mostly generic and then allowing extra
+callbacks to be added to perform the more specific UI touches, but this
+seemed kind of pointless.
+
+I basically looked through the code, found references to Camel, and split
+the code into three parts -- the bit before the Camel calls, the bit after,
+and the Camel calls. These were mapped onto the template, given a name,
+and added to mail-ops.c. Additionally, I simplified the common tasks that
+were taken care of in mail-tools.c, making some functions much simpler.
+
+Ninety-nine percent of the time, whatever operation is being done is being
+done in a callback, so all that has to be done is this:
+
+==================
+
+void my_callback (GtkObject *obj, gchar *uid)
+{
+	camel_do_something (uid);
+}
+
+==== becomes ====
+
+void my_callback (GtkObject *obj, gchar *uid)
+{
+	mail_do_do_something (uid);
+}
+
+=================
+
+There are, however, a few belligerents. Particularly, the function
+mail_uri_to_folder returns a CamelFolder and yet should really be
+asynchronous. This is called in a CORBA call that is sychronous, and
+additionally is used in the filter code.
+
+I changed the first usage to return the folder immediately but
+still fetch the CamelFolder asyncrhonously, and in the second case,
+made filtering asynchronous, so the fact that the call is synchronous
+doesn't matter.
+
+The function was renamed to mail_tool_uri_to_folder to emphasize that
+it's a synchronous Camel call.
+
+e. The dispatcher
+
+mail_operation_queue () takes its parameters and assembles them in a
+closure_t structure, which I abbreviate clur. It sets a timeout to
+display a progress window if an operation is still running one second
+later (we're not smart enough to check if it's the same operation,
+but the issue is not a big deal). The other thread and some communication
+pipes are created.
+
+The dispatcher thread sits in a loop reading from a pipe. Every time
+the main thread queues an operation, it writes the closure_t into the pipe.
+The dispatcher reads the closure, sends a STARTING message to the main
+thread (see below for explanation), calls the callback specified in the
+closure, and sends a FINISHED message. It then goes back to reading
+from its pipe; it will either block until another operation comes along,
+or find one right away and start it. This the pipe takes care of queueing
+operations.
+
+The dispatch thread communicates with the main thread with another pipe;
+however, the main thread has other things to do than read from the pipe,
+so it adds registers a GIOReader that checks for messages in the glib
+main loop. In addition to starting and finishing messages, the other
+thread can communicate to the user using messages and a progress bar.
+(This is currently implemented but unused.)
+
+5. ISSUES
+
+	* Operations are queued and dequeued stupidly. Like if you click
+	  on one message then click on another, the first will be retrieved
+	  and displayed then overwritten by the second. Operations that could
+	  be performed at the same time safely aren't.
+	* The CamelObject system is workable, but it'd be nice to work with
+	  something established like the GtkObject
+	* The whole threading idea is not great. Concensus is that an 
+	  asynchronous interface is the Right Thing, eventually.
+	* Care still needs to be taken when designing evolution-mail code to
+	  work with the asynchronous mail_do_ functions
+	* Some of the operations are extremely hacky.
+	* IMAP's timeout to send a NOOP had to be removed because we can't
+	  use GTK. We need an alternative for this.
\ No newline at end of file