diff options
Diffstat (limited to 'mail')
| -rw-r--r-- | mail/README.async | 357 |
1 files changed, 357 insertions, 0 deletions
diff --git a/mail/README.async b/mail/README.async new file mode 100644 index 0000000000..a7b9d9d92d --- /dev/null +++ b/mail/README.async @@ -0,0 +1,357 @@ +Asynchronous Mailer Information +Peter Williams <peterw@helixcode.com> +8/4/2000 + +1. INTRODUCTION + +It's pretty clear that the Evolution mailer needs to be asynchronous in +some manner. Blocking the UI completely on IMAP calls or large disk reads +is unnacceptable, and it's really nice to be able to thread the message +view in the background, or do other things while a mailbox is downloading. + +The problem in making Evolution asynchronous is Camel. Camel is not +asynchronous for a few reasons. All of its interfaces are synchronous -- +calls like camel_store_get_folder, camel_folder_get_message, etc. can +take a very long time if they're being performed over a network or with +a large mbox mailbox file. However, these functions have no mechanism +for specifying that the operation is in progress but not complete, and +no mechanism for signaling when to operation does complete. + +2. WHY I DIDN'T MAKE CAMEL ASYNCHRONOUS + +It seems like it would be a good idea, then, to rewrite Camel to be +asynchonous. This presents several problems: + + * Many interfaces must be rewritten to support "completed" + callbacks, etc. Some of these interfaces are connected to + synchronous CORBA calls. + * Everything must be rewritten to be asynchonous. This includes + the CamelStore, CamelFolder, CamelMimeMessage, CamelProvider, + every subclass thereof, and all the code that touches these. + These include the files in camel/, mail/, filter/, and + composer/. The change would be a complete redesign for any + provider implementation. + * All the work on providers (IMAP, mh, mbox, nntp) up to this + point would be wasted. While they were being rewritten + evolution-mail would be useless. + +However, it is worth noting that the solution I chose is not optimal, +and I think that it would be worthwhile to write a libcamel2 or some +such thing that was designed from the ground up to work asynchronously. +Starting fresh from such a design would work, but trying to move the +existing code over would be more trouble than it's worth. + +3. WHY I MADE CAMEL THREADED + +If Camel was not going to be made asynchronous, really the only other +choice was to make it multithreaded. [1] No one has been particularly +excited by this plan, as debugging and writing MT-safe code is hard. +But there wasn't much of a choice, and I believed that a simple thread +wrapper could be written around Camel. + +The important thing to know is that while Camel is multithreaded, we +DO NOT and CANNOT share objects between threads. Instead, +evolution-mail sends a request to a dispatching thread, which performs +the action or queues it to be performed. (See section 4 for details) + +The goal that I was working towards is that there should be no calls +to camel made, ever, in the main thread. I didn't expect to and +didn't do this, but that was the intent. + +[1]. Well, we could fork off another process, but they share so much +data that this would be pretty impractical. + +4. IMPLEMENTATION + +a. CamelObject + +Threading presented a major problem regarding Camel. Camel is based +on the GTK Object system, and uses signals to communicate events. This +is okay in a nonthreaded application, but the GTK Object system is +not thread-safe. + +Particularly, signals and object allocations use static data. Using +either one inside Camel would guarantee that we'd be stuck with +random crashes forevermore. That's Bad (TM). + +There were two choices: make sure to limit our usage of GTK, or stop +using the GTK Object system. I decided to do the latter, as the +former would lead to a mess of "what GTK calls can we make" and +GDK_THREADS_ENTER and accidentally calling UI functions and upgrades +to GTK breaking everything. + +So I wrote a very very simple CamelObject system. It had three goals: + + * Be really straightforward, just encapsulate the type + heirarchy without all that GtkArg silliness or anything. + * Be as compatible as possible with the GTK Object system + to make porting easy + * Be threadsafe + +It supports: + + * Type inheritance + * Events (signals) + * Type checking + * Normal refcounting + * No unref/destroy messiness + * Threadsafety + * Class functions + +The entire code to the object system is in camel/camel-object.c. It's +a naive implementation and not full of features, but intentionally that +way. The main differences between GTK Objects and Camel Objects are: + + * s,gtk,camel,i of course + * Finalize is no longer a GtkObjectClass function. You specify + a finalize function along with an init function when declaring + a type, and it is called automatically and chained automatically. + * Declaring a type is a slightly different API + * The signal system is replaced with a not-so-clever event system. + Every event is equivalent to a NONE__POINTER signal. The syntax + is slightly different: a class "declares" an event and specifies + a name and a "prep func", that is called before the event is + triggered and can cancel it. + * There is only one CamelXXXClass in existence for every type. + All objects share it. + +There is a shell script, tools/make-camel-object.sh that will do all of +the common substitutions to make a file CamelObject-compatible. Usually +all that needs to be done is move the implementation of the finalize +event out of the class init, modify the get_type function, and replace +signals with events. + +Pitfalls in the transition that I ran into were: + + * gtk_object_ref -> camel_object_ref or you coredump + * some files return 'guint' instead of GtkType and must be changed + * Remove the #include <gtk/gtk.h> + * gtk_object_set_datas must be changed (This happened once; I + added a static hashtable) + * signals have to be fudged a bit to match the gpointer input + * the BAST_CASTARD option is on, meaning failed typecasts will + return NULL, almost guaranteeing a segfault -- gets those + bugs fixed double-quick! + +b. API -- mail_operation_spec + +I worked by creating a very specific definition of a "mail operation" +and wrote an engine to queue and dispatch them. + +A mail operation is defined by a structure mail_operation_spec +prototyped in mail-threads.h. It comes in three logical parts -- a +"setup" phase, executed in the main thread; a "do" phase, executed +in the dispatch thread; and a "cleanup" phase, executed in the main +thread. These three phases are guaranteed to be performed in order +and atomically with respect to other mail operations. + +Each of these phases is represented by a function pointer in the +mail_operation_spec structure. The function mail_operation_queue() is +called and passed a pointer to a mail_operation_spec and a user_data-style +pointer that fills in the operation's parameters. The "setup" callback +is called immediately, though that may change. + +Each callback is passed three parameters: a pointer to the user_data, +a pointer to the "operation data", and a pointer to a CamelException. +The "operation data" is allocated automatically and freed when the operation +completes. Internal data that needs to be shared between phases should +be stored here. The size allocated is specified in the mail_operation_spec +structure. + +Because all of the callbacks use Camel calls at some point, the +CamelException is provided as utility. The dispatcher will catch exceptions +and display error dialogs, unlike the synchronous code which lets +exceptions fall through the cracks fairly easily. + +I tried to implement all the operations following this convention. Basically +I used this skeleton code for all the operations, just filling in the +specifics: + +=================================== + +typedef struct operation_name_input_s { + parameters to operation +} operation_name_input_t; + +typedef struct operation_name_data_s { + internal data to operation, if any + (if none, omit the structure and set opdata_size to 0) +} operation_name_data_t; + +static gchar *describe_operation_name (gpointer in_data, gboolean gerund); +static void setup_operation_name (gpointer in_data, gpointer op_data, CamelException *ex); +static void do_operation_name (gpointer in_data, gpointer op_data, CamelException *ex); +static void cleanup_operation_name (gpointer in_data, gpointer op_data, CamelException *ex); + +static gchar *describe_operation_name (gpointer in_data, gboolean gerund) +{ + if (gerund) { + return a g_strdup'ed string describing what we're doing + } else { + return a g_strdup'ed string describing what we're about to do + } +} + +static void setup_operation_name (gpointer in_data, gpointer op_data, CamelException *ex) +{ + operation_name_input_t *input = (operation_name_input_t *) in_data; + + verify that parameters are valid + + initialize op_data + + reference objects +} + +static void do_operation_name (gpointer in_data, gpointer op_data, CamelException *ex) +{ + operation_name_input_t *input = (operation_name_input_t *) in_data; + operation_name_data_t *data = (operation_name_data_t *) op_data; + + perform camel operations +} + +static void cleanup_operation_name (gpointer in_data, gpointer op_data, CamelException *ex) +{ + operation_name_input_t *input = (operation_name_input_t *) in_data; + operation_name_data_t *data = (operation_name_data_t *) op_data; + + perform UI updates + + free allocations + + dereference objects +} + +static const mail_operation_spec op_operation_name = { + describe_operation_name, + sizoef (operation_name_data_t), + setup_operation_name, + do_operation_name, + cleanup_operation_name +}; + +void +mail_do_operation_name (parameters) +{ + operation_name_input_t *input + + input = g_new (operation_name_input_t, 1); + + store parameters in input + + mail_operation_queue (&op_operation_name, input, TRUE); +} + +=========================================== + +c. mail-ops.c + +Has been drawn and quartered. It has been split into: + + * mail-callbacks.c: the UI callbacks + * mail-tools.c: useful sequences wrapping common Camel operations + * mail-ops.c: implementations of all the mail_operation_specs + +An important part of mail-ops.c are the global functions +mail_tool_camel_lock_{up,down}. These simulate a recursize mutex around +camel. There are an extreme few, supposedly safe, calls to Camel made in +the main thread. These functions should go around evey call to Camel or +group thereof. I don't think they're necessary but it's nice to know +they're there. + +If you look at mail-tools.c, you'll notice that all the Camel calls are +protected with these functions. Remember that a mail tool is really +just another Camel call, so don't use them in the main thread either. + +All the mail operations are implemented in mail-ops.c EXCEPT: + + * filter-driver.c: the filter_mail operation + * message-list.c: the regenerate_messagelist operation + * message-thread.c: the thread_messages operation + +d. Using the operations + +The mail operations as implemented are very specific to evolution-mail. I +was thinking about leaving them mostly generic and then allowing extra +callbacks to be added to perform the more specific UI touches, but this +seemed kind of pointless. + +I basically looked through the code, found references to Camel, and split +the code into three parts -- the bit before the Camel calls, the bit after, +and the Camel calls. These were mapped onto the template, given a name, +and added to mail-ops.c. Additionally, I simplified the common tasks that +were taken care of in mail-tools.c, making some functions much simpler. + +Ninety-nine percent of the time, whatever operation is being done is being +done in a callback, so all that has to be done is this: + +================== + +void my_callback (GtkObject *obj, gchar *uid) +{ + camel_do_something (uid); +} + +==== becomes ==== + +void my_callback (GtkObject *obj, gchar *uid) +{ + mail_do_do_something (uid); +} + +================= + +There are, however, a few belligerents. Particularly, the function +mail_uri_to_folder returns a CamelFolder and yet should really be +asynchronous. This is called in a CORBA call that is sychronous, and +additionally is used in the filter code. + +I changed the first usage to return the folder immediately but +still fetch the CamelFolder asyncrhonously, and in the second case, +made filtering asynchronous, so the fact that the call is synchronous +doesn't matter. + +The function was renamed to mail_tool_uri_to_folder to emphasize that +it's a synchronous Camel call. + +e. The dispatcher + +mail_operation_queue () takes its parameters and assembles them in a +closure_t structure, which I abbreviate clur. It sets a timeout to +display a progress window if an operation is still running one second +later (we're not smart enough to check if it's the same operation, +but the issue is not a big deal). The other thread and some communication +pipes are created. + +The dispatcher thread sits in a loop reading from a pipe. Every time +the main thread queues an operation, it writes the closure_t into the pipe. +The dispatcher reads the closure, sends a STARTING message to the main +thread (see below for explanation), calls the callback specified in the +closure, and sends a FINISHED message. It then goes back to reading +from its pipe; it will either block until another operation comes along, +or find one right away and start it. This the pipe takes care of queueing +operations. + +The dispatch thread communicates with the main thread with another pipe; +however, the main thread has other things to do than read from the pipe, +so it adds registers a GIOReader that checks for messages in the glib +main loop. In addition to starting and finishing messages, the other +thread can communicate to the user using messages and a progress bar. +(This is currently implemented but unused.) + +5. ISSUES + + * Operations are queued and dequeued stupidly. Like if you click + on one message then click on another, the first will be retrieved + and displayed then overwritten by the second. Operations that could + be performed at the same time safely aren't. + * The CamelObject system is workable, but it'd be nice to work with + something established like the GtkObject + * The whole threading idea is not great. Concensus is that an + asynchronous interface is the Right Thing, eventually. + * Care still needs to be taken when designing evolution-mail code to + work with the asynchronous mail_do_ functions + * Some of the operations are extremely hacky. + * IMAP's timeout to send a NOOP had to be removed because we can't + use GTK. We need an alternative for this.
\ No newline at end of file |