This project is read-only.

IDLE Support

May 10, 2011 at 9:56 PM

So in the last few days I've been looking at the various open source IMAP libraries and your CodeProject article is spot on, none of them are "great". One of the specific features I'm looking for is IDLE support and I haven't found a library that had a working implementation of it. I noticed in one of your recent commits that you've added a skeleton for IDLE support (sadly not working as of yet), do you have a feel for when you'll have fleshed it out? I'm curious as to how it will interface with the LINQ aspect of your library.

May 11, 2011 at 3:30 PM

The Idle command is already implemented and "should" work, but could use some polishing, since the design has a small but non critical flaw.
Unfortunately the docu section does not yet cover the IDLE command, but the usage is pretty simple.
After connecting and authenticating you must subscribe to the 'StatusUpdateReceived' event and then call Idle().
Through this 'active' connection the server will now push status updates, which will be reflected through the StatusUpdateReceived event.
By subscribing to this event you can then decide what to do by checking the status changes, for example: open second connection and fetch new messages.

        ///   This method is blocking.
        ///   The IDLE command may be used with any IMAP4 server implementation
        ///   that returns "IDLE" as one of the supported capabilities to the
        ///   CAPABILITY command.  If the server does not advertise the IDLE
        ///   capability, the client MUST NOT use the IDLE command and must poll
        ///   for mailbox updates.
        ///   http://tools.ietf.org/html/rfc2177
        /// 

        public void Idle()
        {
            if (!Capability.CanIdle)
            {
                const string message = "Server does not support the idle command. Please check Capability.CanIdle before calling Idle.";
                throw new InvalidOperationException(message);
            }

            var command = new RawCommand("IDLE");
            Send(command);
            IsIdling = true;
            while (true)
            {
                var timeout = TimeSpan.FromMinutes(30);
                var reader = new ResponseReader(this);
                reader.ReadNextLine(timeout);
                UpdateStatusIfNecessary(reader);

                if (!reader.IsContinuation)
                {
                    continue;
                }

                var @continue = InvokeIdleContinuationPending();
                if (@continue)
                {
                    continue;
                }
                var c = new RawCommand("DONE");
                SendAndReceive(c);
                IsIdling = false;
            }
        }


May 11, 2011 at 4:51 PM
Edited May 11, 2011 at 6:01 PM

Yup, I had found the .Idle method and had wired up the StatusUpdateReceived and IdleContinuationPending events. Unfortunately, it's not handling the server response correctly (the continuation) as it falls through the entire while loop immediately and thus ending the IDLE state with the DONE.

8:37:50 AM >> a00 CAPABILITY
8:37:50 AM << * CAPABILITY IMAP4REV1 NAMESPACE AUTH=CRAM-MD5 UIDPLUS IDLE VOICEMESSAGES ACL ID CHILDREN
8:37:50 AM << a00 OK CAPABILITY completed
8:37:50 AM >> a01 AUTHENTICATE CRAM-MD5
8:37:50 AM << + (snip)
8:37:50 AM >> (snip)
8:37:52 AM << a01 OK CRAM-MD5 login successful
8:37:55 AM >> a03 IDLE
8:37:55 AM << + Idling
8:37:55 AM >> a04 DONE
8:37:55 AM << a03 OK IDLE terminated

I'm not familiar enough with the IMAP RFC or your code to suggest a fix yet :)

Edit

Ok if you just change "if (@continue)" to "if (!@continue)" that solves that problem. However, you run into another problem in the UpdateStatusIfNecessary() where it hangs on a reader.ReadNextLine() expecting more data from the server when there is no more, and thus never throws the StatusUpdateReceived event.

9:45:51 AM >> a00 CAPABILITY
9:45:51 AM << * CAPABILITY IMAP4REV1 NAMESPACE AUTH=CRAM-MD5 UIDPLUS IDLE VOICEMESSAGES ACL ID CHILDREN
9:45:51 AM << a00 OK CAPABILITY completed
9:45:51 AM >> a01 AUTHENTICATE CRAM-MD5
9:45:51 AM << + (snip)
9:45:51 AM >> (snip)
9:45:53 AM << a01 OK CRAM-MD5 login successful
9:45:53 AM >> a03 SELECT "INBOX"
9:45:53 AM << * 69 EXISTS
9:45:53 AM << * 1 RECENT
9:45:53 AM << * OK [UNSEEN 69] Message 69 is first unseen
9:45:53 AM << * FLAGS (\Deleted \Seen \Flagged \Draft \Recent)
9:46:20 AM << * OK [PERMANENTFLAGS (\Deleted \Seen \Flagged)] Limited
9:46:20 AM << * OK [UIDVALIDITY 1] UIDs valid
9:46:20 AM << a03 OK [READ-WRITE] SELECT completed
9:46:25 AM >> a04 IDLE
9:46:25 AM << + Idling
9:46:39 AM << * 1 RECENT
9:46:58 AM << * 67 EXISTS

The last two events were received at the same time but they have different timestamps because I was stepping through the code.

Oh well, I suppose I probably should submit a bug on it :)

May 11, 2011 at 8:00 PM

Thanks for the post,

since it was long time overdue I fixed the IDLE command, it is now working as expected.
The IDLE state can be cancelled asynchronously by calling StopIdle(). Starting Idle is now done by using the StartIdle() method.
The library will cancel the IDLE session after 29 minutes, because @ 30 the server will kick you.
Just use the latest source and you should be fine. 

Alex

May 12, 2011 at 5:08 PM

Thanks I got it working, was able to spin up a new IMAP connection in the StatusUpdateReceived to retrieve the new messages as they came in. Ideally it would be great to issue a StopIdle so I could re-use the existing connection and save a few seconds of having to create a new connection and authenticate, but that's the nature of the blocking .Readline.

I did run into an issue with the IMAP server I'm working with though, it appears to interpret ALL sequence sets as *:* which really breaks everything. Even simple commands like "SEARCH 10" will return a search result of "1 2 3 4 5 6 7 8 9 10". hah.

Anyways, do you plan on adding support for an anonymous return type for the .Select?

var query = imap2.Messages
                      .Where(x => x.InternalDate >= DateTime.Now.AddDays(-1) && !x.Flags.HasFlag(Crystalbyte.Equinox.MessageFlags.Seen))
                      .Select(x => new { Envelope = x.Envelope, SequenceNumber = x.SequenceNumber, UID = x.Uid });

Currently throws an exception for me :(

System.InvalidCastException was unhandled  Message=Unable to cast object of type 'Crystalbyte.Equinox.Imap.Envelope' to type '<>f__AnonymousType0`3[Crystalbyte.Equinox.Imap.Envelope,System.Int32,System.Int32]'.  Source=Anonymously Hosted DynamicMethods Assembly

Thanks again for the great library.

May 12, 2011 at 7:55 PM

I'm glad you got it working,

according to MSDN you should be able to call StopIdle() asynchronously even while its still listening.
In addition you can call StopIdle() inside the StatusUpdateReceived event handler, fetch the messages and call StartIdle() again.

I did run into an issue with the IMAP server I'm working with though, it appears to interpret ALL sequence sets as *:* which really breaks everything. Even simple commands like "SEARCH 10" will return a search result of "1 2 3 4 5 6 7 8 9 10". hah.

What kind of server is it ?

Anyways, do you plan on adding support for an anonymous return type for the .Select?

Currently no, the gain is much to low compared to the work involved, there are currently much more pressing concerns like Pop3, Proxy usage and S/MIME.

Regards Alex

May 13, 2011 at 6:14 PM

The IMAP server's greeting: * OK FirstClass IMAP4rev1 server v10.010 at xxx.xxx.com ready

Played with it some more and I can't quite get it working as it seems to be clobbering the connection.

My first setup I have a wrapper/helper class around the ImapClient and one of the things it does after establishing a connection is that it spins up a thread and in that thread it issues the blocking .StartIdle() call. Right now it's just a loop, but eventually it will handle timeouts, etc. Now when a new mail arrives it throws the StatusUpdateReceived event (twice, once for RECENT and then for EXISTS), on the EXISTS message it called .StopIdle, waited a second, and then tried to do the imap linq query to retrieve the new mail envelope. During this, the idle thread would have returned from the blocking .StartIdle(), waited a couple of seconds and then called .StartIdle() again.

Due to multiple threads writing to the console, these log lines are not quite in the right sequence.

==================
Event calls imap.StopIdle(), waits a second and then does imap linq query
==================
9:08:31 AM >> a00 CAPABILITY
9:08:31 AM << * CAPABILITY IMAP4REV1 NAMESPACE AUTH=CRAM-MD5 UIDPLUS IDLE VOICEMESSAGES ACL ID CHILDREN
9:08:32 AM << a00 OK CAPABILITY completed
9:08:32 AM >> a01 AUTHENTICATE CRAM-MD5
9:08:32 AM << (snip)
9:08:32 AM >> (snip)
9:08:33 AM << a01 OK CRAM-MD5 login successful
9:08:33 AM >> a03 EXAMINE INBOX
9:08:34 AM << * 91 EXISTS
9:08:34 AM << * 0 RECENT
9:08:34 AM << * FLAGS (\Deleted \Seen \Flagged \Draft \Recent)
9:08:34 AM << * OK [PERMANENTFLAGS ()] No permanent flags permitted
9:08:34 AM << * OK [UIDVALIDITY 1] UIDs valid
9:08:34 AM << a03 OK [READ-ONLY] EXAMINE completed
9:08:34 AM >> a04 IDLE
9:08:34 AM << + Idling
9:09:13 AM << * 1 RECENT
StatusUpdateRecieved_Enter E0 R1 U0
StatusUpdateRecieved_Leave E0 R1 U0
9:09:14 AM << * 92 EXISTS
9:09:14 AM >> a05 DONE
StatusUpdateRecieved_Enter E92 R0 U0
StopIdle issued
Exec imap query
9:09:14 AM >> a06 SEARCH SINCE 12-May-2011 NOT SEEN
9:09:14 AM << a04 OK IDLE terminated
StatusUpdateRecieved_Leave E92 R0 U0
9:09:15 AM << * SEARCH 92
9:09:15 AM << a06 OK SEARCH complete
ConnectAndIdle exit
9:09:17 AM >> a07 IDLE
9:09:17 AM << + Idling
9:09:48 AM >> a08 DONE
9:09:48 AM << a07 OK IDLE terminated
ConnectAndIdle exit
9:09:51 AM >> a09 LOGOUT
9:09:51 AM << * BYE FirstClass IMAP4 server logging out

So, I decided to be more explicit in waiting for the .StopIdle() to return and then not start a new .StartIdle() until the event is done processing. So in the event it sets a flag called PauseIdle, calls .StopIdle() and then goes into a loop waiting for .IsIdling to be false. In the idlethread, when it returns from the .StartIdle() it will fall into a loop waiting for PauseIdle to be set back to false before issuing a new .StartIdle(). This blew up worse than my first attempt lol :p

Due to multiple threads writing to the console, these log lines are not quite in the right sequence.

==================
Event changed to call imap.StopIdle() and then go into a loop waiting for imap.IsIdling to return false (loop just contains a Thread.Sleep(250))
==================
9:18:32 AM >> a00 CAPABILITY
9:18:32 AM << * CAPABILITY IMAP4REV1 NAMESPACE AUTH=CRAM-MD5 UIDPLUS IDLE VOICEMESSAGES ACL ID CHILDREN
9:18:32 AM << a00 OK CAPABILITY completed
9:18:32 AM >> a01 AUTHENTICATE CRAM-MD5
9:18:32 AM << (snip)
9:18:32 AM >> (snip)
9:18:34 AM << a01 OK CRAM-MD5 login successful
9:18:34 AM >> a03 EXAMINE INBOX
9:18:34 AM << * 92 EXISTS
9:18:34 AM << * 1 RECENT
9:18:34 AM << * OK [UNSEEN 92] Message 92 is first unseen
9:18:34 AM << * FLAGS (\Deleted \Seen \Flagged \Draft \Recent)
9:18:34 AM << * OK [PERMANENTFLAGS ()] No permanent flags permitted
9:18:34 AM << * OK [UIDVALIDITY 1] UIDs valid
9:18:34 AM << a03 OK [READ-ONLY] EXAMINE completed
9:18:34 AM >> a04 IDLE
.StartIdle() starting
9:18:34 AM << + Idling
9:19:04 AM << * 1 RECENT
StatusUpdateRecieved_Enter E0 R1 U0
StatusUpdateRecieved_Leave E0 R1 U0
9:19:05 AM << * 93 EXISTS
9:19:05 AM >> a05 DONE
StatusUpdateRecieved_Enter E93 R0 U0
StopIdle() issued and PauseIdle set

Aside from the fact that I might have a horrible implementation logic fail going on here, I'm guessing that the StatusUpdateReceived event itself is blocking. Additional incoming messages from the server don't seem to be processed until I leave the event, or while in the event a new command is issued with SendAndReceive which then processes an incoming message. It's also possible that the messages are being received but not by the sender so it is not the message they were expecting and it is thrown away.

May 13, 2011 at 9:32 PM

Hi vermis,

I did run into an issue with the IMAP server I'm working with though, it appears to interpret ALL sequence sets as *:* which really breaks everything. Even simple commands like "SEARCH 10" will return a search result of "1 2 3 4 5 6 7 8 9 10". hah.

well the server is a FirstClass Groupware server, see http://www.kki.de/.
I'm currently not sure whether its a server problem or if the client has a defect, I haven't had the opportunity to test on all servers, yet.

Using timers in an asynchronous environment is not a good idea, it invokes race conditions with unpredictable outcomes.

I tried to implement my suggestions and had similar problems, but I know what the problem is and more importantly how to fix it.
I will fix something up tomorrow, it should ease the process considerably.

Thanks for your support

Alex

May 15, 2011 at 10:51 PM

Hi vermis,

I didn't quite get around implementing things yet, the problem is, we can't send a "done command" until we received the "+ idling" response from the server, therefor I need to reintroduce the IdleContinuationRequested event.

I will try to get things done within the next days, keep ya head up ;)

Alex

May 18, 2011 at 8:45 PM

Hi vermis,

unfortunately I don't see a way how it might be possible to sustain the connection and disconnect the IDLE state. In order to do that, we must wait until we receive all server responses, which we cant because we get stuck inside the read line. There is no way to determine whether the current response is the last. If you find more informations about this, please contact me, I'm curious myself.

Alex

May 18, 2011 at 9:30 PM

It's probably not something that can be solved while using the TCPClient as it is blocking to keep it simple. You would need to change your whole networking core over to use non-blocking sockets, which I'm guessing would be far from trivial.

I have gotten my utility mostly working by spinning up a new imap connection to download the the mails from received IDLE events and it works. It's probably not the most efficient way of doing things with the expense of opening a new connection and going through the logon process, but it does work. When I get some more time (buried under work atm) I'll play with it some more and polish it up (I'm concerned that if I receive 5 new emails in rapid succession that i'll spin up and close 5 connections with the server also rapidly).

Do you expose a method that will retrieve a message (or just the envelope) without doing a SEARCH first? I noticed that since .FetchMessageBySequenceNumber(int) issues a SEARCH first that with this broken mail server it retrieves every mail in the mailbox then :(

Thanks again.

May 18, 2011 at 11:49 PM
Edited May 18, 2011 at 11:51 PM

Hi vermis,

I tweaked the LINQ parser in the latest change set.

The search command will now only be triggered if necessary.

A clause as

.Where(x => x.SequenceNumber == 1)

will no longer trigger a search since we already have all information we need to fetch the correct message.

Clauses as

.Where(x => x.SequenceNumber > 1 && x.Date < DateTime.Now) 

will still trigger a search command since only the server knows which messages fit the criteria.

Since all fetching inside the client is done using LINQ, all methods will now behave this way.

Its late and although all unit tests are in the green, I still haven't tested it as much as I probably should.

If you run into any strange behaviour please let me know.

Alex