Foxbrush

The HTTP protocol is the basis of the web, and provides many powerful capabilities for building Comet communication. In particular, HTTP is built on the principals of Representational State Transfer (REST). The REST concepts that underlie HTTP facilitate important scalability capabilities for agents that choose to take advantage of them, and enable meaningful communication about resources, which can benefit Comet usage. Comet was not specifically incorporated into HTTP, but HTTP provides a powerful vocabulary that can provide a substantial foundation for Comet communication. I propose an approach for RESTful Comet that would utilize the existing HTTP 1.1 and MIME standards as much as possible. This approach has important implications for future scalability possibilities, proxy service utilization, and REST data/messaging interaction. This approach maintains the critical HTTP principle of statelessness which reduces server resource consumption. This proposal can be integrated into proxy servers to provide increased scalability through distributed messaging with HTTP proxy servers. It could also even be utilized by proxy servers to maintain fresher caches independently of Comet applications.

A Simple HTTP Comet Protocol for Single Resources

For simple single resource HTTP GETs, it is possible to add a single simple header and provide a very intuitive technique for using Comet with a single resource that maintains the principles of HTTP. A “When-Modified-After” header is defined on a GET request that instructs an HTTP server that it should not respond to a request until a modification to a resource is made. The value of the header should be a time stamp, indicated the point in time after which new data should trigger a response. If a modification has already been made since the given time stamp, the server should immediately respond to the request with the current resource using a standard HTTP response. If there have been no modifications to the resource since the given “When-Modified-After” date, then the server should wait until a modification is made before responding to the request (with a standard HTTP response). The response should include a standard “Last-Modified” date, so the client knows what date value to use for the next “When-Modified-After” date to resume monitoring.

Using this simple extension to standard HTTP protocol, a client can create a connection and issue a request that stays open, and the response will not occur until a change has been made to the resource. This is effectively a long-polling Comet connection for monitoring a single resource. Here is an example:

-->
GET /myResource HTTP/1.1
Accept: */*
When-Modified-After: Wed, 26 Dec 2007 16:00:15 GMT

In the meantime, suppose the resource content is replaced with “Hello World!”, which triggers the monitored resource response below:

<--
HTTP/1.1 200 OK
Last-Modified: Wed, 26 Dec 2007 18:00:15 GMT

Hello World!


One very useful application of this approach would be for monitoring an Atom feed. Rather than polling for changes in an Atom feed, a newsreader client could issue a request with a “When-Modified-After” header, and the connection would remain open and waiting for a response until a new Atom entry was added.

REST Methods

While we can simply return a standard HTTP 200 (OK) response to provide an updated copy of the current resource, the REST concepts behind HTTP provide the impetus for a wider vocabulary of HTTP responses, which yields more flexible and efficient communication of resource modifications. When a resource is deleted while monitoring, a 404 (Not Found) response can be used to indicate that the resource has been removed. When data has been appended to a resource while monitoring, the HTTP range capabilities can be utilized to return a 206 (Partial Content) response. A 404 response is very simple, but returning partial content in a response is a bit more nuanced. A response may only return a 206 if the request has included a Range header indicating what range is desired. By including a Range header specifying the range units and range, a request can make it known that a 206 response is acceptable and which range units are acceptable. However, a Range header alone should only receive a successful response of 206 (not 200). In order to allow a 200 or a 206 response, the If-Range header can be used conditionally allow for a 200 or a 206 response, but with resource monitoring, the If-Range header as defined by HTTP is not useful. The condition for selecting between a 200 or a 206 response is based on the date of the resource, where resource monitoring should select the response type based on whether the resource had been modified in such a way that a partial content delivery is applicable. Therefore, I propose that combining the When-Modified-After header with the Range header should indicate that both 200 and 206 responses are acceptable and the selection should be based on the utility of a 206 response in communicating the modification. The HTTP specifications define a range unit of bytes. Below is an example:

-->
GET /myResource HTTP/1.1
Accept: */*
When-Modified-After: Wed, 26 Dec 2007 16:00:15 GMT
Range: bytes 12-/* This indicates that the bytes after byte 12 (the current end of the resource) should be returned

In the meantime, suppose the string “How are you?” was appended to the resource’s content. This action triggers the monitored resource response below:

<--
HTTP/1.1 206 OK
Content-Range: bytes 12-23/24

How are you?

With allowance for conditional 206 responses, monitored resource HTTP response codes have a correspondence to their causal HTTP method verbs:

• PUT action -> 200 - full resource content replacement results in a 200 full content response for monitoring request)
• DELETE action -> 404 - deletion of a resource results in a 404 response for monitoring request)
• POST action -> 206 - if the POST results in simple appending or replacement of data to the resource, and the response modification can be described with a content range, the result is a 206 response. Otherwise a full content 200 response may be necessary.

The byte range units may not be the most appropriate range unit for many modern format types. While the HTTP specification only defines the byte range unit, it allows for other range units. For modern data formats such as XML/Atom and JSON, the alternate range unit “updated” defined by the Atom Publishing Protocol is generally preferable. Let us consider the case of an Atom feed. When a new news entry is added to an Atom feed, it does not result in data being appended to the end of the content; instead the entry is embedded in the XML file. However, we can easily use a updated range unit to efficiently communicate the addition of a new entry:

GET /myAtomFeed HTTP/1.1
Accept: */*
When-Modified-After: Wed, 26 Dec 2007 16:00:15 GMT
Range: updated Wed, 26 Dec 2007 16:00:15 GMT-/*

In the meantime a new entry is added to the Atom feed, triggering this response:

HTTP/1.1 206 Partial Content
Content-Range: updated Wed, 26 Dec 2007 16:00:15 GMT-/Wed, 26 Dec 2007 20:00:25 GMT
<?xml version="1.0" encoding="UTF-8"?>
<feed>
<entry>
<title>REST Comet Protocol Possibilities</title>
<author>Kris Zyp</author>
<published>2007-12-26T20:00:25Z</published>
<content>...</content>
</entry>
</feed>


Streaming for Resource Monitoring

In a simple implementation of the “When-Modified-After” approach, the response will be a standard HTTP response to a GET request, which will end the HTTP conversation. To resume monitoring, a new HTTP request needs to be issued. However, it may often be preferable to keep the HTTP response alive and indefinitely continue to receive updates on a resource rather than only receiving a single update. The HTTP protocol is designed for streaming data (in particular with chunked encoding). However, it is still necessary to have some protocol for the partitioning of update messages from the data stream. Here the MIME standard provides standards-based direction for coherent segmentation. We can use multipart/mixed to break the stream into individual resource updates. In particular, we can use the multiplepart/mixed variant MIME type of multipart/mixed-replace, which indicates that each message is intended to be a replacement for the resource, superseding the previous message. When a 200 response is provided, the HTTP response can remain open indefinitely receiving new updates/replacements to the resource. This MIME type, in particular, has received attention for Comet because of Firefox’s support for it in its XmlHttpRequest object.

The multipart/mixed-replace is limiting, lacking the ability to transfer REST information with each update. The parts of the multipart/mixed MIME type are defined to only contain the content and content-type. Each update must be full replacement; partial content and deletion updates can not be included. However, another MIME type is applicable for this purpose. The multipart/digest MIME type is for sending whole messages as the parts of the outer message. While the MIME protocol defines its usage for email messages as being embedded email messages, it is easily inferred that in a different context such as HTTP, the parts would be the same type as the outer message, in this case HTTP response messages. Therefore a monitoring response could consist of a content stream of the multipart/digest type and contain within it multiple HTTP response messages with their own status codes, headers, and content.

Some browsers engines (Gecko, WebKit, and Opera) support incremental loading through XmlHttpRequest, but Internet Explorer does not. Without incremental loading, using a multipart response to provide a stream of data that remains open for future messages is problematic. If the stream stays open, an IE XHR request will not return until the response is finished. Therefore content negotiation via the Accept header should be used to indicate the browser’s preference for multipart. For example, a client that can handle multipart/digest and multipart/mixed-replace may send an Accept header on Firefox to indicate that multipart/digest (for the purpose of streaming) is preferred:

Accept: multipart/digest, multipart/mixed-replace;q=0.7, */*;q=0.5

And in Internet Explorer it could send:

Accept: */*, multipart/digest;q=0.5

When multipart/digest is a lower preference than the default content type, it can be assumed that no advantage is provided through multipart streaming (because the client does not support incremental loading).

Monitoring Multiple Resources with One Connection

If each resource requires its own HTTP request/response connection, we will quickly encounter scalability problems when attempting to monitor multiple resources. Here again, the multipart/digest type can rescue us with a means to deliver multiple HTTP requests in one HTTP request. The Content-Type of an HTTP request can be set to multipart/digest, and then multiple requests can be included in the body of the HTTP request using the MIME multipart definition. It may be useful when sending a request that encapsulates multiple monitoring requests to designate a single location as the target for these types of requests. For now, I will suggest “/subscribe”. I would also suggest that one can assume that the sub-requests can conceptually inherit headers from their parent request.

This technique does introduce a problem in correlating responses with requests. In HTTP, responses are normally correlated to requests simply by their TCP/IP connection, but when sending multiple HTTP requests embedded within a single HTTP request, this clearly is not adequate. In this case responses must use the HTTP Content-Location header to indicate what resource the HTTP response message is referring to. It is important to note that using multipart/digest type to encapsulate multiple responses in a single response does not necessitate multipart requests, nor vice versa. Request and response multiparts can be used independently or in combination. A multipart request can receive a single response, a single request can receive a multipart stream of response messages, and a multipart request can receive a multipart stream response.

Now let us look at an example incorporating these ideas. The following request encompasses two requests that indicate that the /news and /weather resources should be subscribed to or monitored:

-->
POST /subscribe HTTP/1.1
Accept: multipart/digest, multipart/mixed-replace;q=0.7, */*;q=0.5
When-Modified-After: Wed, 26 Dec 2007 16:00:15 GMT
Content-Type: multipart/digest; boundary=separation

--separation
GET /weather
Range: updated Wed, 26 Dec 2007 16:00:15 GMT-/*

--separation
GET /news
Range: updated Wed, 26 Dec 2007 16:00:15 GMT-/*

--separation

The response will now send parts of the content stream as resource modifications take place.

<--
HTTP/1.1 200 OK
Content-Type: multipart/digest; boundary=separation
Transfer-Encoding: chunked

after a news event occurred:

50; chunked encoding length
–separation
HTTP/1.1 206 Partial Content append to the news stories
Content-Location: /news
Content-Type: text/xml
Content-Range: updated Wed, 26 Dec 2007 16:00:15 GMT-/Wed, 26 Dec 2007 20:00:25 GMT

<?xml version="1.0" encoding="UTF-8"?>
<feed><entry><title>Big news today...</title>....</entry></feed>

--separation

after the weather update:

30
HTTP/1.1 200 OK replace the old weather report
Content-Type: text/plain
Content-Location: /weather

New weather report: Sunny weather ahead.

--separation

if the news feed is deleted, this would occur:

20
HTTP/1.1 404 Not Found
Content-Location: /news

--separation


Message Delivery Reliability with Incremental Time

Rather than relying on a stateful session to reliably receive a stream of messages, this proposal relies on time segments to define message delivery. When an agent subscribes to a Comet resource, the agent provides a beginning timestamp in the When-Modified-After header (which is normally the current time). When a response message arrives, the response message must include a timestamp (in the Last-Modified header or the Content-Range header if it is using a dated range). The agent can then be assured that it has received all (relevant) messages up to that timestamp. If the connection is broken (or ended in the case of long-polling), the agent can send the last timestamp in the When-Modified-After request header when reconnecting to the server. If another message that should be sent to the client was triggered while the connection was broken, the server can immediately fulfill the request; if no messages are ready to be sent, the server can wait for the next available message. If a server sends a message but the client does not receive it due to a broken connection, the client can still resume the connection with the previous timestamp. This technique allows clients to seamlessly receive all messages for a given timeframe even with an unreliable underlying connection. Furthermore this technique does not rely on a session and is therefore not prone to session timeouts and unnecessary server resource utilization.

An implication of this technique is that it is necessary for servers and clients to be capable of giving every message a unique timestamp. Therefore, servers must be able to assign timestamps with decimals for the seconds and provide a means for continually incrementing the timestamp for uniqueness when multiple messages are prepared within a single second.

Publish/Subscribe Messaging

A publish/subscribe event-based messaging system can still be handled with this approach. This type of system is important for some applications like a chat application. A resource can be treated like a pub/sub channel, and subscribing to a channel is done by monitoring a resource as described above. Messages can be published by posting to a resource using the HTTP POST method. A POST, as the HTTP protocol suggests, can imply appending a record/message to the resource. This can easily translate into the message being broadcast to all resource subscribers as 206 Partial Content messages. In this way the message can be published and subscribers will receive it.

Authentication

Since this proposal is basically an interpretation of the HTTP/1.1 specification for Comet, one can naturally use existing HTTP means for authentication, including the authentication scheme defined in the HTTP specification or alternate application level security that may use cookies or other methods for authentication.

Conclusion

This proposal for using existing different definitions in the HTTP tool set with minimal addition provides a standards-based approach for Comet with integrated REST resource interaction. Only one necessary addition is defined here (When-Modified-After)—everything else is leveraging existing standards. While there may be other approaches that are less verbose and more efficient for particular applications, I believe an approach like this that relies on existing standards and includes the capabilities for interacting with highly scalable world of RESTful resources has potential for significantly improving the interoperability of Comet applications and their integration with RESTful data services. I would like to follow up on this with some more discussion of practicalities like browser limitations, optimizations, globbing, and making this work with other transports like streaming iframes and JSONP scripts.