Sorting the event stream

Question

It seems like the data returned from the /events endpoint is flawed by being returned in descending order. For example:

{
  "items": [
    {
      "event_type": "comment_posted",
      "event_id": 11811324,
      "creation_date": 1329699806
    },
    {
      "event_type": "question_posted",
      "event_id": 9354990,
      "creation_date": 1329699800
    },
    ...
}

Wouldn't it make more sense for the stream to be returned in ascending order? That way you could clearly see that the creation_date of the last item in the stream is the since date of the next time you request events. As the API stands now, I have to do this:

1. Request events since time A
2. Check to see if the last event occurred at time A
3. If it didn't, try the request again, with a different page
4. Repeat the previous two steps until you find time A

In other words, it looks like this:

You jump forward in time, then make a number of small steps back, then jump forward, then make a number of small steps back...

And you have to do this because the event stream comes back in the wrong order.

If the event stream came back in ascending order (with the newest event last in the array), then you'd only have to do this:

Obviously the default sorting of the event stream can't be changed at this point, but can there be a new (and optional) sort field, so that I can request /events?since=123456789&sort=asc? That would make things so much simpler.

Answer 1

I don't follow, for date times the most recent date is the largest date.

Sample return by events

{
  "event_type": "question_posted",
  "event_id": 9369546,
  "creation_date": 1329777738
},
{
  "event_type": "comment_posted",
  "event_id": 11832402,
  "creation_date": 1329777736
}

1329777738 => Mon, 20 Feb 2012 22:42:18 GMT
1329777736 => Mon, 20 Feb 2012 22:42:16 GMT

A passed since also means the last item in the returns has the same creation_date (provided all events since then fit in your pagesize).

Example: /2.0/events?pagesize=100&since=1329780267&site=stackoverflow

{
  "event_type": "user_created",
  "event_id": 1222226,
  "creation_date": 1329780274
},
{
  "event_type": "comment_posted",
  "event_id": 11833065,
  "creation_date": 1329780274
},
{
  "event_type": "comment_posted",
  "event_id": 11833064,
  "creation_date": 1329780274
},
{
  "event_type": "comment_posted",
  "event_id": 11833062,
  "creation_date": 1329780268
},
{
  "event_type": "comment_posted",
  "event_id": 11833061,
  "creation_date": **1329780267**
}

If you're trying to get the most recent events, you constantly grab from page 1. If you want to avoid grabbing lots of events you've seen before, you use a since of the most recent (ie. highest) date you've seen; you can then use has_more or total to be intelligent about querying.

---

An ascending sort would also be really, really painful to use.

Consider /events covers a 15 minute window of activity, this means that the oldest (ie. lowest) dates are falling out of the set constantly.

This means that if you're paging over the results (moving left to right) the underlying stream could be moving right to left (deletions occur at the head, basically). To make sure you weren't missing events you'd have to always over select by one, otherwise you couldn't be sure you weren't seeing gaps.

In this diagram the stream is in ascending order (1 is the oldest), if between calls to page=1 and page=2 1 becomes > 15 minutes old it'll be removed, and 5 will never be returned.

The equivalent for descending order is selecting an event you've already seen before, which is much easier to detect and recover from.

---

Fundamentally, it's a little weird that we have a paging view of a stream of events. I don't recall off hand how much of this made it into the pre-beta spec, but very early on we evaluated WebSockets for /events. They weren't adequately stable at the time, but we're actively pursuing them in other areas with an eye towards revisiting the issue in a future API release.