5

I've written an IRC bot that can notify channels when new questions with a certain tag are posted. However, currently I'm polling the URL #{APIURL}questions?pagesize=100&fromdate=#{@_lastQuestionsDate}&tagged=node.js&key=#{KEY} every minute.

I think that a streaming API would be better - easier to use, faster notifications and even less network traffic.
How about implementing one?
It could probably reuse the filter logic from the query API and apply it to all events.

Improve this question
8
  • Have you seen this?
    – Shog9
    Commented Jan 5, 2012 at 17:44
  • @Shog9: Hmm, no, I haven't. Mhm... that's not streaming, but HTTP push notifications. Would probably work for me if I write some stuff around it, but a simple JSON stream like the one twitter offers is much easier to consume. Also, people behind NAT or with dynamic IPs would probably benefit from an API you can just connect to.
    – thejh
    Commented Jan 5, 2012 at 17:54
  • In case your existing code isn't able to handle that so well, I'd offer building a stream server in node.js that can take all events, filter them and spit them out. :P
    – thejh
    Commented Jan 5, 2012 at 18:12
  • @Jonathan. "resources"? as in hardware resources or as in developer time?
    – thejh
    Commented Jan 5, 2012 at 18:49
  • @thejh, as in hardware/bandwidth, and developer time, see how long it took for v2 of the API.
    – Jonathan.
    Commented Jan 5, 2012 at 19:00
  • @Jonathan. I don't really think it'd take very long... at least if this API only gives you a few functions like "tag stream", "question stream" and "user stream" together with two or three options ("include full text" and "also alert on replies/comments" or so).
    – thejh
    Commented Jan 5, 2012 at 19:37
  • @Jonathan. What kinds of hardware resources are you concerned about? RAM?
    – thejh
    Commented Jan 5, 2012 at 19:39
  • @thejh, I don't know the details, I'm just saying what Kevin has previously said/implied. Also twitter rolled out their stream quite slowly,
    – Jonathan.
    Commented Jan 5, 2012 at 19:44

1 Answer 1

Reset to default
1

Developing this wouldn't be hard, but scaling it would be rough.

The problem isn't bandwidth or cpu/ram (at least, not directly) but connections. I did some investigation of COMET (aka. Long Polling) approaches (like Twitter does) for V2.0, and didn't come away very confident.

Web Hooks would be an alternative, but there are some serious DOS concerns there as (unlike with polling, long or otherwise) the client isn't dedicating any resources but we are. They're also much harder to consume than COMET approaches.

Alternatives to polling are enticing, certainly, but I don't want to roll any out that we can't guarantee reasonable performance and availability for.

Improve this answer
3
  • Connections? As long as CPU, RAM and bandwidth aren't problematic, how can the connections themselves be problematic?
    – thejh
    Commented Jan 11, 2012 at 18:29
  • Also, about scaling: Are you saying it's hard to make it handle more load by throwing more machines at it or that you can't handle many users per server with streaming connections?
    – thejh
    Commented Jan 11, 2012 at 19:57
  • Websockets for reputation change notifications are ok, but a nice JSON-over-HTTP-stream-API or so isn't?
    – thejh
    Commented Mar 30, 2013 at 23:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .