up vote 3 down vote favorite
share [g+] share [fb]

I've written an IRC bot that can notify channels when new questions with a certain tag are posted. However, currently I'm polling the URL #{APIURL}questions?pagesize=100&fromdate=#{@_lastQuestionsDate}&tagged=node.js&key=#{KEY} every minute. I think that a streaming API would be better - easier to use, faster notifications and even (although that's probably not so important) less network traffic. How about implementing one? It could probably reuse the filter logic from the query API and apply it to all events.

link|improve this question
Have you seen this? – Shog9♦ Jan 5 at 17:44
@Shog9: Hmm, no, I haven't. Mhm... that's not streaming, but HTTP push notifications. Would probably work for me if I write some stuff around it, but a simple JSON stream like the one twitter offers is much easier to consume. Also, people behind NAT or with dynamic IPs would probably benefit from an API you can just connect to. – thejh Jan 5 at 17:54
In case your existing code isn't able to handle that so well, I'd offer building a stream server in node.js that can take all events, filter them and spit them out. :P – thejh Jan 5 at 18:12
+1, good idea, but I doubt it will be implemented any time soon, it uses a lot of resources. – Jonathan. Jan 5 at 18:47
@Jonathan. "resources"? as in hardware resources or as in developer time? – thejh Jan 5 at 18:49
@thejh, as in hardware/bandwidth, and developer time, see how long it took for v2 of the API. – Jonathan. Jan 5 at 19:00
@Jonathan. I don't really think it'd take very long... at least if this API only gives you a few functions like "tag stream", "question stream" and "user stream" together with two or three options ("include full text" and "also alert on replies/comments" or so). – thejh Jan 5 at 19:37
@Jonathan. What kinds of hardware resources are you concerned about? RAM? – thejh Jan 5 at 19:39
@thejh, I don't know the details, I'm just saying what Kevin has previously said/implied. Also twitter rolled out their stream quite slowly, – Jonathan. Jan 5 at 19:44
feedback

migrated from meta.stackoverflow.com Jan 5 at 18:19

This question came from our site for the Stack Exchange engine powering these sites.

1 Answer

up vote 1 down vote

Developing this wouldn't be hard, but scaling it would be rough.

The problem isn't bandwidth or cpu/ram (at least, not directly) but connections. I did some investigation of COMET (aka. Long Polling) approaches (like Twitter does) for V2.0, and didn't come away very confident.

Web Hooks would be an alternative, but there are some serious DOS concerns there as (unlike with polling, long or otherwise) the client isn't dedicating any resources but we are. They're also much harder to consume than COMET approaches.

Alternatives to polling are enticing, certainly, but I don't want to roll any out that we can't guarantee reasonable performance and availability for.

link|improve this answer
Connections? As long as CPU, RAM and bandwidth aren't problematic, how can the connections themselves be problematic? – thejh Jan 11 at 18:29
Also, about scaling: Are you saying it's hard to make it handle more load by throwing more machines at it or that you can't handle many users per server with streaming connections? – thejh Jan 11 at 19:57
I think I don't really understand what the problem is... – thejh Jan 12 at 20:27
Are you there? Please say something. :( – thejh Jan 21 at 15:07
feedback

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged