4

For a particular userscript feature, I need to get a list of all Stack Exchange employees. The list would be updated every day (per SE website). What is the most efficient way to do so?

I looked at users/moderators, and was hoping to filter all user_ids with is_employee set to true, but it turns out that quite a few employees don't have the diamond. (1, 2, 3)

If not this, then what is another (and efficient) way to achieve the result?

3

Updated thanks to comments by Brock Adams and user43888.


There are two methods to achieve this:

Method 1: inefficient, but future-proof as it only depends on the SE API.

  1. Query the SE API for all Meta.SE users, in groups of 100 (maximum allowed ID)
  2. Get an access token and an API key, to be able to run the above call nearly 4000 times (equivalent to the total number of Meta.SE users, as of writing)
  3. filter those with is_employee set to true, and store them in localStorage.

I have written a complete script that fetches the employee list by this method (you'll need to get an access token and API key yourself). However, note that this a lot of time depending on your internet speed. On my slow internet (4 Mbps), it took nearly twenty minutes to run through all the user IDs.

With a generous caching level (employee changes shouldn't be very sudden), you should run the above the tasks once a week. Also, keep checking the above current employee list daily, and in case of changes, run the above tasks again.

Method 2: efficient, but not necessarily future-proof, as it depends on a Google Apps Script page (which depends on SE API)

A simpler way is to simply fetch the data from an existing Google Apps Script, maintained by user43888, which already tracks the employee list, as well as their changes and deletions. I have also written the script to do so, here it is. This takes less than a minute at the maximum to generate all user ids. As described by user43888 themselves, the GA Script works as:

Once a a week, it scrapes of all MetaSE users, recording their is_employee status. Once a day, it refreshes the list by re-checking the users last known as employee.

Note that you need to set Access-Control-Allow-Origin: * headers for this to work, using extensions like this one.


Failed methods: You could look at the different users who add [tag:status-] to questions (credits: rene). As we know this facility is only available to mods and employees, and then we could filter the employees out. However, we cannot do this in a userscript, only SEDE.

  • By your math, it could take as little as 3 minutes to sift through every user on MSE. And who is/isn't an employee should change rather rarely. So, you should be caching this information, heavily, anyway. ... Go ahead and fetch all users and store the employee ids, in JSON format in a public file that your script will fetch. You need only update the file every 2-3 weeks, and you could set a cron job to do that. – Brock Adams Jul 5 '18 at 6:15
  • 2
    The comment by Brock Adams describes what my Google Apps Script does. Once a a week, it scrapes of all MetaSE users, recording their is_employee status. Once a day, it refreshes the list by re-checking the users last known as employees. – user43888 Jul 5 '18 at 11:41
  • @BrockAdams I have updated the post based on the method you posted. It gave the exact results as expected, however it took nearly 20 minutes on my slow internet :( – Gaurang Tandon Jul 6 '18 at 5:42
  • @user43888 I updated the post. Hope you don't mind the method 2 ;P – Gaurang Tandon Jul 6 '18 at 5:42
  • That's good and 20 minutes is not so bad if you have a chron job doing it during off hours and store it to a public file (not localStorage). Why are you sending an access_token? That should not be needed. Also, if you use a filter to cut out unneeded fields, it will run faster. (See this Q&A also.) – Brock Adams Jul 6 '18 at 6:02
  • @BrockAdams "20 minutes is not so bad if ..." alright. "Why are you sending an access_token?" Without an access token, all apps - which are without an access token - sending requests to SE API are dumped into the same quota. Considering that I am using up nearly 40% of that quota in one go (almost 4000 API calls), I thought getting a separate access token for my app would be better, and ensure uninterrupted access to the API. I read that here. Also, I am already using a filter=!*MxJcsv91Tcz6yRH ;) – Gaurang Tandon Jul 6 '18 at 6:35
  • Oh, sorry. I somehow overlooked the filter in the script you linked. And IIRC that access token thing will only help you if your other API calls, from the same IP address, use a token too, but one for a different app. – Brock Adams Jul 6 '18 at 6:52
  • @BrockAdams Ah, I see. I missed the part about it being the same quota per IP. Hmm, I think we can remove it then. I will check it once and update results accordingly. Thanks for the heads up :D – Gaurang Tandon Jul 6 '18 at 6:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .