StackUsers.com - everything you ever wanted to know but were afraid to ask. (about an SE user. almost.)

Question

About

http://stackusers.com

All Stack Exchange users at your fingertips.

The core of this application is a robust data acquisition and maintenance platform based on Soapi.CS that reliably and economically maintains a full user info data set.

Features

Current features include:

• Find Stack Exchange Users By UserName OR Email
• Query Stack Exchange users via Filtered Sortable grid and/or Entity SQL statement
• CSV to Clipboard export
• Drill down functionality on any user
• Optional client side live update via Soapi.JS
• Generate Ranked and Aggregated Custom Flair
• JSON API
• your idea here...

What's Next?

It is up to you!

• Your feature requests are welcomed.
• A planned feature, for registered users, is local hosting of JavaScript, both server and client side, that can access the underlying dataset maintained by Soapi and present it in any way you choose.

Cost

Free for everyone. Always.

If you would like to help mitigate hosting costs and possibly upgrade the hosting plan for faster response and perhaps a more open API, means to do so at no cost present themselves to you on every page of the site. ;-)

Platform

Browser Based

Contact

Sky Sanders

Code

Stack Users is the first application deployed upon a robust server side background processing engine creating using the Soapi.CS/Soapi.JS2 client libraries.

NOTE: This site is me learning MVC2 as you watch, while breaking in a new web/data host and dog-fooding Soapi all at the same time. Expect the unexpected ;-)

Answer 1

Querying Stack Exchange Users

You may leverage the Filtering and Sorting capabilities of the Ajax grid to shape results based upon the entire Stack Exchange user base.

You may export the contents of the current grid page to CSV.

Answer 2

Find Stack Exchange Users By UserName OR Email

You may find any user in the Stackiverse by the email address they supply when registering on any Stack Exchange site or by full or partial user name (display_name).

JSON API

{
    "email_address": "[email protected]",  // yes - this is a real account
    "items": [{
        "site_url": "http://stackoverflow.com/",
        "user_id": 370536,
        "association_id": "00000000-0000-0000-0000-000000000000"
    }]
}

Programmatic access to this data may be granted by IP address. Please email [email protected] with a brief explaination of the intended usage and expected traffic pattern.

Once you are authorized, you may access this data via /users/email/[email protected]

This endpoint will be discontinued shortly. Please see the announcement of the StackUsers JSON API

Answer 3

JSON API

---

Note:

I am just one developer and I do not have the time/money/resources to maintain an industrial grade public data source. If it is determined for any reason that you are not using the API in the capacity and terms indicated, your IP will be banned with a quickness.

I will shortly implement a private key system. Those interested in using the API now should monitor this post for the announcement of the process to get a key. There will be a 1 week grace period once the key system is in place for you to get a key. After which non-keyed requests will be denied and abusive IPs will be banned.

When and if a need for more resources and a way to finance it presents itself this policy will be reconsidered.

If you would like to help mitigate hosting costs and possibly upgrade the hosting plan for faster response and perhaps a more open API, means to do so at no cost present themselves to you on every page of the site. ;-)

---

User Search

   http://stackusers.com/1.0/users/{term}/search

Where term is an email address or a full or partial display name.

Example:

http://stackusers.com/1.0/users/[email protected]/search

Results:

{
    "term": "[email protected]",
    "items": [{
        "site_url": "http://stackoverflow.com/",
        "site_name": "Stack Overflow",
        "endpoint": "http://api.stackoverflow.com/",
        "user_id": 370536,
        "association_id": "00000000-0000-0000-0000-000000000000",
        "email_hash": "f3ada405ce890b6f8204094deb12d8a8",
        "display_name": "Cale Gibbard",
        "reputation": 111
    }]
}

Results are limited to the first 50 matches. There is no paging. If you cannot find what you are looking for in 50 responses, be more specific.

How It Works:

If the incoming term is an email address, all of the users with either matching email_hash or association_id are returned. This will catch the case of a user that registers on multiple sites with different email addresses. (not sure if this is a valid case, i would have to check with jeff but it is covered anyway)

If the incoming term is not an email address, a simple if(DisplayName.Contains(term)) is performed.

TERMS:

This route is provided as a service to developers of applications and websites that would like to streamline the user lookup/identification process.

Again, this route is meant to be used for interactive processes as a means for your application to identify a user in response to an interaction with your end user.

You are encouraged to persist the results of a successful match.

You should not treat this, or any StackUsers endpoint, as a bulk data store. You may build your own data store from the Stack Exchange API.

If you have any questions about these terms please email [email protected].

Answer 4

Client-side Live Updates

---

With 370,000 rows of user data to maintain, it is an impossible task to keep every record updated up to the minute.

To mitigate relatively stale data on volatile fields such as last_access_date, StackUsers offers optional client-side updates of results using Soapi.JS2.

You must opt in by either clicking the Enable Live Update button or ticking the appropriate check box in your account info page.

How it works

When Live Updates are enabled, as data is fetched for the grid, it is preprocessed and fresh data is pulled from the api by your browser using Soapi.JS2.

Differences are indexed and the dataset is presented in the grid.

Both datasets are available for viewing and fields that have changed are indicated by a visual cue.

A mouseover tooltip will present the previous and current values.

You may toggle the datasets, as shown below, by clicking the Show Live Data button.

CSV export is aware and will produce data for the current grid view.

NOTE: Sorting is server-side and is not aware of the client side live updates (yet?) so if the Show Live Data mode is active it is the underlying dataset that is being sorted and the visible results may not appear to be sorted. Just an FYI.

---

From Database

With Live Updates

Convenient Tooltips

Details

You may monitor all requests issued by Soapi.JS2 by opening the StackUsers Client Log window.

All requests are batched to make the client side processing economical and fast.

The dot next to the Soapi icon acts as a visual cue as to the current state of client side processing.

All site data and site icons are cached and served from StackUsers to mitigate load on the API server(s?) as well as reducing rate-limit consumption.

Answer 5

Drill-Down

---

An evolving feature of the StackApps platform is drill down on any user. The utility and polish of the features are somewhat affected by the fact that I am cutting my teeth on the completely different, and refreshing, paradigm that MVC presents. So bear with me. And email me suggestions as to what you would like to see.

Current features

Standard link out to profile page

---

Visual online status indicators

When backed with live update you will be able to see up-to-the-second last_access_date.

---

Aggregated User Info PopOut

This feature needs a better head for design than mine and input from you as to what kind of data points would be interesting.

It seems that efficient use of space, which I am obviously not adept, is the key to success here.

If you would like to get involved, open Photoshop, Gimp or notepad and mock up some ideas. Your input is actively solicited.

Email requests, suggestions, rough sketched, full blown professionally designed royalty free designs and money to [email protected] ;-)

The about page 'thank you' list needs some entries.

Answer 6

CSV output

Any grid on StackUsers provides an Export CSV to Clipboard button that will copy the data from the current grid view. This includes all user fields, not limited to those shown in the grid.

The output is standard CSV and can be opened or consumed by any tabular data editor.

Anonymous StackUsers can page 50 rows, while registered users are provided larger datasets.

If you have registered and would like a record count increase to satisfy a particular usage requirement, just email me @ [email protected].

Implementation

For those interested in converting a homogeneous array of JavaScript objects to CSV, here is the code that I use on StackUsers.

It is freely offered without restriction.

jsonToCSV

function jsonToCSV(sourceRows, omitHeader, fieldsToIgnore)
{

    /// <summary>
    /// Converts an array of JSON objects to CSV.
    ///</summary>
    /// <param name="sourceRows" type="Object[]">
    /// An array of objects containing a single level of scalar fields.
    /// e.g. [{ foo: "bar", fu: true }, { foo: "baarbar", fu: false }]
    /// 
    /// The first element in the array will define the header row.
    /// </param>
    /// <param name="omitHeader" type="Boolean" optional="true">if true, header row is not emitted</param>
    /// <param name="fieldsToIgnore" type="Map" optional="true">a map of fields to ignore e.g. { field1:1,field4:1 }</param>

    /// <returns type="String"></returns>
    /// <author name="sky sanders" contact="http://skysanders.net/subtext" date="2010-09-19"/>

    function quote(value)
    {
        return '"' + value.replace(/"/g, '""').replace(/\r/g, "\\r").replace(/\n/g, "\\b") + '"';
    };
    function pad(n)
    {
        return n < 10 ? '0' + n : n;
    };

    var header = "";
    var rows = "";
    var headerComplete = false;

    for (var i = 0; i < sourceRows.length; i++)
    {
        var firstElement = true;
        var row = "";
        for (var key in sourceRows[i])
        {
            if (sourceRows[i].hasOwnProperty(key))
            {
                if (fieldsToIgnore && (key in fieldsToIgnore))
                {
                    continue;
                }

                if (!headerComplete)
                {
                    if (!firstElement)
                    {
                        header = header.concat(", ");
                    };
                    header = header.concat(key);
                };

                if (!firstElement)
                {
                    row = row.concat(", ");
                };

                var value = sourceRows[i][key];

                if (typeof value != 'undefined' && value !== null)
                {
                    if (value instanceof Date)
                    {
                        var dateResult = value.getUTCFullYear() + '-'
                                + pad(value.getUTCMonth() + 1) + '-'
                                + pad(value.getUTCDate()) + 'T'
                                + pad(value.getUTCHours()) + ':'
                                + pad(value.getUTCMinutes()) + ':'
                                + pad(value.getUTCSeconds()) + 'Z';
                        row = row.concat(dateResult);
                    }
                    else if ((value instanceof Boolean) || !isNaN(value))
                    {
                        row = row.concat(value.valueOf());
                    }
                    else
                    {
                        row = row.concat(quote(value.valueOf()));
                    }
                }
                firstElement = false;
            }
        }
        rows = rows.concat(row).concat("\r\n");
        headerComplete = true;
    }
    return omitHeader ? rows : header.concat("\r\n").concat(rows);
};

Answer 7

Just wanted to ask / mention a few things and I thought an answer would be more appropriate:

• In regards to an API: I understand your concern about bandwidth. Would you be willing to do what the SO team did and offer a private beta while you work out a key/throttle method?
• In regards to hosting: how will that work? Are you basically just giving registered users XX MB for JavaScript hosting or what? Could StackCenter be moved there if I removed the PHP code?
• Regarding apps: I saw that somewhere... is there some kind of app system you're setting up?

_{Answering questions like this in comments is silly. We can carry on a conversation here...}

[sky]:

r.e. Bandwidth: the usage itself is certainly a concern, but the salient concern is DOS, whether it be a hard shutdown via abuse or an administrative shutdown triggered by hogging the resources of the shared host. I will spend a few hours, soon, coming up with a secure method of providing access to certain data.

Currently I provide access to users/email/{address} on a per-IP authorized basis. Please see this post for more details.

r.e. Hosting: I intend to provide an environment that will support direct access to the datastore via Ajax (same domain, I do not have the resources to expose JSONP to external consumers) as well as server hosted scripting languages (and perhaps even c# using a sandboxed mono compiler I built last year) for saving state for your javascript apps. The server code will be limited and not suitable for developing apps, only for specifying connector endpoints for the browser based javascript/flash/silverlight etc.

[George]
Flash won't be a problem because it supports cross-domain policy files.

As for the API, no hurry :)