If I am developing a client side application, what kind of domain should I provide to 0Auth callback? Can it be a custom application-centric scheme (like myapp://oauthcallback/)?
For native applications (ie. Desktop apps), in order to participate in an OAuth 2.0 flow you need to embed a browser control (or otherwise instrument one so you can read the current location, including hash) and do an implicit (ie. Client Side) flow with a redirect_uri of
https://stackexchange.com/oauth/login_success. When you detect navigation to that url, you need to read the hash out and proceed like a normal client side app.
This detail is buried down at bottom of our authentication docs for the moment.
During registration, desktop apps can have anything in the Domain field. I'd suggest setting it to the same domain as your website, just in case you want a more traditional flow there in the future (of course, you can edit your app's domain at any time).
Finally, no, you cannot use custom schemes for your
redirect_uris. Only HTTP and HTTPS, and even then you should really try to use HTTPS if at all possible.