About
With Stack Exchange, Inc.'s recent data dump restrictions, the official SE data dump on archive.org is dead, and single-click downloads of the entire data dump are too.
This program is meant to auto-download and auto-transform the data dump into one of (hopefully) several more convenient formats, and to contribute to data archival efforts.
Download
Download from GitHub. Instructions for setting up and running are listed in the README.
As of 2.1.0:
- The downloader is available as a pypi package
- For x86_64 Linux users only (packaging elsewhere, especially automatically, is hard), the transformer is available as a pre-compiled binary. Other operating systems and CPU architectures still need to build from source.
Platform
Downloader: Theoretically anything able to use Python 3.10 and newer; Verified with Linux Mint 21.3 with Python 3.10.12 and Windows 10 with some version of Python newer than 3.10.
Support for macOS is limited due to macOS being functionally impossible to develop for without spending thousands of euros, which as of 2.1.0 means macOS isn't able to use the same modified geckodriver as Windows and Linux. The modified geckodriver is necessary to beat Cloudflare into submission, so macOS users will likely experience a higher Cloudflare block rate, and a higher rate of Cloudflare captcha loops. For more information, see the README for further information. It'll still work as long as you don't get blocked by Cloudflare, which you have a significantly increased risk of being. Using a non-macOS device, or at worst a Linux VM is strongly recommended.
Transformer: Theoretically anything able to compile C++20; It's auto-built on Linux (Ubuntu), Windows, and macOS for good measure, but not tested outside my distro at this time. Docker is also supported, expanding support to anything able to support Docker.
Contact
Use GitHub to report any issues. General questions can be posted on GitHub Discussions.
License
The code is under the MIT license; see the LICENSE file.
The data downloaded and produced is under various versions of CC-By-SA, as per Stack Exchange's licensing rules, possibly in addition to whatever extra rules they try to impose on the data dump.
Code
The code is available on GitHub, and is split in two components:
- The downloader: Python-based. This component automatically downloads the data dumps from the entire network. It's currently fully functional, and as of 2.0.0, has been hardened to deal with SE's increasingly aggressive Cloudflare configuration, that is now bad enough that it blocks legitimate users just based on the country they're downloading from. A fork of the abandoned undetected_geckodriver has been created to solve this particular issue
- The transformer: C++-based. Takes the data dump as input, and outputs it in an alternate format. Only JSON and SQLite have been implemented so far. This component is also (theoretically) compatible with most previously uploaded data dumps - however, only the recent data dump is actively supported, as they're the ones that see the most use.
