About
With Stack Exchange, Inc.'s recent data dump restrictions, the official SE data dump on archive.org is dead, and single-click downloads of the entire data dump are too.
This program is meant to auto-download and auto-transform the data dump into one of (hopefully) several more convenient formats, and to contribute to data archival efforts.
Download
Download from GitHub. Instructions for setting up and running are listed in the README
Platform
Downloader: Theoretically anything able to use Python 3.10 and newer; Verified with Linux Mint 21.3 with Python 3.10.12 and Windows 10 with some version of Python newer than 3.10
Transformer: Theoretically anything able to compile C++20; It's auto-built on Linux (Ubuntu), Windows, and macOS for good measure, but not tested outside my distro at this time. Docker is also supported, expanding support to anything able to support Docker.
Contact
Use GitHub to report any issues. General questions can be posted on GitHub Discussions.
License
The code is under the MIT license; see the LICENSE file.
The data downloaded and produced is under various versions of CC-By-SA, as per Stack Exchange's licensing rules, possibly in addition to whatever extra rules they try to impose on the data dump.
Code
The code is available on GitHub, and is split in two components:
- The downloader: Python-based. This component automatically downloads the data dumps from the entire network. It's currently fully functional, and will actually download the data dumps as the feature becomes available to more users.
- The transformer: C++-based. Takes the data dump as input, and outputs it in an alternate format. Only JSON and SQLite have been implemented so far. This component is also (theoretically) compatible with most previously uploaded data dumps - however, only the recent data dump is actively supported, as they're the ones that see the most use.
