4

Why?
I got tired of searching for the same duplicate questions over and over again.

What?
These are a couple of scripts I threw together in order to cut down search time for common questions and vote to close targets. dupes.txt is the output in a flat text file of the form:

Link: url Count: number of times this was cited as the original for a dupe closure
Link: url Count: number of times this was cited as the original for a dupe closure
...

It's sorted in descending order of duplicates

Usage: ./dupes.sh posts.xml
Yields: ./dupes.txt

dupes.sh

#/bin/bash
cat $1 | grep ' ClosedDate\=\"' | grep 'Possible Duplicate' | awk -F 'Possible Duplicate:</strong><br>
  <a href="' '{print $2}' | awk -F '">' '{print $1}' | grep questions > dupes.txt  
python dupes.py

dupes.py

lines = []
with open('dupes.txt', 'r') as dupeList:
    for line in dupeList.readlines():
        lines.append(line.rstrip())

links = {}
for line in lines:
    try:
        links[line] += 1
    except:
        links[line] = 1
thelinks = []
for link in links.keys():
    thelinks.append((link, links[link]))

thelinksort = sorted(thelinks, key=lambda alink: alink[1])

thelinksort.reverse()

with open('dupes.txt', 'w') as dupeList:
    for line in thelinksort:
        dupeList.write('Link: ' + line[0] + ' Count: ' + str(line[1]) + '\n')
  • Worth mentioning this can also be done with SEDE – rene May 31 '19 at 20:00
  • 1
    I was doing it locally with the data explorer API of text dumps. – RobotHumans Jun 1 '19 at 14:36
  • Yes, I saw that and that is fine. Datadumps are a bit less frequently released, quarterly-ish, while SEDE is refreshed weekly. – rene Jun 1 '19 at 16:14

You must log in to answer this question.

Browse other questions tagged .