38 lines
2.2 KiB
Markdown
38 lines
2.2 KiB
Markdown
# Survivor Library
|
|
|
|
Various scripts for scraping and parsing survivorlibrary.com
|
|
|
|
Keep in mind it was meant to be a quick-and-dirty project, so things were kind of hotglued together as I went along.
|
|
|
|
## Requirements
|
|
|
|
1. Node.js + npm for `parse_html_pages.js`
|
|
- I was using `v16.13.2` (LTS) at the time of writing.
|
|
- Remember to run `npm install` before attempting to run `node parse_html_pages.js`
|
|
2. `pdfinfo` via `poppler-utils`
|
|
- Used by one of the Bash scripts to validate the downloaded PDF files
|
|
3. Bash for the various scripts
|
|
- Bash scripts were used on a Debian 10 (Buster) machine, which has it by default. Theoretically they should work on Windows (e.g. via Git Bash), but due to requirement #2 it might not work as expected.
|
|
4. `curl` - Used to actually download all the pages via `pdfUrls.sh`.
|
|
|
|
## Order of scripts
|
|
|
|
1. Browser: `get_page_urls_browser.js`
|
|
1. Add URLs into file `survivorlibrary_pages.txt`
|
|
2. Bash: `get_pages_with_pdfs.sh`
|
|
1. This one will take a while, since it downloads the HTML of all the category pages and dumps it into the `pages/` directory.
|
|
3. Node: `parse_html_pages.js`
|
|
1. Generates files such as `pdfUrls.sh` and `folderLink.sh`
|
|
4. Bash: `pdfUrls.sh`
|
|
1. Will download all the PDFs into the currently specified directory.
|
|
2. Since downloads are *not done* in parallel, it's going to take a while.
|
|
1. This was intentional to avoid any rate limits/blocks.
|
|
2. When I ran it on my server, it took about 2 days to complete. It's 13k files individually downloaded, so I hope you have patience.
|
|
5. Bash: `validate_pdfs.sh`
|
|
1. Take the list of PDFs from `validate.log` and add them to a new file: `broken_pdfs.txt`
|
|
1. Place this file into the same folder as all the unsorted PDFs.
|
|
2. Some text editor skills might be necessary to extract a clean list of filenames, as the log format is something like: `[2022-02-01 01:47:03] filename.pdf is broken`.
|
|
6. Bash: `broken_pdf_remove.sh`
|
|
1. Place this in the same directory as the unsorted PDFs and the `broken_pdfs.txt` file.
|
|
2. Run it once as just `bash broken_pdf_remove.sh`. If it looks good. Run `bash broken_pdf_remove.sh delete`.
|
|
7. Bash: `folderLink.sh` |