Add docs into the repo

[jtojnar]

I downloaded the HTML from live site and created a script (markdownify.py) that will convert them to Markdown. This PR attempts to make everything work. We will remove/update the outdated content in a follow-up PR.

To regenerate the markdown files, run python3 markdownify.py in the SimplePie branch after installing python3-beautifulsoup4, python3-pypandoc and prettier. Or if you have Nix, you can just run nix-shell -I 'nixpkgs=channel:nixos-unstable' -p 'python3.withPackages (ps: with ps; [ beautifulsoup4 pypandoc ])' nodePackages.prettier --run 'python3 markdownify.py'.

Currently I am using Zola as the generator, as I am most familiar with it. A different SSG can be used if preferred.

To preview run zola serve in the docs subdirectory.

TODO

• Have index auto-generated for blog.
• Move h1 out of individual pages into template.
• Strip blog junk.
• Strip wiki junk.
• Ensure links are correct.
• Figure out what to do with the demo.
• Add README to the docs/ directory.
• Remove the old files.
• Set up ApiGen.
• Set up GitHub pages CI.
  • https://docs.github.com/en/pages/configuring-a-custom-domain-for-your-github-pages-site/managing-a-custom-domain-for-your-github-pages-site

Fixes: #543

[Art4]

Great work so far. 👍 What do you think about having the website on an orphan gh-pages branch to maintain the website and deployment tools independent from the master branch?

We should also put the name change of Sam on the todo list (#543 (comment))

[jtojnar]

Using a different repo would be even cleaner. But then it is harder to do coordinated changes (e.g. updating tutorial to new API), as Git & GitHub does not really support this. Plus “out of sight out of mind” applies. Since the goal of this effort is to make updating the docs simpler, I think using the same branch is probably the best choice here. Zola is essentially zero-config (only really requires setting the site URL config.toml and few templates) so it should not bloat the repo much.

Opened #757 for the todos.

[jtojnar]

@mblaney The last remaining question content-wise is what to do with the demo:

• Upload the static website to the FTP server (instead of GitHub pages) using GitHub actions, keep the demo as a PHP script.
• Move the demo to a demo.simplepie.org subdomain, occasionally updated manually, and point the website to that.
• Move only the backend part of the demo to a subdomain, and convert the frontend to use AJAX.
• Remove the demo completely.

We should also decide on the host we want to use:

• Use the existing web host
  • ➕ Allows running demo using PHP.
  • ➕ Supports redirects using .htaccess.
  • ➕ No need to fiddle with DNS.
  • ➕ Trivial setup.
  • ➖ No preview support.
• Netlify
  • ➕ Supports _redirects.
  • ➕ Will create a subdomain for each pull requests as a preview.
  • ➖ DNS changes necessary.
  • ➕ Easy setup.
  • ➖ Free plan only allows a single user to access administration.
  • ➕ though most things can be configured in a config file in the repository.
• Cloudflare Pages
  • ➕ Supports _redirects file.
  • ➕ Will create a subdomain for each pull requests as a preview.
  • ➖ DNS changes necessary.
  • ❓ Not sure how hard to set up, I have not used this service yet.
• GitHub pages
  • ➖ No support for redirects.
  • ➖ No preview support.
  • ➖ DNS changes necessary.
  • ➕ Easy setup.

[mblaney]

thanks @jtojnar I wouldn't go for the subdomain option because that involves finding someone who can do DNS changes. It looks like the domain is owned by Automatiic, if someone here wants to help make that happen hopefully they will jump in, otherwise I would say continue with the other options.

By process of elimination that means staying with the current host and either removing the demo or possibly using the github action you've suggested. Happy for you to decide on that.

[jtojnar]

@mblaney If we stay with the current host, keeping the demo working is not that hard.

Do you have FTP or SSH credentials for the host? Depending on that, we will need to choose either https://github.com/marketplace/actions/ftp-deploy or https://github.com/marketplace/actions/web-deploy-anything.

And either way, we will need to set up credentials on GitHub: https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository

[mblaney]

I have FTP credentials but I'm not a project owner, so not sure how far I will get. Let me know when you're ready and I can try adding the credentials.

[jtojnar]

I have tested this on my repo, seems to work well, including the demo: http://simplepie.ogion.cz/ (use test as username and password)

So it should be ready now.

@mblaney Now you should:

1. Back up the contents of the FTP server.
2. Add the following repository secrets on https://github.com/simplepie/simplepie/settings/secrets/actions:

FTP_PASSWORD
FTP_SERVER
FTP_USERNAME

3. Merge this PR. Please do not squash the commits so that the HTML files are preserved in the git history to allow us to fix Markdown conversion issues if we notice them later.

[mblaney]

nice one @jtojnar your demo site looks great! I get a 404 for that actions url though?

[jtojnar]

@mblaney This is what I see in my fork:

Or maybe we need someone with member status on the repo?

[Art4]

@jtojnar I noticed some styling errors an the API Docs page. Take a look a the left sidebar:

http://simplepie.ogion.cz/api/

This is how it looks like atm: http://simplepie.org/api/

[jtojnar]

@Art4 Tweaked the style, should be fixed now.

[Art4]

Thank you @jtojnar. I also noted some other things:

1. The source code view has no indentation: http://simplepie.ogion.cz/api/source-src.SimplePie/#690-695
   
2. The blog posts are randomly shuffled and don't show the date somewhere. There is also no pagination, but imho that's not important. http://simplepie.ogion.cz/blog/

[jtojnar]

One concern would be increased repo size:

1. master branch: 8.5 MiB (4.61 MiB compressed)
2. this PR: 14.7 MiB (7.25 MiB compressed)
3. this PR without the original HTML files: 13.9 MiB (6.52 MiB compressed)

Methodology

1. Cloned the repo with git clone [email protected]:simplepie/simplepie.git
2. In the copy of ①, I fetched the PR: gh co 756
3. In copy of ②, I squashed the Remove original website source commit into Convert website into a static site and fetched the branch into a copy of ① with git fetch ../simplepie2 static-docs-clean and git checkout FETCH_HEAD

Then I ran git fsck; git prune; git gc

For the compressed sizes, I ran git clone -v file://$PWD/simplepie3 $(mktemp -d)

This is not that drastic but it would be a permanent cost going forward so perhaps we should store the backup somewhere else.

Also I noticed just wiki/reference takes 1.1 MiB in the repo unpacked. Maybe we will not want to include it and move the content to PHPDoc comments.

[jtojnar]

@mblaney Actually, looks like WordPress supports export without the need to access the database. Are you able to log into the administration and get the export from http://simplepie.org/blog/wp-admin/export.php? And are you able to download the contents of the FTP server and upload it as an archive here?

[skyzyx]

Working on this now. Sorry for the delay.

[skyzyx]

@mblaney, @jtojnar:

Added these secrets:

• SFTP_USERNAME
• SFTP_PASSWORD

Added these variables:

• SFTP_SERVER
• SFTP_PORT

[skyzyx]

Backing up the public_html directory. So far, it's several GBs. I'll update when the backup, tarballing, and uploading is complete.

[jtojnar]

@skyzyx You can also try uploading the following PHP script and run it to create an archive on the server. It should be much faster than downloading individual files:

<?php
set_time_limit(0);
error_reporting(E_ALL);

$zip = new ZipArchive();
$zip->open(__DIR__ . '/simplepie_website_backup.zip', ZipArchive::CREATE);
echo 'Creating zip archive<br>';
$directory = new \RecursiveDirectoryIterator(
    // Or change the directory path.
    __DIR__,
    FilesystemIterator::KEY_AS_PATHNAME | FilesystemIterator::CURRENT_AS_FILEINFO | FilesystemIterator::SKIP_DOTS
);
$iterator = new \RecursiveIteratorIterator($directory);
foreach ($iterator as $info) {
    echo 'Adding ' . $info->getPathname() . '<br>';
    $zip->addFile($info->getPathname());
}
$zip->close();
echo 'Finished<br>';

[skyzyx]

It took some time to finish the download, but it finally completed at 7.4 GiB. I removed the cache files, tarred the directory, and gzipped it with -9. The resulting archive is just under 50 MB. I uploaded it to the root of the SFTP server.

/public_html_2023-05-11T18-20-00Z.tar.gz

[mblaney]

thanks @skyzyx that's great. @jtojnar this PR just needs updating for the SFTP change?

[jtojnar]

@mblaney I wanted to use the files from the backup as a base for generating since the scraping is not perfect (wget I used for mirroring returns a different set of pages each time and I noticed few places where the wiki software produces messed up HTML).

But since I do not have an access to FTP so I cannot access the backup. Could you please re-upload it somewhere publicly available?

Also do you have access to the WordPress administration? The export would be helpful for similar reason.
It should be available on the following URL if you are able to sign in:
http://simplepie.org/blog/wp-admin/export.php

Otherwise, if the WordPress installation is too broken, could you try getting a database dump, e.g. by uploading a tool like Adminer and exporting the database using the credentials from wp-config.php file?
https://www.adminer.org/en/

[mblaney]

hi @jtojnar the backup is just the wordpress install, so nothing usable like that is it? It appears to be too broken to log in, and I don't want to re-upload because it contains login credentials (even though I can't use them).

I can try the database dump if you like, but not sure that will provide anything better than scraping?

[jtojnar]

@mblaney IIRC the wiki system stores the content in the directory so that is the main thing I am after. The issue with scraping is that it is incomplete – there are some pages missing or returning error 500. I managed to get some of them out of internet archive but DB dump would be preferred since we can never be certain if wget did get everything.