Wiki Migration

Forum rules
Warning !
Avoid using binary softwares from untrusted users.
Prefer compiling it yourself and verify sources.
Post Reply
Ademan
Posts: 8
Joined: Sun Dec 01, 2013 6:48 am

Wiki Migration

Post by Ademan »

Hi Everyone,
Moderators, if I'm in the wrong subforum feel free to move this, and please accept my apologies.

I'd like to outline my plan to migrate (the worthwhile) parts of the wiki, and request push access to the github wiki so that I can execute it.

1. Establish a list of pages to migrate. I know indolering wrote such a list, but I believe there are more pages that need to be included, at the very least, I've found that several pages include others. I propose using indolering's list + the wiki homepage, and traversing all internal links and include directives to establish a complete list of good pages, I'll post that list here for review prior to #2. I believe no spam pages will be traversed starting from these known good "root"s.
2. Retrieve all listed pages using action=raw. For instance http://dot-bit.org/List_Of_App_Proxies?action=raw will retrieve the wiki text. Save these files to ie List-Of-App-Proxies.mediawiki in the wiki repository. (Using the .mediawiki extension will inform github to parse it using its built in mediawiki-compatible parser)

It's simple, and should be easy, provided the person executing #2 has push permissions to the github wiki.

Unresolved: I can't figure out how to make page "include" directives work in the gollum wiki, in both the mediawiki syntax and the markdown syntax. We may want to manually include subsections which are only referenced once, in an include directive.

indolering
Posts: 801
Joined: Sun Aug 18, 2013 8:26 pm
os: mac

Re: Wiki Migration

Post by indolering »

Ademan wrote:Hi Everyone,
Moderators, if I'm in the wrong subforum feel free to move this, and please accept my apologies.

I'd like to outline my plan to migrate (the worthwhile) parts of the wiki, and request push access to the github wiki so that I can execute it.

1. Establish a list of pages to migrate. I know indolering wrote such a list, but I believe there are more pages that need to be included, at the very least, I've found that several pages include others. I propose using indolering's list + the wiki homepage, and traversing all internal links and include directives to establish a complete list of good pages, I'll post that list here for review prior to #2. I believe no spam pages will be traversed starting from these known good "root"s.
Using my list as a seed should be enough, the 500th entry has 560 views total. We should deep freeze it and reference it if we need to. You can also download this dump from a few months ago.
Ademan wrote: 2. Retrieve all listed pages using action=raw. For instance http://dot-bit.org/List_Of_App_Proxies?action=raw will retrieve the wiki text. Save these files to ie List-Of-App-Proxies.mediawiki in the wiki repository. (Using the .mediawiki extension will inform github to parse it using its built in mediawiki-compatible parser)
If there are no tables on a page, it's probably better to just use Pandoc (which won't convert tables, to my dismay).
Ademan wrote: Unresolved: I can't figure out how to make page "include" directives work in the gollum wiki, in both the mediawiki syntax and the markdown syntax. We may want to manually include subsections which are only referenced once, in an include directive.
I contacted Github, they haven't implemented the include feature from Gollum yet.

While I like this idea, with only ~40 or so pages to process, it might be faster to just c/p it over. Sadly, much of them need to be rewritten anyway, as many of them are out of date, deprecated, or redundant.
DNS is much more than a key->value datastore.

Ademan
Posts: 8
Joined: Sun Dec 01, 2013 6:48 am

Re: Wiki Migration

Post by Ademan »

If there are no tables on a page, it's probably better to just use Pandoc (which won't convert tables, to my dismay).
Github's wiki can use mediawiki-styled wiki text if you instruct it to (though I've found its table parser to be lackluster, it does "mostly" work). My proposal takes advantage of this feature. Pandoc may be a sane way forward, but I've already implemented my solution, I'll post results tomorrow.

phelix
Posts: 1634
Joined: Thu Aug 18, 2011 6:59 am

Re: Wiki Migration

Post by phelix »

nx.bit - some namecoin stats
nf.bit - shortcut to this forum

Post Reply