Preferred language for NMControl-like things

[biolizard89]

We've been using Python for NMControl, which has worked reasonably well. Independently, Hugo has a parallel implementation (ncdns) which is in Go. And there's DNSChain (Coffeescript). I'd like to get some feedback on what language is preferable for Namecoin application stuff (i.e. the things that NMControl does now).

Personally I'd like to see a Rust implementation, but Rust isn't really mature enough at the moment. Go has better type safety than Python. On one hand I dislike Go's usage of return values for errors... but on the other hand in Python it is difficult to know which exceptions to catch. I haven't used Coffeescript.

Thoughts?

[brandonrobertz]

What's the goal here? To develop a fully-featured namecoin-enabled DNS daemon? Simply NMControl? Etc.

It might be worth looking at why adoption of these different applications hasn't really happened and then try to come at the problem from the perspective of fixing it, as opposed to language-first.

$0.02

EDIT: Coffeescript just compiles down to JS. That said, I wouldn't be comfortable using DNSChains without an audit for common CS problems due to things like implicit return.

[biolizard89]

Coffeescript just compiles down to JS.

What's the goal here? To develop a fully-featured namecoin-enabled DNS daemon? Simply NMControl? Etc.

It might be worth looking at why adoption of these different applications hasn't really happened and then try to come at the problem from the perspective of fixing it, as opposed to language-first.

$0.02

NMControl's goal is to be a Swiss army knife of functionality relevant to naming systems like Namecoin. So DNS daemon, REST interface for end user applications that want to read Namecoin names, SOCKS proxy for Tor/I2P .bit, etc. Generally I think focusing the three projects' developer efforts on one project will produce better functionality and usability.

The main differentiator between the three appears to be choice of language, along with some side effects of that, such as ncdns having access to better DNS libraries than NMControl since Go has DNSSEC libraries. (Of course, a Go DNS server can call NMControl's REST API, so that's not by itself a big difference.) Also ncdns is mainly focusing on d/ applications, and DNSChain is mainly focusing on delegated trust applications (as far as I can tell), while NMControl is somewhat more namespace-agnostic than ncdns and somewhat more local-resolution-focused than DNSChain.

(The above is not intended to thoroughly describe any of the three. I'm obviously leaving out a lot of details.)

I'd just like to get some feedback on language choice before a lot of effort continues to be expended on the three applications.

[sugarpuff]

I'd like to see four different implementations, perhaps five, each in a different language.

I dislike Python probably at least as much brandonrobertz dislikes CoffeeScript. Go seems reasonable, and Rust is definitely praiseworthy.

That said, if you go with Python, I could help in ways other than writing code (as we've discussed). And with how long it's taking to figure this out, it's not clear I will have time to code in any language.

[hla]

It seems to have become apparent that reproducible builds of Python will be infeasible for the foreseeable future. Apparently, the Tor project tried and failed to obtain reproducible builds of Python, and had relative ease obtaining reproducible builds of Go. This puts the future of a Python-language middleware stack in question.

Go: I'd like to make the case for using Go in the hope of obtaining consensus. (Obviously, as the author of ncdns, a Go-language implementation, I have an innate bias.) I think there are many points in favour:

• Go has been ported to the usual platforms (Linux x86, Linux amd64, OS X, Windows).
• Go can produce monolithic binaries which don't require the installation of any runtimes.
• Reproducible builds can be obtained.
• Go has excellent support for performant network IO while avoiding the need to write in an explicitly asynchronous style such as the continuation passing style, keeping code clear and readable.
• Because all code written in Go uses the same concurrency and I/O mechanisms, code reuse is maximized and the full ecosystem of Go libraries can be leveraged without any issues caused by them using different I/O reactors, etc.
• My implementation, ncdns, is the only complete* implementation of domain-names.md, the draft revised d/ specification. It is complete, ready to use, and DNSSEC-enabled. (It was even spontaneously found and tweeted by the author of libsodium, suggesting it is somewhat marketable and easy to use.)
• This domain-names.md implementation is testable, and corresponds to a battery of tests I wrote in a language-independent specification format.
• Go is a simple, easy-to-learn language which is unlikely to constitute a significant barrier to new contributors.
• Go has a mature, widely used and maintained DNS library. Whereas with NMControl, concerns over the level of maintenance of the DNS library has led to an intention to tightly couple NMControl with Unbound. I think this coupling is undesirable and I'm not convinced it will be effective in outsourcing DNS concerns anyway.
• Go is a type-safe language.

*The only hole in ncdns's implementation of the specification is the absence of certain suppression rules. This is basically harmless.

Rust: Rust could be a viable alternative, but the ecosystem is still developing and maturing, and there appears to be a higher degree of variance in e.g. I/O systems which may somewhat fracture the ecosystem and inhibit code reuse. I also don't know what the situation is with e.g. DNS libraries. The provision of things such as REST APIs may be desirable, and Rust's web development ecosystem is also still developing. I should also add that, as laudable as Rust is as an effort, it is essentially an attempt to create a viable replacement to C++ which is safer and saner. As such, the use of Rust doesn't necessarily make sense where a higher-level language is adequate from an engineering perspective. Python, Go and Java, for example, are all viable languages for implementing Namecoin middleware, whereas implementing a web browser in any of those languages is probably not sane. In other words, Rust would be overkill for the project. The slightly more sophisticated type system Rust probably has over Go isn't important enough to merit the use of a lower-level and more complex language, and it would also lead to a need to research and source appropriate I/O systems, DNS libraries, etc.

CoffeeScript (JavaScript): This is not necessarily a completely terrible idea. Node.js is a reasonable, and seemingly pretty lightweight runtime which could be easily redistributed. I don't know whether Node.js can produce reproducible builds. However, I find the continuation passing style of I/O programming distasteful given that the technology to do better has been available for decades (via things like coroutines; see Go, Stackless Python, Erlang, etc.)

I suspect that the main reason to use Node.js for anything, really, is that you have some terribly pressing need to run the same code on both web browser or server, or you have some existing large JavaScript codebase which you need to leverage. I don't think these concerns apply here.

[biolizard89]

To build on Hugo's post:

I looked up the Tor Pluggable Transports Gitian descriptor yesterday. They are easily building Go reproducibly, but they apparently have not figured out how to build Python reproducibly for Windows; they're literally feeding "python.msi" (downloaded from the Python website) into the Gitian script as an input. Mike Perry and the other Tor developers have significantly more funding and experience with this stuff than we do (even given that we have Armory resources backing Python), and if they haven't had any luck at this in the years that they've been doing Gitian stuff, I don't think we have a great chance at doing it either. Granted, they are using Python2 rather than Python3, and Joseph reported that Python3 seemed to be better at cross-compiling... but even so, if it were as simple as porting to Python3, I would have expected Tor to simply do that. As it stands, the Python developers can backdoor several of the pluggable transports in the Tor Browser Bundle quite easily.

I cannot find any examples on GitHub or Google of either Coffeescript or Node.js working with Gitian. That doesn't mean it's necessarily impossible, but it means it seems to be completely uncharted territory, which I'd prefer not to build on (especially given that Go is working just fine for Tor).

I don't have any significant experience coding in Go, Rust, or Coffeescript. I'm okay with investing some time in learning Go or Rust; I have a strong hatred of anything associated with Javascript (having wrestled with the Convergence code) and therefore am against using Coffeescript.

To clarify Hugo's point about DNS libraries: Python doesn't have any good DNS libraries (python-unbound is not that great at what we want; PyDNS and PyMDS are even less suitable). It is, however, possible to use a DNS library in Go (e.g. Hugo's madns) with a Python NMControl backend (which is what I was planning on doing prior to looking up Tor's Gitian results). Using ncdns could be argued to be cleaner than a split-language implementation.

Rust and Go both seem to be better at making it hard to accidentally write bad code than Python or Coffeescript. Rust is better than Go in this respect.

Rust is immature, not packaged in Fedora yet, and does not have a robust ecosystem. Things like Flask and Bottle don't seem to exist in Rust. I'm not sure about DNS libraries. I would like to use Rust in 2-3 years, but it doesn't seem ready at this point.

In short, I would advocate that we seriously consider making the official middleware implementation the Go-based implementation. We could do it as a trial for a little while to see how it goes -- the NMControl code will still be there if we decide it was a bad idea.

Cheers.

[biolizard89]

In the interest of transparency, here is a chatlog from #namecoin-dev (irrelevant side chatter removed):

[20:29:06] <Jeremy_Rand> hl: viewtopic.php?f=5&t=2406
[20:36:09] <brand0> it looks like dnschains already has a working implementation in JS
[20:37:40] <Jeremy_Rand> brand0: I mentioned that in the thread, didn't I?
[20:38:03] <Jeremy_Rand> FWIW I absolutely hate coding in Javascript, and I don't think Coffeescript is much better.
[20:38:09] <brand0> oh you're bioblizard?
[20:38:14] <brand0> sorry I didn't know
[20:38:20] <Jeremy_Rand> brand0: yep, that's me
[20:40:03] <Jeremy_Rand> brand0: Basically, there are 3 implementations (NMControl, ncdns, and DNSChain), and having 3 time the effort expended on such things helps no one
[20:41:31] <Jeremy_Rand> brand0: I wish Rust were a bit more mature, because it's better than all 3 of the existing implementations in terms of both speed and safety
[20:41:56] * Jeremy_Rand needs to spend a couple weekends learning Rust for fun
[20:42:58] <brand0> of course DNChains had to use fucking coffeescript
[20:43:30] <Jeremy_Rand> brand0: I take it you're not a Coffeescript fan?
[20:44:26] <brand0> Jeremy_Rand, I've found coffeescript code to be extremely error prone, mostly due to implicit return ... like lisp, the last statement in a function is the return value for the fucntion by defauly
[20:45:08] <brand0> I've had to run down a lot of memory leaks because someone forgot to end their function with return, and instead returned a value accidentally
[20:45:16] <Jeremy_Rand> also, fwiw, NMControl, ncdns, and DNSChain have overlapping functionality, but not totally overlapping. I.e. for each of the three, there are some things that only that one can do well
[20:45:36] <Jeremy_Rand> brand0: ouch. Just from that fact, I would veto any use of Coffeescript :/
[20:47:38] <Jeremy_Rand> brand0: you should post in that forum thread
[20:48:14] <brand0> just did
[20:58:10] <Jeremy_Rand> brand0: replied
[21:03:32] <sugarpuff> Jeremy_Rand, brand0: replied
[21:03:45] <sugarpuff> btw, i can't post under the .org domain, but i can under the .info domain. weird
[21:05:16] <sugarpuff> and made several typos in the process. brain not coordinating with hands. "is you go"
[21:10:42] <brand0> sugarpuff why do you dislike python?
[21:11:27] <sugarpuff> brand0: https://www.taoeffect.com/other/nupic/# ... and-python
[21:12:52] <sugarpuff> also, that link is just part of it. I think the syntax is extremely poorly thought out. Random special keywords, junk like `if program == __main__` (or w/e), and totally unnecessary use of colons in random locations
[21:13:39] <brand0> I really don't consider whitespace that big an issue in python
[21:13:49] <sugarpuff> i didn't say whitespace
[21:13:56] <Jeremy_Rand> sugarpuff: what's your opinion of Go's usage of return values to indicate errors? Seems like that practice incentivises sloppy coding
[21:14:22] <sugarpuff> err, actually i did say whitespace in that link, but i've come around to being OK with it (as CoffeeScript also makes use of whitespace in the same way)
[21:14:31] <brand0> sugarpuff, i totally agree with the packaging problem
[21:14:49] <Jeremy_Rand> pip doesn't validate signatures
[21:14:52] <brand0> anyone whose ever tried to have a non-developer install/run a python program felt the burn
[21:15:03] <Jeremy_Rand> (No idea what the other packaging systems for the other languages do there)
[21:15:17] <brand0> at least with c++ you can just ship a binary like bitcoin does
[21:15:45] <Jeremy_Rand> brand0: I've had no trouble with installing Python stuff on Linux. On Windows it's annoying, but that's because Windows is crap.
[21:16:19] <Jeremy_Rand> err, I did have trouble with Debian Squeeze and Python. But that's because Squeeze is anncient
[21:16:31] <brand0> yeah on Linux it's easy ... but most ppl running linux could figure out how to run/install *anything*
[21:16:46] <brand0> even a ruby app
[21:16:57] <sugarpuff> Jeremy_Rand: i don't know as I haven't used Go much, but if it's anything like C's use of returning values to indicate errors I think it's a good thing. I don't like how exceptions are implemented in any language except Java. See also: https://www.taoeffect.com/blog/2008/11/ ... nventions/
[21:17:03] <Jeremy_Rand> If there were a GUI package manager on Windows like there is on Linux, Windows users could figure out how to install anything too
[21:18:01] <Jeremy_Rand> sugarpuff: my understanding is that in Go, a lot of programmers just don't check the error value, which will cause horrible things to happen later that are very difficult to debug
[21:18:10] <Jeremy_Rand> even printf returns an error value that you're supposed to check
[21:18:52] <sugarpuff> Jeremy_Rand: yeah, i agree that's crazy, which is why wrote those macros for C. they really solve the problem completely for me. If you can't do that in Go then it's a problem.
[21:19:04] <brand0> I'd call go'd error implementation annoying, but it works if you utilize it
[21:19:40] <brand0> in python the print function can blow up depending on environment and on what you're printing
[21:19:40] <Jeremy_Rand> Are we all agreed that Rust would be a really good solution if it were, say, 2 years more mature?
[21:20:33] <sugarpuff> Jeremy_Rand: i dunno. I think Rust is mature enough now to replace C++.
[21:20:52] <sugarpuff> Terminal.com uses Rust to power all of their backend
[21:20:54] <brand0> I thin with rust you're going to have cross-platform distibution difficuties
[21:20:54] <Jeremy_Rand> sugarpuff: Rust isn't even packaged in Fedora. And Fedora is usually the first for everything.
[21:21:16] <Jeremy_Rand> brand0: doesn't Rust use LLVM as the backend for the compiler?
[21:21:29] <sugarpuff> Jeremy_Rand: yes, rust uses LLVM
[21:21:50] <Jeremy_Rand> Yeah, I mean, if it uses LLVM, it shouldn't be super hard to target arbitrary OS's
[21:22:15] <sugarpuff> Jeremy_Rand: with Rust you get access to all existing C and C++ code out there too
[21:22:49] <sugarpuff> any new project that would use C++ today should use Rust instead
[21:22:51] <brand0> from what I understood deploying rust to windows without MSVC was a major pain in the ass if possible at all (depends on application)
[21:22:52] <Jeremy_Rand> Yes. But C/C++ code is a pain in the rear compared to languages that actually enforce sanity.
[21:23:18] <brand0> that seems to be a ecosystem immaturity problem tho
[21:23:18] <Jeremy_Rand> brand0: hmm, I hadn't heard that
[21:23:47] <Jeremy_Rand> well yes, the ecosystem is immature, there arent a lot of good libraries yet, and it's not packaged in any major Linux distros yet
[21:24:29] <Jeremy_Rand> e.g., I looked for something like Flask or Bottle for Rust a couple months ago, the options were really lousy
[21:25:08] <Jeremy_Rand> I'm unaware of any DNS libraries for Rust
[21:25:23] <Jeremy_Rand> certainly nothing as well-tested as Go's DNS libs
[21:25:37] <Jeremy_Rand> that will fix itself over time
[21:26:04] <brand0> one way to make sure a project is DOA is making it insanely difficult to build/install
[21:27:07] <Jeremy_Rand> brand0: Python is middle of the road by that metric. I think Go would be better under that metric, but I don't have any Go experience, so my opinion means little
[21:27:32] <brand0> Jeremy_Rand, I agree with that. Go was easy for me to get goin when I ran a windows machine, even easier in Linux
[21:27:33] <Jeremy_Rand> Can Go projects be built reproducibly for Windows in Gitian?
[21:27:59] <sugarpuff> Jeremy_Rand: Rust has DNS libraries, they're just in C++
[21:28:09] <sugarpuff> getdns is very mature
[21:28:30] <brand0> does go have c/c++ interop like rust does?
[21:28:50] * sugarpuff doesn't know
[21:28:50] <Jeremy_Rand> C/C++ is easy to do in Gitian. Python difficult, but jbisch has managed it. Rust and Go I have no idea if anyone's even tried.
[21:30:08] <Jeremy_Rand> I don't know about Go/C/C++ interop. I don't like using C++ libraries directly if it's avoidable, they're generally complicated and difficult to use correctly.
[21:30:13] * Jeremy_Rand cough OpenSSL
[21:30:46] <jbisch> Jeremy_Rand: Well, actually the Linux deb package is using the Debian reproducible toolchain, so that's not Gitian. And Windows isn't working because of the ssl module issue I mention elsewhere.
[21:31:15] <Jeremy_Rand> jbisch: right, but my understanding is that outside of the ssl module, things appear to work?
[21:31:25] <Jeremy_Rand> granted, your process is insanely convoluted
[21:31:42] <Jeremy_Rand> and the fact that it's necessary, does not reflect well on Python IMO
[21:32:34] <jbisch> Outside the ssl module. I can get an exe that runs without any need to install Python or other dependencies. But I haven't even looked at wrapping the Windows exe generation process in a Gitian descriptor, because I just wanted to get a working exe in the first place.
[21:33:00] <jbisch> Should be a comma after the first part there.
[21:33:16] <Jeremy_Rand> ok
[21:33:41] <Jeremy_Rand> so yeah, it's convoluted, but maybe workable with substantial effort
[21:33:51] <Jeremy_Rand> in any event, that doesn't look great for Python
[21:34:00] <Jeremy_Rand> I'm just not sure if Go/Rust is even worse
[21:34:57] <jbisch> I don't know about Go/Rust, but you basically need a method of doing what Python calls "freezing" everything to make it easy for users to install before you can even look at making it reproducible.
[21:35:58] <Jeremy_Rand> FWIW, searching GitHub for Rust Gitian yields no useful results
[21:36:02] <Jeremy_Rand> same for Google
[21:36:53] <jbisch> I'm not surprised. Probably nothing for Go either.
[21:37:22] <brand0> has anyone reached out to the DNSChains or ncdns developers?
[21:37:39] <brand0> DNSChain*
[21:37:47] <Jeremy_Rand> jbisch: I just found this: https://trac.torproject.org/projects/tor/ticket/15265
[21:38:03] <Jeremy_Rand> looks like Yawning has some Go stuff working with Gitian for TorBrowser?
[21:38:22] <Jeremy_Rand> brand0: sugarpuff is DNSChain dev, hl is ncdns dev
[21:38:33] <brand0> oh shit, okay
[21:38:48] <brand0> so if everyone here could come to a consensus, then that could be good for all projects
[21:39:18] <Jeremy_Rand> well, we would also want Phelix involved in said consensus since he's one of the two lead NMControl devs (along with me)
[21:39:25] <Jeremy_Rand> and last I heard he's away for the next week?
[21:39:28] <brand0> of course
[21:39:51] <brand0> I didn't know projects other than NMC/NMControl were represented here ... that's awesome
[21:39:58] * brand0 is still learning
[21:40:05] <Jeremy_Rand> learning is fun
[21:40:47] <brand0> lol, well if you stop you die
[21:42:22] <jbisch> Jeremy_Rand: It looks like Yawning is just touching the files with the reference datetime and otherwise isn't doing anything special.
[21:42:49] <Jeremy_Rand> So, it sounds to me like the general feeling is that Rust will be awesome in 2 years or so, less consensus on whether it's good now. Seems like most of us don't actively hate Go (although I don't really like it much either). brand0 and I actively hate Coffeescript, sugarpuff actively hates Python. I know hl likes both Go and Rust, but I think he thinks Rust isn't mature enough yet.
[21:43:13] <brand0> so that leaves Java and JavaScript
[21:43:44] <Jeremy_Rand> brand0: I despise both Java and Javascript
[21:44:06] <Jeremy_Rand> Java has a few redeeming qualities. Not many though.
[21:44:15] <brand0> I'll give you Java, but JavaScript is pretty much the widest deployed system out there
[21:44:51] <Jeremy_Rand> brand0: yeah, and Javascript's broken architecture is still annoying web developers to this day, thanks to Netscape's 4AM last minute design decisions
[21:45:15] <brand0> yeah, you can work around that, but I agree it can be problematic in the wrong hands, like the coffeescript thing
[21:45:44] <sugarpuff> I hear OCaml is nice
[21:45:52] <sugarpuff> but nobody uses it
[21:45:59] <brand0> i've heard the same sugarpuff
[21:46:01] <Jeremy_Rand> brand0: my highest priority with choosing a language is to make it difficult to accidentally write bad code. Rust is a clear winner there.
[21:46:20] <brand0> Jeremy_Rand, yeah I understand that
[21:46:27] <sugarpuff> Jeremy_Rand: that's what I've heard about Rust. Multiple people have said its compiler *teaches* you
[21:46:55] <brand0> go's type system is pretty strict
[21:49:13] <Jeremy_Rand> jbisch: Looks like Tor isn't having much trouble with Go in Gitian, at least for Linux builds https://github.com/Yawning/tor-browser- ... sports.yml
[21:49:23] <Jeremy_Rand> everything Tor is doing there is really straightforward
[21:51:05] <Jeremy_Rand> looks amazingly simple for Windows too
[21:51:09] <Jeremy_Rand> https://github.com/Yawning/tor-browser- ... sports.yml
[21:51:14] <brand0> nice
[21:51:55] <brand0> two+ years ago when I was writing go, it wasn't ready at all ecosystem-wise, but if that's changed, that could be a good option
[21:52:35] <Jeremy_Rand> haha, Tor's Gitian descriptor has "python.msi" as an input. I assume that's the official Python binary, i.e. not Gitian built?
[21:52:38] <Jeremy_Rand> jbisch: ^
[21:53:13] <jbisch> I'm not sure. Let me check.
[21:54:09] <Jeremy_Rand> if Mike Perry couldn't even find a way to make Python work properly in Gitian, that's a sign that we should be staying as far away from Python as possible. Mike knows his shit.
[21:55:27] <jbisch> Jeremy_Rand: https://github.com/Yawning/tor-browser- ... sions#L117
[21:56:01] <jbisch> So yes, it is downloaded from the official Python website.
[21:56:19] <Jeremy_Rand> Jeez. So the Python devs can backdoor TorBrowser whenever they want.
[21:58:26] <Jeremy_Rand> ok yeah, so I am very tempted to take the position that Mike Perry has both more experience and more funding than we do, and if he hasn't gotten it working in the past couple years, we probably shouldn't expect it to work well for us
[21:58:52] <Jeremy_Rand> (no offense to jbisch )
[21:59:44] <jbisch> Yeah, this doesn't look good for Armory upstream if TBB can't even build Python and include it in a reproducible build.
[22:01:23] <Jeremy_Rand> jbisch: yeah. Sorry for the bad news, but at least now you know.
[22:03:34] <brand0> I really think Python should be ruled out
[22:06:44] <Jeremy_Rand> brand0: unless we magically find that Mike found a way to make Python work reproducibly on Windows, I think I agree that Python is ruled out, as is anything else that is a nightmare to do in Gitian
[22:07:11] <Jeremy_Rand> So of the remaining languages, sounds like the one we hate the least is Go?
[22:07:54] <Jeremy_Rand> thoughts on this jbisch?
[22:08:27] <brand0> +1 go
[22:09:13] <Jeremy_Rand> it looks like Tor is using Python2. Is it likely that Python3 will work much better? If so, why isn't Tor using Python3?
[22:09:23] <Jeremy_Rand> I know Python3 worked better for jbisch
[22:09:50] <Jeremy_Rand> I guess Tor is using Twisted...
[22:09:59] <Jeremy_Rand> which isn't fully ported to Python3
[22:13:45] <Jeremy_Rand> well, maybe jbisch will have more luck with Python3 than Mike did with Python2. But if that's the case, I would have expected Tor devs to use Python3 and rewrite any code that needed Python2.
[22:15:43] <jbisch> I don't recall having any issues with Python 3 and Twisted, so I'm not sure what issue Tor is having.
[22:17:21] <Jeremy_Rand> yeah, I don't use Twisted so I have no idea. I just recall that Twisted was only partially working with Python3 as of a year or two ago
[22:17:27] <Jeremy_Rand> maybe they've fixed it since then
[22:18:38] <jbisch> Yeah, I think a lot has changed since then.
[22:19:08] <jbisch> As far as Go is concerned, it seems like it will be easy to use with Gitian, so I say go for it.
[22:19:42] <Jeremy_Rand> sugarpuff: opinions?
[22:31:01] <Jeremy_Rand> hl: would be good to get your input on this too
[00:49:52] <sugarpuff> Jeremy_Rand: I am not experienced with Gitian. If you're asking for an opinion on what language to use for a third or fourth implementation of a Namecoin proxy, then Go seems alright, but I don't have a lot of experience with it. People seem to be able to GTD with it though, so that's a good sign. Rust might be too low-level, but it gets bragging rights.
[00:51:05] <sugarpuff> Jeremy_Rand: personally though, i don't think a third or fourth implementation makes a whole lot of sense given NMControl/DNSChain... DNSChain has a lot of features already, lots of contributors, lots of community potential, and if you just want a name change then that could be done without much difficulty.
[00:51:42] <sugarpuff> more important than another blockchain proxy are thin clients
[00:51:54] <sugarpuff> or, if you want to rewrite something, rewrite Namecoin
[00:52:00] <sugarpuff> C++ is a horrible language
[01:12:52] <Jeremy_Rand> sugarpuff: hl already has an implementation in Go (ncdns)
[01:14:11] <Jeremy_Rand> sugarpuff: I agree that thin clients are useful -- but generally a different set of people work on those (Daniel is mostly working on UNO commitments, while Phelix and I are mostly working on applications e.g. NMControl)
[01:14:20] <Jeremy_Rand> hl does work on both to some extent
[01:14:51] <Jeremy_Rand> sugarpuff: hl has a Namecoin port of btcd, I think. btcd is in Go.
[01:16:52] <Jeremy_Rand> sugarpuff: ncdns is pretty good in its current state; I haven't done much with it myself, but from what I gather it integrates well with Unbound and it supports most of the d/ spec. I don't think it has a REST API yet though, although I could be wrong
[01:20:06] <Jeremy_Rand> sugarpuff: do I recall correctly that BitShares was originally going to fund a Go version of DNSChain before they ran out of money?
[01:25:23] <Jeremy_Rand> Other than my strong dislike of Javascript/Coffeescript, I'm unaware of anyone who's successfully using Coffeescript with Gitian. I'm pretty sure neither Tor nor Bitcoin is doing it, and they're the two big users of Gitian that I'm aware of.
[01:26:05] <Jeremy_Rand> And if Python is disqualified because of poor Gitian support, then it's only fair to do the same to other languages that don't handle it well
[01:27:44] <Jeremy_Rand> Googling "Coffeescript Gitian" yields no obviously relevant results
[01:28:31] <Jeremy_Rand> sugarpuff: if you're interested in learning some stuff about Gitian and why it's useful, check out Mike Perry's talk at the CCCongress last December
[01:29:19] <Jeremy_Rand> (it includes a hilarious demonstration of a rootkit)
[01:38:00] <jbisch> Well, JavaScript is interpreted, so I guess you would be bundling the interpreter?? And so it comes down to whether the interpreter can be made reproducible.
[02:12:22] <sugarpuff> Jeremy_Rand: yeah I saw that CCC talk, and indeed as jbisch points out with interpreted languages it doesn't really make sense to use Gitian for the source
[02:12:56] <sugarpuff> for interpreted languages it makes more sense to do a hash of all of the source files and use Gitian on the interpreter
[02:13:31] <sugarpuff> (there is no "build" to reproduce in interpreted languages)
[02:13:45] <sugarpuff> (except the interpreter)
[02:24:59] <Jeremy_Rand> sugarpuff: true. I don't know of anyone who's gotten a Coffeescript compiler or Javascript interpreter to build in Gitian. I'm not sure how hard it would be, but there don't appear to be any experts in that area at the moment.
[02:25:44] <Jeremy_Rand> (Other than that, there's the fact that I've suffered through writing FreeSpeechMe in Javascript, and the sooner I can stop doing stuff in Javascript, the sooner my mental health will return to semi-sane)
[02:29:00] <Jeremy_Rand> sugarpuff: what Javascript interpreter does DNSChain use? Node.js? There are no obvious hits on Google for node.js Gitian
[02:29:44] <Jeremy_Rand> (I don't know why I'm even checking this, other than intellectual curiosity, given my strong dislike of Javascript's safety properties. But I'm a curious person.)
[02:41:46] <sugarpuff> Jeremy_Rand: DNSChain runs on any of the nodejs platforms out there. Basically Google's v8, plus node stuff. Almost nothing uses Gitian unfortunately, so i'm not surprised you haven't found hits.
[02:47:07] <brand0> gitan only makes sense when you need to compile something
[02:47:41] <brand0> I haven't seen anyone use it for node.js/browser, mainly just signed js bundles
[03:55:41] <hl> hi.
[04:08:52] <hl> Personally I think Go is ideal for this application. Go is... something like a DSL for writing network daemons. I have a problem with languages like JS because they make you use the continuation-passing style to do async I/O, which I find distasteful given that the technology to make languages that take care of that for you has been around for decades
[04:09:57] <hl> Rust, as far as I can tell, is not using a reactor-backed fibre system, unlike Go. You no doubt could use such a system with Rust, but the issue I have is that you're inventing your own version of Rust to code in essentially, which reduces opportunities for code reuse.
[04:10:11] <hl> See this essay I recently wrote: https://www.devever.net/~hl/nexuses
[04:10:30] <hl> Basically, because Rust does not dictate one I/O model, it becomes a cross-cutting consideration which fractures the language.
[04:11:04] <hl> That's understandable because it's a systems language and so shouldn't lock you into any particular model, but it makes Go superior, in my view, for this use case.
[04:12:04] <hl> Python, also, has the same problem. There are too many ways to do I/O in Python - blocking IO, Twisted, Stackless Python, the various Stackless-like systems which work with CPython, etc.
[04:12:51] <hl> This makes the statement that a given network library is written in Python a bit meaningless. If there's a DNS library that looks good, but it uses some completely different reactor system... etc.
[04:18:22] <hl> Jeremy_Rand: I also think your aversion to return codes is flawed; you're assuming exception-based code is always written well to handle all exceptions which could occur, which is just not true
[04:18:56] <hl> Jeremy_Rand: For fatal exceptions, if you aren't aware, there is panic(), which causes Goroutine death if it is caught, or process death otherwise
[04:19:11] <hl> Jeremy_Rand: i.e., it's quite possible to throw exceptions which people can't ignore
[05:31:58] <Jeremy_Rand> hl: I'm definitely aware that exception-based code is tricky in Python, mainly because you don't always know which exceptions can occur. I think I said that, didn't I?
[05:32:50] <hl> Jeremy_Rand: Hi.
[05:33:04] <hl> Jeremy_Rand: So...
[05:34:13] <Jeremy_Rand> hl: if there were a clear leader in the areas of Rust libraries that we would want, then I think that wouldn't be a problem. As it stands, the Rust ecosystem is very immature and it's not at all clear what the ecosystem will look like 6 months from now, let alone 3 years. So that means that any nexuses that exist now, may not be popular then, and conversely any nexuses that may exist in 3 years, probably don't exist now
[05:34:35] <hl> Yea
[05:34:53] <hl> I think Rust used to have a libuv (the Node.js reactor) based IO system in core, but removed it
[05:34:59] <hl> So there's clearly a lot up in the air.
[05:35:16] <Jeremy_Rand> I hope that some nexuses emerge as popular in Rust's ecosystem within the next few years. But it's definitely not the case right now
[05:35:30] <Jeremy_Rand> (btw, that's an excellent article.)
[05:35:50] <hl> The only practical alternative to Go which has as good support for IO is probably Erlang, but that's not something within any of our expertise, I don't care to develop network programs in a functional style, and the runtime is way too heavyweight for embedded use
[05:35:55] <hl> Jeremy_Rand: thanks.
[05:36:25] <hl> Jeremy_Rand: I did recently find vibed, a D library which does Go-style fibres-on-event-system, but, obviously, that isn't anywhere near as supported as Golang
[05:36:44] <hl> Jeremy_Rand: (I also somewhat resent D for the fact that it failed to solve one of the most important issues with C++, the pimpl issue)
[05:37:02] <Jeremy_Rand> pimpl issue?
[05:37:29] <hl> Jeremy_Rand: C++ has things like public, protected, private, right? It's all about implementation hiding. Except... when you write a C++ class, it goes something like this:
[05:37:33] <hl> struct Foo {
[05:37:35] <hl> int DoStuff();
[05:37:37] <hl> private:
[05:37:39] <hl> int _x;
[05:37:41] <hl> };
[05:37:51] <hl> So the _x is hidden from consumers semantically, but not lexically.
[05:38:09] <hl> If you instantiate Foo as 'Foo x', you're allocating it on the stack and any changes to the contents of Foo breaks the ABI.
[05:38:26] <hl> And if you change it, you have to rebuild everything dependent on it, of course.
[05:38:29] <hl> Which in C++ can take ages.
[05:38:37] <hl> So C++ is very bad at actually concealing implementation details.
[05:38:50] <hl> If you want to actually conceal implementation details in C++, you have a few options:
[05:39:18] <hl> - Put everything in an interface, don't expose the actual struct which implements it in a .h file, instead put it in a .cpp file and expose a function like IFoo *CreateFoo();
[05:40:06] <hl> - Implement a pimpl class, which is basically a class which contains only a pointer to the 'real' data. And you implement proxy methods for all the methods for the 'real' class. But the real class is just forward declared as 'struct FooImpl;' and only implemented privately in the .cpp file.
[05:40:13] <hl> (This is not a sick joke. People actually do this.)
[05:40:39] <hl> For more information on how badly designed C++ is (and I say this as someone who has to use it) see yosefk's C++ "FQA".
[05:43:22] <Jeremy_Rand> oh wow. I searched for the FQA on Disconnect Search, and found a major bug in Disconnect
[05:43:33] <hl> Jeremy_Rand: ?
[05:43:40] <hl> http://yosefk.com/c++fqa/
[05:43:51] <Jeremy_Rand> Disconnect replaces any + characters in the search result URL links with spaces. So the link doesn't work.
[05:44:00] <Jeremy_Rand> Someone should probably report that to the Disconnect guys
[05:44:06] <Jeremy_Rand> but I figured it out
[05:44:10] <hl> Yikes.
[05:44:32] <hl> Yeah, the world wide web has always been annoyingly vague about whether + == ' ' or not. I really don't know.
[05:44:41] <hl> It seems to be a holdover from something.
[05:45:06] <hl> I think all new code encodes ' ' as %20, but I'm sure some systems still accept + as space
[05:46:35] <hl> Oh yeah, so C++ is basically impossible to parse. You know the 'typename' keyword in C++? That keyword was introduced to help the compiler parse what would otherwise be ambiguous.
[05:47:21] <hl> Even more obscurely, there are some cases where you have to gratuitously stick the keyword 'template' before identifiers. I don't even bother to remember when, because it's so rare. I just do what the compiler tells me to.
[05:47:56] <hl> Here's a lovely example:
[05:48:15] <hl> typedef typename std::remove_reference<typename function_traits<TransF>::template arg<0>::type>::type T;
[05:48:34] <hl> If you remove the 'help the compiler parse it' keywords, it's
[05:48:48] <hl> typedef std::remove_reference<function_traits<TransF>::arg<0>::type>::type T;
[06:00:21] <hl> Come to think of it, it's quite ironic that C++ has a very complicated type system, but not the safety that that usually implies with things like ML-derived languages.
[08:39:05] <hl> OK.
[08:39:22] <hl> Jeremy_Rand: ^
[08:39:40] <Jeremy_Rand> hey
[08:41:13] <Jeremy_Rand> um, so I don't want to speak for Phelix or any other devs... but I think the fact that even Tor couldn't get Python to build reproducibly, while they got Go to work just fine with no issues, is a strong indicator that we should be using Go rather than Python. And I say this despite not having much experience with Go, knowing that I'm going to have to brush up on it.
[08:41:49] <hl> Alright. So whose consensus do we need to still obtain? Domob, phelix, indolering, midnightmagic, khal maybe?
[08:41:56] <hl> And ryan-c.
[08:43:09] <Jeremy_Rand> also, while I don't support making technical decisions on diplomatic grounds, it occurs to me that discarding both the Python and Coffeescript codebases in favor of a third codebase might be an interesting way to resolve the longstanding rivalry betweeen certian advocates of those two codebases
[08:43:32] <hl> That... seems horribly silly, but also like human nature.
[08:43:47] <hl> Humans have a weird tendency to choose Option 3 when they can't choose between Options 1 and 2.
[08:44:05] <Jeremy_Rand> hl: yes. Does seem silly. Hence why I said I don't want to make a technical decision based on that.
[08:44:46] <hl> I don't know whether by 'Go' you mean 'ncdns' or 'something new', but FWIW, ncdns is the only basically complete implementation of domain-names.md that I'm aware of.
[08:45:20] <Jeremy_Rand> hl: I mean ncdns, unless there's another Go-based codebase competing with ncdns that I don't know of
[08:45:30] <hl> There isn't that I'm aware of.
[08:46:40] <Jeremy_Rand> hl: anyway... so I think brand0, jbisch, you, and I are (to varying extents) are in favor of Go. I'd say that for anything this major, we would want to get input from literally every current Namecoin dev, both major and minor
[08:46:50] <hl> Of course.
[08:48:35] <hl> Jeremy_Rand: oh, looks like you already made a thread. https://forum.namecoin.info/viewtopic.php?f=5&t=2406

[biolizard89]

I asked on the Tor-Dev mailing list yesterday about why they're not building Python in Gitian. Here is the thread.

https://lists.torproject.org/pipermail/ ... 09422.html

Basically, they tried 2 years ago, and it was very hard, and they gave up. They haven't tried to do it since then. So, there is a chance that it has gotten easier since then (I think Joseph is going to talk to the Tor guys about it soon). Still, Go has a huge advantage here.

[biolizard89]

Okay, so brand0, Joseph, Hugo, sugarpuff, and I seem to be in favor of Go. Still waiting on feedback from Daniel, Phelix, Indolering, Ryan, Cassini, and anyone else who wants to weigh in.

[somename]

I asked on the Tor-Dev mailing list yesterday about why they're not building Python in Gitian. Here is the thread.

https://lists.torproject.org/pipermail/ ... 09422.html

Basically, they tried 2 years ago, and it was very hard, and they gave up. They haven't tried to do it since then. So, there is a chance that it has gotten easier since then (I think Joseph is going to talk to the Tor guys about it soon). Still, Go has a huge advantage here.

Following those links I checked the current status of 2 quoted bugs - they were related to Python 3.2 and were since both fixed:
https://bugs.winehq.org/show_bug.cgi?id=29764
https://bugs.winehq.org/show_bug.cgi?id=30515

(Personally I have no preference, not being a developer, but Python 3 seems easier to deal with from a user perspective).