Dot-Bit Test Suite?

[biolizard89]

I brought this up on the IRC, but I might as well bring it up here too. At the moment testing FreeSpeechMe has become a bit tedious since there are a lot of different .bit features which could independently break. I think it would be useful to develop a .bit test suite, basically an HTML page that iframes a bunch of test .bit domains, so that we can easily detect when an implementation is deviating from the spec. This would also help us develop an objective way to evaluate the comparative compliance of various implementations, e.g. nmcontrol, FreeSpeechMe, NamecoinToBind, Speech.is, etc.

As a side note, this would also quite possibly be capable of doing a couple of bits of fingerprinting the .bit implementation in use. Presumably this isn't a big deal unless you're using Tor (which is already known to not be properly supported).

Thoughts?

[domob]

That would be great, IMHO! Or at least set up a bunch of .bit names to have different characteristics (invalid certificate, resolve via .onion, and so on) and publish the list somewhere easy to find.

[biolizard89]

I wonder how hard it would be to make such a test suite use AJAX, so that the results could be automatically determined and displayed without iframes. I think CORS is the only thing necessary for that. This could be adapted to perform automated tests via continuous integration of the GitHub repo.

[indolering]

Hands down, the best way to do this is to either use Slenium/Webdriver or Casper.js (which can drive both WebKit and Gecko-based browsers). I like Casper as it is headless and I can write quick unit tests in Javascript.

However, you should probably go with Webdriver as it has bindings for Java, C#, PHP, JavaScript, Ruby, Python, and Perl and the sponsoring company provides unlimited testing for FOSS projects.

Real unit tests, no need to fuck around with iframes/CORS/sing-origin-policy/JS injection, and you can test it across every browser there is (I know they used to allow for addons...).

We could setup test.namecoin.info and put our acid tests there.

[biolizard89]

Hands down, the best way to do this is to either use Slenium/Webdriver or Casper.js (which can drive both WebKit and Gecko-based browsers). I like Casper as it is headless and I can write quick unit tests in Javascript.

However, you should probably go with Webdriver as it has bindings for Java, C#, PHP, JavaScript, Ruby, Python, and Perl and the sponsoring company provides unlimited testing for FOSS projects.

Real unit tests, no need to fuck around with iframes/CORS/sing-origin-policy/JS injection, and you can test it across every browser there is (I know they used to allow for addons...).

We could setup test.namecoin.info and put our acid tests there.

To my knowledge Selenium will only test browser-based solutions, i.e. it won't test stuff like nmcontrol and NamecoinToBind. Using AJAX+CORS should be able to work with anything that implements .bit, right? Will AJAX mess up stuff like Speech.is? Also, I like the idea of users being able to test their configuration (particularly when we put more emphasis on "private mode" resolution, TLS enforcement, and multiple resolvers, all of which have user settings, so users will be able to make sure their configuration is what they intend).

[indolering]

To my knowledge Selenium will only test browser-based solutions, i.e. it won't test stuff like nmcontrol and NamecoinToBind.

I don't understand, a webpage with iFrames entails using a browser to perform testing.

If it's not happening in the browser then it should be tested using whatever protocol is native to that platform.

Using AJAX+CORS should be able to work with anything that implements .bit, right?

Yes, but that doesn't make it any less of a PITA.

Will AJAX mess up stuff like Speech.is?

As long as they are using relative links, it should be fine.

Also, I like the idea of users being able to test their configuration (particularly when we put more emphasis on "private mode" resolution, TLS enforcement, and multiple resolvers, all of which have user settings, so users will be able to make sure their configuration is what they intend).

This setup does not preclude end user tests, but I would suggest that the application test itself without human intervention.

However, I agree that a debugging page is very useful (I've got one at bits.name right now) but if you are going to build a test suite .... I was thinking more along the lines of unit tests.

At any rate, whatever you build will be useful.

[indolering]

Let me expand on Ajax and CORS:

It sucks to have to program them with them, it's hell to do so in a cross-platform manner (IE/Safari/FF/Chrome/Android/iPhone), and CORS means you have to keep a specific server setup as well, frames makes shit really confusing.

I would choose writing generic unit tests to writing Ajax calls any day of the week ツ

[biolizard89]

Yeah, so I think both Selenium-based tests and end-user tests would be good to have. I'll probably play around with end-user tests first, just because I'm more used to them.

[indolering]

Yeah, frames are a black box if loaded from an origin other than the parent browsing context. You cannot inspect the page directly, you have to do browser-level injection at which point you have to use either PhantomJS or Selenium anyway. So you might as well just write a Selenium test which visits the pages in sequence.