Posted by & filed under Observations.

A question from a friend that I got this morning and wrote back so much that we figured I should blog it.

seen this?

interested to hear your views!!

http://ivory.idyll.org/blog/automated-testing-and-research-software.html

I’m doing Test Driven Development for my own work so I have some experience with it. I like where this guy is going, it’s pragmatic; “do as much as you can”. I think what’s missing here is a proportional approach to testing. You test some stuff a lot more, and some stuff almost not at all. Generally speaking; the more mission critical the code is, the more I test it. Saying “if I can’t test 100% I shouldn’t test at all” is total horse shit.

For example, take WaveMetrics. They probably unit tested the absolute hell out of the ‘wave’ datatype. Every possible attribute and aspect of a wave had multiple tests so that even the slightest change would show an error and they would jump all over that. Whereas the About Box would have little or no testing.

Both extremes are bad. Too many tests and the code feels to ‘rigid’ and it’s hard to get anything done. Too few tests and it’s the wild west. The right blend is to have the outskirts of town a little ‘wild westy’ but when you go in town (into the core) you need to put on your ‘sunday go to meeting clothes’ even if they fee a little rigid.

Just safe and sane. Check in your code. Back it up. Have ‘releases’ even if it’s just to you. Test the critical code. Run the tests before checkins. If you can’t checkin the test data, at least check in the stuff you used to build the test data. I liken writing “exploratory software” to being on an island and seeing a little piece of dry land just in the distance. You cobble together whatever you can to get there. And then if you like it you backfill to permanently annex this new land to your old island.

If the island is the base of your software, and the little piece of land is that feature that you think you might want but your not sure, then if you find that you like it, you need to ‘formalize it’, by doing the work to make sure that it’s a genuine solid feature.

There is an actual term for this process of cobbling stuff together, it’s called ‘spiking’. And it’s where you take one guy off the team and have him cobble together a feature on top of the current software using whatever tools he can quickly. The idea is to determine two things; is it feasible? is it worthwhile? If it’s both you throw it away and put the real team on it to do it properly.

Way, way, way too much of what I did in our lab was that kind of spiker code. Not to mention it being out of source control, and yadda yadda yadda. (Though as an aside I’m an amazing spiker and it’s because of the work I did in the lab.)

That being said, I was onto something with ‘Acquisition Objects’ as I recall. Where I could build out an acquisition protocol in an OO abstraction then have it rendered out in ‘real time’. That type of structure; Acquisition Objects, is exactly the kind of thing it’s worth putting a lot of tests around, and formalizing. And that would take more time. But it’s worth doing because it gives a lot of confidence in the underlying correctness of the acquisition. It would also have been good because it would give a language between the Product Manager, Customer and the development team; “Let’s build a new type of acquisition object that takes the input in real time, phase shifts it, then sends it back out the output…” Or something like that. Everyone doesn’t need to know the details, but they do need to understand the abstractions.

A related anecdote for you. An acquaintance of mine is working on a data recovery project for a lab that had something in the range of 200TB. They RAIDed it, but for speed, not redundancy, by mistake, and they had some drives blow out. Basic stuff, done wrong, too risky. Result was an expensive mistake.

There is a lot that science software could take away from the was consumer software is written. A lot of stuff is just ‘known’ and ‘standard’ now. Wanna interoperate? It’s REST and JSON. Wanna store transactional or highly structured data, SQL, unstructured mutable data, NoSQL. Etc. Having lots of stuff ‘known’ and ‘standard’ is good because it means that people can come up to speed quickly, that customers and users have expectations that start at a higher level, and particularly that you can concentrate the bulk of your efforts on building your “unique value”. Ideally a project should be spending 90-100% of it’s time writing code (and tests) that are the unique value proposition of your project. The rest is a waste.

Last anecdote, an old boss of mine paid a consultant for three months to build a different scrollbar implementation for “Random Mac Science Product” on the Mac because he didn’t like the look and feel of the standard Mac scrollbar. That makes me want to punch either or both of them. Oh, and we didn’t end up using it because it was too buggy.

Leave a Reply

  • (will not be published)