If you can convert a program from one language to another (which is non-trivially different), in the time it takes to complete your masters, I'm pretty sure it wasn't a very interesting program. Further, the quality of the developers is going to play a large role in how effective any tool (and make no mistake, static typing is a tool) is. This is not intended as a disparaging remark to the authors, but in the 30 seconds I spent reviewing each of these code bases, I was totally unimpressed: none of them seemed to follow PEP8, and several of their test files weren't even unit tests, they were just a random scrip that appeared to exercise a tiny part of the codebase. I therefore conclude that the methodology used in operating this experiment was flawed and, consequently the conclusion cannot be taken as scientifically valid.
I have not looked at the programs, but programs don't have to be long to be interesting. There are many "interesting" programs under 100 lines of code - and they can be important if they form the kernel of a larger program.
The author doesn't really respond to the point, he simply says they were non-trivial in complexity, which may be true, but that doesn't mean they're non-trivial in their dynamacism (is that a word?). Moreover it in no way responds to the claim that these codebases aren't very good.
Does it matter if "these codebases aren't very good"? Have you ever actually seen a good code base? Most code I've seen has some poorly written corners; something where someone was in a hurry, or didn't know what they were doing, or someone inexperienced with the project started working on it, or the like.
The point is, these were real-world codebases with substantial unit tests, and they had type errors that weren't caught by these unit tests.
Honestly, in several groups I've worked with, I've had trouble getting people to add unit tests even to code in dynamic languages. A codebase which already has substantial unit tests is likely to be better than average, on that basis alone.
This is about saying, in the messy real world, do unit tests actually make type safety obsolete. And the answer is, no, they don't, even code with fairly good test coverage can be improved by adding typechecking. Now, there is the question of whether writing in the dynamic language allowed people to write code faster (it's generally a lot easier to translate correct code once it's written than to write it in the first place); or whether some of the more highly dynamic features of dynamic languages benefit writing or deploying code.
The Wikipedia disambiguation for Dynamism (last edited in March) includes:
"Dynamism (computing), When any process in computer is using Dynamic management methods for its processing/computing/memory management/parallelism handling for being able to give more user friendly work that are more easy to interact and modify."
If most code doesn't use "dynamacism" (we'll pretend it is a word), then it isn't an important value for most people. You are suggesting people should use dynamic languages to gain no benefit because some other software might in theory benefit from it.
It seems like you're two criticisms are as follows:
1. The sample size is too small.
2. The quality of the Python code isn't that great.
I (the author) would like to address both of these. First of all I completely agree that more research needs to be done. I mention this in the paper. I have provided a data point not a proof. It took me a couple months of several hours a day to the do the translation, I hope more people translate more programs to see if the results hold in the face of a larger data set. Second I agree the quality of the Python code isn't that great. I wanted to see whether unit testing obviated static typing in practice. In order to avoid selection bias I choose the projects at random. I picked the first four projects that were < 2000 lines of code and that had some sort of unit testing.
I believe that my methodology, nor my conclusions are flawed but that all should remember that a single experiment does not make a scientific proof. I hope that others will try to replicate this experiment on many more code bases. If they do it will be interesting to see the results.
