|
In this document I attempt to explain the decisions made while
developing TUT.
One day I ran into need of unit test framework for C++.
So, I've made a small research and discovered C++Unit,
boost::test and a bunch of similar libraries.. Though they
were usable, I was not satisfied with the approach they offered;
therefor I designed my own variant of unit test framework based
on the following restrictions:
- No of C-style macros.
- No manual registration for test groups and methods.
- No libraries of any kind.
- Neutrality to user interface.
- No Javisms.
Usually C++ Unit test frameworks define a lot of
macroses to achieve the goals other languages have as built-in features:
for example, Java is able to show you the whole exception stack;
and C++ cannot. So, to achieve the same (or similar) results, C++ frameworks
often declare a macro to catch any exception and trace __FILE__ and __LINE__
variables.
The problem is that it turns the C++ code into something that
is hard to read, where "hard to read" actually means "hard to maintain".
Macros don't recognize namespace borders, so a simple
macro can expand in the user code into something unexpected.
To avoid this, we have to give macros unique prefixes, and this,
in turn, reduces code readability even more.
From bad to worse, C-style macros can't handle
modern C++ templates, so comma separated template arguments
will break the macro, since preprocessor will handle the template as
two arguments (separated by the comma used in the template) to this macro.
And the final contra for macros is that even if used they cannot
achieve the same usability level as the native language tools; for example,
macros cannot generate a full stack trace (at least, in a platform-independent
manner). So it looks like we loose readability and get almost nothing for this.
See also Bjarne Stroustrup notices about macros harmness:
So, what's wrong with using macros?
In JUnit (Java-based Unit Tests framework) reflection is
used to recognize user-written test methods. C++ has no
reflection or similar mechanism, so user must somehow
tell the framework that "this, this and that" methods should be
considered as test methods, and others are just helpers for them.
The same can be said about test groups, which have to be registered
in test runner object.
Again, most C++ frameworks introduce macros or at least
methods to register a freestanding function or method as
a test instance. I find writing redundant code rather annoying:
I have to write test itself and then I have to write more
code to mark that my test is a test. Even more, if I forget to register
the method, nothing will warn me or somehow indicate that I have not
done what I should.
Most of C++ frameworks require building a library that
user must link to test application to be able to run tests.
The problem is that ways various platforms imply different
ways for building libraries.
One popular C++ framework
has more than 60% bugs in bug database that sound like
"cannot build library on platform XXX" or "built library
doesn't work on platform YYY".
Besides, some platforms has complexities in library memory
management (e.g. Win32).
Finally, building library is an additional step in integrating
test framework with existing code; and each additional step makes
the task more complex.
Some frameworks provide predefined formatters for
output results, for example CSV or XML.
This restricts users in test results presentation options.
What if a user wants some completely different format?
Of course, he can implement his own formatter, but why
should frameworks provide useless formatters then?
The ideal test framework must do only one thing: run tests.
Anything beyond that is the responsibility of the user code.
Framework provides the test results, and
the user code then represents them in any desired form.
Most implementors of C++ test frameworks knew about JUnit
and were inspired by this exciting tool. But, carelessly
copying a Java implementation to C++, we can get strange and
ugly design.
Rather obvious example: JUnit has methods for setting up a test (setUp)
and for cleaning after it (tearDown). I know at least two C++
frameworks that have these methods with the same semantics and names.
But in C++ the job these methods do is the responsibility of constructor
and destructor! In Java we don't have guaranteed destruction, so JUnit
authors had to invent their own replacement for it - tearDown(); and it was
natural then to introduce constructing counterpart - setUp(). Doing
the same in C++ is absolutely redundant, at least to say.
C++ has its own way of working, and whenever possible,
I am going to stay at the C++ roads, and will not repeat Java
implementation just because it is really good for Java.
The solution is that simple: just do not
use any macros. I personally never needed a macro during development.
Since C++ has no reflection, the only way to
mark a method as a test is to give it a kind of
predefined name.
There is simple solution: create a set of virtual methods
in test object base class, and allow user to overwrite them.
The code might look like:
struct a_test_group : public test_group
{
virtual void test1()
{
...
}
virtual void test2()
{
...
}
};
Unfortunately, this approach has some drawbacks:
- It scales badly. Consider, we have created 100 virtual
test methods in a test group, but user needs 200. How can
he achieve that? There is no proper way. Frankly speaking, such a
demand will arise rarely (mostly in script-generated tests),
but even the possibility of it makes this kind of design
seemingly poor.
- It requires a lot of redundant manual work and looks ugly.
Just suppose you have an assignment to write a hundred of
empty virtual methods testN() in framework, or you simply
look at those hundred methods in the framework header... don't you
think the design is at least strange, if not weak?
- Additionally, there is no way to iterate virtual methods automatically.
We would end up writing code that
calls test1(), then test2(), and so on, each with its own
exception handling and reporting. Unattractive.
Another possible solution is to substitute reflection with a
dynamic loading. User then would write static functions with predefined names,
and TUT would use dlsym()/GetProcAddress() to find out the implemented
tests.
But I rejected the solution due to its platform and library
operations dependencies.
As I described above, the library operations are quite different
on various platform.
There was also an idea to have a small parser, that can scan the
user code and generate registration procedure. This
solution only looks simple; parsing free-standing user
code can be a tricky procedure, and might easily overgrow
TUT itself in complexity.
Fortunately, modern C++ compilers already have a tool that
can parse the user code and iterate methods. It is
compiler template processing engine. To be more precise,
it is template specialization technique.
The following code iterates all methods named test<N>
ranging from n to 0, and takes the address of each:
template <class Test,class Group,int n>
struct tests_registerer
{
static void reg(Group& group)
{
group.reg(n,&Test::template test<n>);
tests_registerer<Test,Group,n-1>::reg(group);
}
};
template<class Test,class Group>
struct tests_registerer<Test,Group,0>
{
static void reg(Group&){};
};
...
test_registerer<test,group,100>.reg(grp);
This code generates recursive template instantiations
until it reaches tests_registerer<Test,Group,0> which has
empty specialization. There the recursion stops.
The code is suitable for our needs because in the specialization
preference is given to the user-written code, not to the generic one.
Suppose we have a default method test<N> in our test group,
and the user-written
specialization for test<1>. So while iterating, compiler
will get the address of the default method for all N, except 1, since user
has supplied a special version of that method.
template<int N>
void test()
{
};
...
template<>
void test_group::test<1>()
{
// user code here
}
This approach can be regarded as kind of compile-time virtual functions, since
the user-written methods replace the default implementation. At the same time,
it scales well - one just has to specify another test number upper bound at compile time.
The method also allows iteration of methods, keeping code compact.
Since we dig into the template processing, it is natural to not
build any libraries, therefor this problem mostly disappeares.
Unfortunately, not completely: our code still needs some central point where it
could register itself. But that point (singleton) is so small that
it would be an overkill to create library just to put there one single object.
Instead, we assume that the user code will contain our singleton somewhere
in the main module of test application.
Our code will perform only minimamum set of
tasks: TUT shall run tests. But we still need a way
to adapt the end-user presentation requirements. For some of users
it would be enough to see only failed tests in listing;
others would like to see the complete plain-text report; some would
prefer to get XML reports, and some would not want to get
any reports at all since they draw their test execution log in GUI plugin
for an IDE.
"Many users" means "many demands", and satisfying all of them
is quite a hard task. Attempt to use a kind of silver bullet
(like XML) is not the right solution, since user would lack XML
parser in his environment, or just would not want to put it
into his project due to integration complexities.
The decision was made to allow users to form their reports
by themselfs. TUT will report an event, and the user code will
form some kind of an artefact based on this event.
The implementation of this decision is interface tut::callback.
The user code creates a callback object, and passes
it to the runner. When an appropriate event occures, the test runner invokes callback methods.
User code can do anything, from dumping test results
to std::cout to drawing 3D images, if desired.
Initially, there were plans to make TUT traits-based
in order not to restrict oneself with STL only, but have a possibility
to use other kinds of strings (TString, CString), containers
and intercepted exceptions.
In the current version, these plans are not implemented due to
relative complexity of the task. For example, the actual set
of operations can be quite different for various map
implementations and this makes writing generic code much harder.
Still, I don't give up these ideas and will make such an attempt,
if there is a demand for it, and if I have some time and motivation.
But so far TUT is completely STL-based, since STL is
the only library existing virtually on every platform.
|