Typemock Isolator++ review - To Kill a Mocking Code

Updated: July 2, 2018

Testing software is like educating children. The earlier you do a good job the cheaper it is in the long run. When you code new solutions, ideally, you will have made a perfect job in one go. Alas, for things more complicated than Hello World!, the issue of bugs will inevitably come up. And with object-oriented programming, the complexity of real-case scenario simulations is exponentially higher.

The ultimate goal is to find bugs - as well as logical problems - before you hit the production environment. To wit, a variety of methodologies exist, all designed to help simplify multi-varied testing with minimal interaction. Mock testing is one of the popular and acceptable ways of doing this. Indeed, several days ago, I was asked by the Typemock team to take their Isolator++ tool for a spin. This unit testing solution supports both Windows and Linux, and works with C and C++ code. I will be testing the Linux version here. After me.

Teaser

Setup

I installed Isolator++ in CentOS 7.4 (albeit with kernel 4.16). The installation was fairly straightforward. Isolator++ comes as a single, compact rpm package, and once it's installed and you provide your serial key, or add it into the .ini configuration file, you delve through the documentation. Here I noticed a few interesting chicken & egg issues.

As the product page states, you need a unit test project first. Then, on its own, Isolator++ does not supply a test framework, and certainly no GUI. The program logic is contained in a clever set of shared libraries, which you use to compile your code with. We'll discuss this in detail shortly.

Looking online, Typemock Isolator++ has detailed documentation, but it's mostly Windows focused, and it shows how to integrate the program into Visual Studio (focusing on the .NET edition of the tool). I've not seen any mention of similar integration into open-source IDE.

Environment

Isolator++ takes several steps to configure and run properly. First, you need some code - C or C++. There's another snag here, we'll discuss it in a jiffy. Then, you need to include the Isolator.h file in your sources so that you have the Isolator namespace and its associated macros available. Finally, you need to specify a few environment variables, similar to what I've shown you in my fourth hacking article.

To begin with, I decided to use the Null Pointer Exception and Heap Overflow examples from my GDB tutorial. While this is not really dynamic code (although for malloc, one could argue that this is something worth testing), I thought this would a good starting point to explore the program's capabilities. My expectation was that Isolator++ would be able to analyze the code and then recommend intelligent tweaks. I also based my test journey on previous work experience. I am familiar with tools of somewhat similar nature, like Valgrind and VTune Amplifier, plus I've done a fair share of software debugging in my life.

First test: C language

I added the define and include directives into the code and then tried to compile the code. I hit a bunch of errors right away. The Web doesn't haven anything specific for Typemock Isolator++, but similar threads quickly allowed me to narrow down the issue.

/usr/lib64/typemock/libisolator.so: undefined reference to `sem_unlink'
/usr/lib64/typemock/libisolator.so: undefined reference to `pthread_mutexattr_settype'
/usr/lib64/typemock/libisolator.so: undefined reference to `sem_close'

First, I needed to expand the compilation flags set from the official subset written in the documentation to something more elaborate, e.g.

gcc -g3 simple.c -O0 -pthread -ldl -fPIC -I/usr/include/typemock -L/usr/lib64/typemock -l:libisolator.so -o simple.bin

But then, I hit issues with the actual inclusion of the Isolator.h program:

simple.c:4:22: fatal error: Isolator.h: No such file or directory
#include <Isolator.h>

At this point, I decided to contact the Typemock support. The guys were quite helpful, and we walked through a whole bunch of errors. Now, as it turns out - Isolator.h is available only for C++ code and not for C. This means you can't meaningfully use the macros with any C code, which is a shame. You need to wrap it, but that's double work.

I decided to drop the inclusion of Isolator.h into my sources (the NPE and HP examples), just to get moving forward, fully aware that I'd have limited Isolator++ functionality at this point. Once both programs compiled well, I had them running.

file simple.bin
simple.bin: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=5999f075b15c3a0072794cd194e1c59112162e1e, not stripped

ldd simple.bin
linux-vdso.so.1 => (0x00007ffed89f3000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f38a9102000)
libisolator.so => /usr/lib64/typemock/libisolator.so (0x00007f38a8d61000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f38a8b45000)
libc.so.6 => /lib64/libc.so.6 (0x00007f38a8782000)
/lib64/ld-linux-x86-64.so.2 (0x00007f38a9306000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f38a847a000)
libm.so.6 => /lib64/libm.so.6 (0x00007f38a8178000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f38a7f62000)

The NPE program quit without any great verbosity:

./simple.bin
Logger: Loaded .ini file from '/usr/share/typemock/typemock.ini'
Isolator log info file: /var/log/typemock/Log-169.06.2018-14-18-52.log

*** Initializing Typemock Isolator++ 4.0.0.7 (64bit) ***
Documentation: https://www.typemock.com/docs?book=Ipp

Processing debug info...
Error occured

P.S. There's a typo in the output - occured should read occurred.

Next, the relevant log file didn't show much useful information:

14:18:52:108 [tid 7737] Leave CDwarfAccess::Init
14:18:52:108 [tid 7737] Leave CDwarfAgent::Init
backtrace:
/usr/lib64/typemock/libisolator.so(_Z23HandlerToPrintBacktracei+0x3d)
[0x7faa91490fbd]
/lib64/libc.so.6(+0x35270)[0x7faa90e44270]
./simple.bin[0x4006c8]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7faa90e30c05]
./simple.bin[0x4005e9]

For the slightly more complex BO example, I got:

...
Error occured
*** Error in `./advanced.bin': malloc(): memory corruption: 0x0000000000a84f90 ***

This is better than just wordless segmentation fault, as we've seen in the GDB article. Now, as I mentioned earlier, my magical expectation was that, with Isolator++ library linked in, the library would then suggest or auto-detect the bad code, and tweak the loop counter to avoid the overflow - or at the very least point it out without any need for additional troubleshooting. Typemock has more on how to mock C functions, but this is not trivial.

Second test: C++ language

I decided to try a similar example to the above, only using C++ now. Here it is:

#include <iostream>
#include <cstring>

using namespace std;

int main(int argc, char *argv[])
{
char buffer[10];
strcpy(buffer, argv[1]);
}

Imagine this is part of some bigger code. I added the necessary #define and #include, tried to compile, and this did not work, either!

usr/include/typemock/Isolator.h: In function ‘T& IS_REF(TLambda)’:
/usr/include/typemock/IsolatorInternals.h:1626:48: error: must #include <typeinfo> before using typeid
#define __INTERNAL_ANY_REF(T) ANY_REF2<T>::GET()

Again, the Typemock team were quite helpful. Still more compilation flags needed:

g++ -g3 cppnpe.cpp -pthread -ldl -fPIC -std=c++11 -MMD -I/usr/include/typemock -L/usr/lib64/typemock -l:libisolator.so -o cppnpe.bin

Once this first compilation went through just fine, I was able to add various macros into the source, including the very important CLEANUP (similar to this example). But again, it's not trivial, and you really need to invest time in changing your code so that it is mockable. If you've never done this before, you won't solve it just by linking in the shared library.

...
Processing debug info...
Error occured

Checking the logs, there was more useful info now - but still not detailed enough:

16:32:54:287 [tid 6260] Leave CDwarfAccess::Init
16:32:54:287 [tid 6260] failed init demangling: _ZNSsC1Ev@@GLIBCXX_3.4

16:32:54:287 [tid 6260] failed init demangling: _ZNSt8ios_base4InitC1Ev@@GLIBCXX_3.4

16:32:54:287 [tid 6260] failed init demangling: _ZNSt8ios_base4InitD1Ev@@GLIBCXX_3.4

16:32:54:287 [tid 6260] failed init demangling: _ZNSsD1Ev@@GLIBCXX_3.4

16:32:54:287 [tid 6260] Leave CDwarfAgent::Init
backtrace:
/usr/lib64/typemock/libisolator.so(_Z23HandlerToPrintBacktracei+0x3d)
[0x7f564cc62fbd]
/lib64/libc.so.6(+0x35270)[0x7f564bdf6270]
/lib64/libc.so.6(+0x157436)[0x7f564bf18436]
./cppnpe.bin[0x400926]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f564bde2c05]
./cppnpe.bin[0x400839]

Third test: Example test suite

Isolator++ comes with a complete set of examples, based on Google Test. Once you compile this set, you will see what and how Isolator++ behaves (you can tweak the visual output of course). And you can also use the existing examples as a baseline for your work. Looks very neat and powerful - I was finally able to see what Isolator++ can do. But you need a test framework, and you need to build your tests in a meaningful way.

Examples

Sample code

I noticed some rather interesting features in the online documentation - Suggest (next unit test generation) and SmartRunner, which runs only specific subsets of tests based on the previous iterations. Alas, both examples showcases usage through Visual Studio with Isolator for .NET only, and the Analyzing Bugs page is empty. I spoke to the support once again, and there seem to be plans to develop an IDE as well as allow for intelligent analysis of code on the fly, but at the moment there are still not available in the official Isolator++ build.

Conclusion

From my relatively short testing, my overall impression of Isolator++ is mixed. Primarily because it seems mostly geared toward Windows usage (including the documentation), and C language takes a back seat. I think there should be a separate set of libraries for C sources. Moreover, a standalone UI would be most welcome, one that would allow modular use of test frameworks. That would also allow Isolator++ to capture more errors with fewer code changes.

This kind of program will never be trivial, so some problems around compilation and setup are expected. Still, the setup and configuration steps ought to be more detailed. Lastly, the program comes with a very steep price of EUR399, which guarantees only serious, professional use, but you can request an evaluation license that is good for 60 days, more than enough to figure out your first test case.

Despite the technical limitations and niggles I encountered, I was able to make progress relatively quickly. Isolator++ is fairly intuitive to use, and it does resonate with how my brain is architected. But the old programming adage - there is not such thing as a free lunch - still holds true. Mock testing takes a lot of effort to get right, and this is something that Isolator++ does not solve on its own. Worth testing, but don't expect miracles, and I want to see a fully fledged UI and better support for the good ole C. Grade: 7/10. Take care.

Cheers.

You may also like: