parameterized-tests.md (7081B)
1 ## Parameterized test cases 2 3 Test cases can be parameterized easily by type and indirectly by value. 4 5 ## Value-parameterized test cases 6 7 There will be proper support for this in the future. For now there are 2 ways of doing data-driven testing in doctest: 8 9 - extracting the asserts in a helper function and calling it with a user-constructed array of data: 10 11 ```c++ 12 void doChecks(int data) { 13 // do asserts with data 14 } 15 16 TEST_CASE("test name") { 17 std::vector<int> data {1, 2, 3, 4, 5, 6}; 18 19 for(auto& i : data) { 20 CAPTURE(i); // log the current input data 21 doChecks(i); 22 } 23 } 24 ``` 25 26 This has several drawbacks: 27 - in case of an exception (or a ```REQUIRE``` assert failing) the entire test case ends and the checks are not done for the rest of the input data 28 - the user has to manually log the data with calls to ```CAPTURE()``` ( or ```INFO()```) 29 - more boilerplate - doctest should supply primitives for generating data but currently doesnt - so the user has to write their own data generation 30 31 - using subcases to initialize data differently: 32 33 ```c++ 34 TEST_CASE("test name") { 35 int data; 36 SUBCASE("") { data = 1; } 37 SUBCASE("") { data = 2; } 38 39 CAPTURE(data); 40 41 // do asserts with data 42 } 43 ``` 44 45 This has the following drawbacks: 46 - doesn't scale well - it is very impractical to write such code for more than a few different inputs 47 - the user has to manually log the data with calls to ```CAPTURE()``` (or ```INFO()```) 48 49 -------------------------------- 50 51 There is however an easy way to encapsulate this into a macro (written with C++14 for simplicity): 52 53 ```c++ 54 #include <algorithm> 55 #include <string> 56 57 #define DOCTEST_VALUE_PARAMETERIZED_DATA(data, data_container) \ 58 static size_t _doctest_subcase_idx = 0; \ 59 std::for_each(data_container.begin(), data_container.end(), [&](const auto& in) { \ 60 DOCTEST_SUBCASE((std::string(#data_container "[") + \ 61 std::to_string(_doctest_subcase_idx++) + "]").c_str()) { data = in; } \ 62 }); \ 63 _doctest_subcase_idx = 0 64 ``` 65 66 and now this can be used as follows: 67 68 ```c++ 69 TEST_CASE("test name") { 70 int data; 71 std::list<int> data_container = {1, 2, 3, 4}; // must be iterable - std::vector<> would work as well 72 73 DOCTEST_VALUE_PARAMETERIZED_DATA(data, data_container); 74 75 printf("%d\n", data); 76 } 77 ``` 78 79 and will print the 4 numbers by re-entering the test case 3 times (after the first entry) - just like subcases work: 80 81 ``` 82 1 83 2 84 3 85 4 86 ``` 87 88 The big limitation of this approach is that the macro cannot be used with other subcases at the same code block {} indentation level (will act weird) - it can only be used within a subcase. 89 90 Stay tuned for proper value-parameterization in doctest! 91 92 ## Templated test cases - parameterized by type 93 94 Suppose you have multiple implementations of the same interface and want to make sure that all of them satisfy some common requirements. Or, you may have defined several types that are supposed to conform to the same "concept" and you want to verify it. In both cases, you want the same test logic repeated for different types. 95 96 While you can write one ```TEST_CASE``` for each type you want to test (and you may even factor the test logic into a function template that you invoke from the test case), it's tedious and doesn't scale: if you want ```M``` tests over ```N``` types, you'll end up writing ```M * N``` tests. 97 98 Templated tests allow you to repeat the same test logic over a list of types. You only need to write the test logic once. 99 100 There are 2 ways to do it: 101 102 - directly pass the list of types to the templated test case 103 104 ```c++ 105 TEST_CASE_TEMPLATE("signed integers stuff", T, char, short, int, long long int) { 106 T var = T(); 107 --var; 108 CHECK(var == -1); 109 } 110 ``` 111 112 - define the templated test case with a specific unique name (identifier) for later instantiation 113 114 ```c++ 115 TEST_CASE_TEMPLATE_DEFINE("signed integer stuff", T, test_id) { 116 T var = T(); 117 --var; 118 CHECK(var == -1); 119 } 120 121 TEST_CASE_TEMPLATE_INVOKE(test_id, char, short, int, long long int); 122 123 TEST_CASE_TEMPLATE_APPLY(test_id, std::tuple<float, double>); 124 ``` 125 If you are designing an interface or concept, you can define a suite of type-parameterized tests to verify properties that any valid implementation of the interface/concept should have. Then, the author of each implementation can just instantiate the test suite with their type to verify that it conforms to the requirements, without having to write similar tests repeatedly. 126 127 128 A test case named ```signed integers stuff``` instantiated for type ```int``` will yield the following test case name: 129 130 ``` 131 signed integers stuff<int> 132 ``` 133 134 By default all primitive types (fundamental - ```int```, ```bool```, ```float```...) have stringification provided by the library. For all other types the user will have to use the ```TYPE_TO_STRING(type)``` macro - like this: 135 136 ```c++ 137 TYPE_TO_STRING(std::vector<int>); 138 ``` 139 140 The ```TYPE_TO_STRING``` macro has an effect only in the current source file and thus needs to be used in some header if the same type will be used in separate source files for templated test cases. 141 142 Other testing frameworks use the header ```<typeinfo>``` in addition to demangling to get the string for types automatically but doctest cannot afford to include any header in it's forward declaration part (the public one) of the header - so the user has to teach the framework for each type. This is done to achieve [maximal compile time performance](benchmarks.md). 143 144 Some notes: 145 146 - types are NOT filtered for uniqueness - the same templated test case can be instantiated multiple times for the same type - preventing that is left up to the user 147 - you don't need to provide stringification for every type as that plays a role only in the test case name - the default is ```<>``` - the tests will still work and be distinct 148 - if you need parameterization on more than 1 type you can package multiple types in a single one like this: 149 150 ```c++ 151 template <typename first, typename second> 152 struct TypePair 153 { 154 typedef first A; 155 typedef second B; 156 }; 157 158 #define pairs \ 159 TypePair<int, char>, \ 160 TypePair<char, int> 161 162 TEST_CASE_TEMPLATE("multiple types", T, pairs) { 163 typedef typename T::A T1; 164 typedef typename T::B T2; 165 // use T1 and T2 types 166 } 167 ``` 168 169 ------ 170 171 - Check out the [**example**](../../examples/all_features/templated_test_cases.cpp) which shows how all of these are used. 172 173 --- 174 175 [Home](readme.md#reference) 176 177 <p align="center"><img src="../../scripts/data/logo/icon_2.svg"></p>