COMPILE.md (10423B)
1 Compilation 2 =========== 3 4 First: the compilation script is only really tested under Linux, even though it 5 should compile on any POSIX system. The official windows versions are 6 cross-compiled on Linux. I avoid using that piece of shit as much as possible. 7 8 Requirements: 9 10 * Compiler: gcc-8 (?) or clang-8 11 * Python 3+ (for the compile script) 12 * boost-1.84 (might work with older, also see notes below) 13 * Optional: lua-5.3 14 15 A note on boost: if you're using a system/your own compiled boost, make sure 16 it's compiled with the c++17 ABI. It might work in c++11/14 mode as boost 17 doesn't really use noexcept, but it's better to be safe than sorry. If you don't 18 want to deal with this problem, just use the `get_boost.sh` to download a recent 19 boost tarball and let the build system handle it. 20 21 Compiling 22 --------- 23 24 Make sure submodules are checked out: 25 26 git submodule update --init --recursive 27 28 If you're not going to use your own boost, make sure to get it too: 29 30 libshit/get_boost.sh 31 32 Alternatively just make sure a recent boost tarball is extracted/symlinked to 33 `libshit/ext/boost/boost`. 34 35 In an ideal world, you can just run: 36 37 ./waf configure 38 ./waf 39 40 You can specify compile flags on configure: 41 42 CXX=g++ CXXFLAGS="-O3 -DNDEBUG" LINKFLAGS="-whatever" ./waf configure 43 44 You can use `CXXFLAGS_<appname>`, `LINKFLAGS_<appname>`, etc. to add flags only 45 used during compiling <appname> (e.g. `NEPTOOLS`), and `CXXFLAGS_EXT` etc. to 46 add flags only used during compiling bundled libs. 47 48 Some useful flags for configure: 49 50 * `--optimize`: produce some default optimization, but keep assertions enabled. 51 * `--optimize-ext`: optimize ext libs even if Neptools itself is not optimized 52 (will also remove debug info). 53 * `--release`: `--optimize` + no asserts 54 * `--system-boost`: use external Boost, see next section for warnings 55 * `--with-lua`, `--luac-mode`, `--lua-dll`: see section about lua 56 57 Vita 58 ---- 59 60 There is some experimental vita support, mostly in neptools. You need to get 61 [v1033][vitasdk] of Vita SDK (it might work with slightly newer version, but 62 don't try the latest, unless you want to fix the bugs and submit PRs). 63 64 mkdir -p /usr/local/vitasdk 65 cd /usr/local/vitasdk/ 66 curl -L https://github.com/vitasdk/autobuilds/releases/download/master-linux-v1033/vitasdk-x86_64-linux-gnu-2019-04-28_16-30-10.tar.bz2 | tar xj --strip-components=1 67 68 You'll also need my patched vita-elf-create. Clone 69 https://git.neptards.moe/neptards/vita-toolchain or 70 https://gitgud.io/neptards/vita-toolchain, checkout `variable-import` branch, 71 compile, then overwrite `/usr/local/vitasdk/bin/vita-elf-create` with the 72 freshly compiled `vita-elf-create`. 73 74 Also, take `vita_db.patch`, update `/usr/local/vitasdk/share/db.yml` and 75 regenerate (at least) `libSceLibc_stub.a` and `libSceLibc_stub_weak.a` with 76 `vita-libs-gen`. If you don't do this, neptools will compile but it will crash 77 your game (and likely your system too) on startup. 78 79 You also need taihen: 80 81 git clone https://github.com/vitasdk/vdpm 82 cd vdpm 83 VITASDK=/usr/local/vitasdk ./vdpm taihen 84 85 Note the build scripts doesn't automatically create .suprx yet. With neptools, 86 this means you need to run 87 88 vita-elf-create -e src/vita_plugin/plugin.yml build/vita_plugin build/vita_plugin.velf 89 vita-make-fself -c build/vita_plugin.velf build/vita_plugin.suprx 90 91 after `./waf`. 92 93 (Cross) Compiling to Windows 94 ---------------------------- 95 96 Currently only clang (probably patched, see next section) is supported, with 97 MSVC 2013 lib files. In case of Neptools, it's pretty much a requirement. 98 You'll also need [lld] if you want LTO or want to cross compile. I've only 99 tested cross compiling, but it should be possible to compile on Windows too. 100 101 Install MSVC 2013 on a Windows (virtual) machine. If you want to cross compile, 102 you'll need to copy directories named `include` and `lib` to your Linux box from 103 `Program Files (x86)/Microsoft Visual Studio 12.0/VC` and `Program Files 104 (x86)/Windows Kits/8.1` too (assuming default install location). 105 106 Pro tip #0: Open `include/stdarg.h`, and replace it with an `#error`. If this 107 file gets included, that means you fucked up the include order (since clang has 108 its own `stdarg.h`). In this case the program will compile, but vararg functions 109 will crash. 110 111 Problem #1: Linux filesystems are usually case-sensitive, but MSVC headers 112 pretty much expect a case-insensitive file lookup. 113 Solution 1: store the files on a case-insensitive fs (fat, ntfs, etc, or just 114 mount your Windows fs). 115 Solution 2: use [ciopfs]. Make sure you mount `ciopfs` first, and copy into that 116 directory, otherwise you'll manually have to convert all files to downcase. 117 Solution 3: use something like [icasefile] to make clang-cl and lld-link 118 case-insensitive. 119 Solution 4: convert all files and directories to downcase and fix the headers. 120 Something like this will do: 121 122 find path/to/msvc/files -name '*[A-Z]*' -print0 | sort -rz | xargs -0n1 sh -c 'mv "$0" "$(echo -n "$0" | awk -F/ "{ORS=\"/\"; for (i=1;i<NF;i++) print \$i; ORS=\"\"; print tolower(\$NF)}")"' 123 find path/to/msvc/includes -type f -exec sh -c 'echo "$(awk "/^[[:space:]]*#[[:space:]]*include/ {print tolower(\$0); next} {print \$0}" "$0")" > "$0"' {} \; 124 125 You'll also have to fix boost in this case... open 126 `libs/filesystem/src/unique_path.cpp` inside boost (inside `ext/boost` if you 127 use the bundled one) and replace `Advapi32.lib` with `advapi32.lib` in 128 `# pragma comment(lib, "Advapi32.lib")`. I personally use solution 2 and 4, 129 the other two solutions may or may not work. 130 131 132 Problem #2: clang will default to compile for the host (we're using `clang` and 133 not `clang-cl`!) and it won't know where are your files, so you'll need some 134 compiler flags. For compiling you'll need: `-target i386-pc-windows-msvc18 135 -Xclang -internal-system -Xclang $vc/include -Xclang -internal-system -Xclang 136 $winkit/include/um -Xclang -internal-system -Xclang $winkit/include/shared` 137 where `$vc` and `$winkit` refers to the folders you previously copied. Using 138 `-internal-system` will make sure your include paths are correct (this is whan 139 `-imsvc` uses under the hood, but that only works with `clang-cl`, not `clang`). 140 For linking, you'll need `-fuse-ld=lld -L$vc/lib -L$winkit/lib/winv6.3/um/x86`. 141 142 In the end, you'll end up with something like this: 143 ``` 144 CC=$clangbin/clang CXX=$clangbin/clang AR=$clangbin/llvm-ar CFLAGS="-target i386-pc-windows-msvc18 -Xclang -internal-system -Xclang $vc/include -Xclang -internal-system -Xclang $winkit/include/um -Xclang -internal-system -Xclang $winkit/include/shared" CXXFLAGS="$CFLAGS" LINKFLAGS="-fuse-ld=lld -L$vc/lib -L$winkit/lib/winv6.3/um/x86" ./waf configure [--release] [--with-tests] 145 ``` 146 147 Some potential problems with the clang toolchain 148 ------------------------------------------------ 149 150 Using lld 8.0.0 to link, the generated executable will crash when the first 151 exception is thrown and it won't have icons. Use 152 `ebuilds/sys-devel/lld/files/lld-8.0-u3.patch` to fix this. 153 154 Second problem: llvm/clang doesn't support the `/EHsa` flag, only `/EHs`, but 155 that won't catch LuaJIT exceptions. The 156 `ebuilds/sys-devel/llvm/files/llvm-8.0-u3.patch` includes a quick hack that'll 157 at least make sure destructors are called when unwinding lua exceptions (and 158 exceptions are handled manually by `__try`/`__except`). 159 160 Lua 161 --- 162 163 Libshit includes the most horrible lua binding generator that you'll ever see. 164 It officially supports plain lua 5.3, but it might work with LuaJIT, LJX or 165 whatever other lua variant you have. 166 167 Use `--with-lua=` to select a lua version: 168 169 * `none`: build without lua. Equivalent to `--without-lua`. 170 * `lua`: build plain lua 5.3. 171 * `system`: use a system lua. Specify the name of the correct pkg-config package 172 name with `--lua-pc-name=`. It should point to lua 5.3 (or something 173 compatible). 174 175 To run libshit, you have to embed a few lua scripts into the executable. There 176 are multiple methods to do this, use `--luac-mode=` to select it: 177 178 * `copy`: copy the lua source without changes. Use if you have no better option. 179 * `system-luajit`: use a system binary with `luajit` like command line to 180 compile lua to byte code. Select the binary by setting the `LUAC` environment 181 variable. Make sure the tool produces byte code that the selected lua 182 understands! 183 * `luac`: (only with `lua`) use the built lua's luac to compile scripts. 184 * `system-luac`: like `system-luac`, but expects a `luac` like binary. 185 * `luac-wrapper`: (only with `lua`) use the built luac, but use a wrapper to run 186 it. See below. 187 188 **Note** when using plain lua: unfortunately plain lua's byte code is not 189 portable, you can't use a luac built for platform A and load the byte code on 190 platform B. Here are your options: 191 192 * if you can run the binaries compiled for the target platform (for example 193 cross-compiling on `amd64` linux to 32-bit linux), just use `luac` if 194 compiling lua, or `system-luac` and make sure you specify the correct binary 195 (32-bit `luac` in the example). 196 * if you can run the compiled binaries with a wrapper (like `wine`, or the qemu 197 user emulation), use `luac-wrapper` and specify the wrapper if building lua. 198 In case of a system lua, you can use `LUAC=wrapper /path/to/luac` with 199 `system-luac`. 200 * For more complicated scenarios: either write a script that takes arguments 201 similar to luac and solves the problem somehow, or just use `--luac-mode=copy` 202 (which should be the default if the build script detects cross-compilation). 203 204 Lua binding generator 205 --------------------- 206 207 This is only relevant if you want to develop code. The generated binding files 208 are checked into the repository, so you don't have to do anything with it if you 209 only want to compile. 210 211 Requirements: 212 213 * `ruby-3.1` + `ffi` gem 214 * patched clang: apply `ebuilds/sys-devel/clang/files/clang-8.0-u3.patch` to 215 clang 8.0.0. 216 217 Generate bindings: 218 219 ./gen_binding.rb 220 221 If you have the patched clang in a non-standard location: 222 223 PREFIX=llvm/install/prefix ./gen_binding.rb 224 225 In the source, mark method which you do not want to export with `LIBSHIT_NOLUA`. 226 There's also `LIBSHIT_LUAGEN` which allows fine tuning of binding generation. 227 Edit `gen_binding.rb` if you need to generate binding for a new file. 228 229 TODO: write much more documentation 230 231 [vitasdk]: https://github.com/vitasdk/autobuilds/releases?q=master-linux-v1033 232 [lld]: http://lld.llvm.org/ 233 [ciopfs]: http://www.brain-dump.org/projects/ciopfs/ 234 [icasefile]: http://wnd.katei.fi/icasefile/