Summary:
If this variable is set to a ccache symlink then the NCCL build will
also use the cache. The NCCL build is the slowest component of a cached
build without this change
Closes https://github.com/caffe2/caffe2/pull/1416
Reviewed By: Yangqing
Differential Revision: D6214008
Pulled By: pietern
fbshipit-source-id: e0a90e27de9b1c5a1fdc0e5bad5fb61f9fa924c3
Summary:
Caffe2 fails to build with some old CMake versions because it doesn't figure out that the build implicitly depends on NNPACK build.
This commit adds this dependency explicitly.
Closes https://github.com/caffe2/caffe2/pull/1414
Differential Revision: D6203486
Pulled By: Maratyszcza
fbshipit-source-id: 86f6d9d88976656820f44e3416c57ddf22350362
Summary:
This will help releasing models that are using Caffe2 but have their own operator implementations and extensions. More detailed docs to arrive later. Let's see what contbuild says.
Closes https://github.com/caffe2/caffe2/pull/1378
Differential Revision: D6155045
Pulled By: Yangqing
fbshipit-source-id: 657a4c8de2f8e095bad5ed5db5b3e476b2a877e1
Summary:
NNPACK now supports building with CMake, and its build scripts have advantages over the ones in Caffe2:
- They automatically download all dependencies, no need to keep them in submodules anymore
- They automatically download and setup PeachPy for x86-64 build
- The same scripts are used for server/desktop (Linux, macOS) and mobile (Android/iOS)
- They unblock Caffe2 build with Ninja
Closes https://github.com/caffe2/caffe2/pull/1382
Reviewed By: Yangqing
Differential Revision: D6150723
Pulled By: Maratyszcza
fbshipit-source-id: 7c3e4e3406f60d4cc059e1c8112cb10aa3d75ece
Summary:
(1) use the cmake files of the corresponding libs
(2) allow static linkage of gtest and gbenchmark.
(3) Helps removing the temp solution in #1112
We are yet to disable the installation of the benchmark library, and I have an open pull request at https://github.com/google/benchmark/pull/463 - once it is merged I will do submodule update.
cc lukeyeager pietern who had this issue before - hopefully this makes the solution cleaner.
Closes https://github.com/caffe2/caffe2/pull/1358
Differential Revision: D6111404
Pulled By: Yangqing
fbshipit-source-id: 17468d32cef27f96e9445d119eb869c9c7913118
Summary:
This introduces a few things:
- It enables us to create Caffe2Config.cmake that can be used down the road for building dependent libraries, so they do not need to explicitly write FindCaffe2.cmake.
- The config file will automatically figure out transitive dependency of Caffe2 as well as compiler flags.
- This diff also disables the RPATH setting since it is kind of a mess right now. In principle, we should figure out a clearer rpath setting following the typical rpath setting choices (https://cmake.org/Wiki/CMake_RPATH_handling) - I can send a follow up PR to clean this up.
- Minor: removed old gflags ang glog files.
Closes https://github.com/caffe2/caffe2/pull/1354
Reviewed By: dzhulgakov
Differential Revision: D6098014
Pulled By: Yangqing
fbshipit-source-id: cb06c41a7ef60fddb78b24887b6b3e82684b7c6b
Summary: observer framework can now be used in python + a small writeup of how to use it. this is D6035393 with a fix for ct-scan
Reviewed By: salexspb
Differential Revision: D6066380
fbshipit-source-id: 896c4c580d4387240b81ac2dbbc43db51d4bfeb9
Summary:
The scripts/build_local.sh script would always build protoc from the
third_party protobuf tree and override the PROTOBUF_PROTOC_EXECUTABLE
CMake variable. This variable is used by the protobuf CMake files, so
it doesn't let us detect whether the protoc was specified by the user
or by the protobuf CMake files (e.g. an existing installation). This
in turn led to a problem where system installed headers would be
picked up while using protoc built from third_party. This only works
if the system installed version matches the version included in the
Caffe2 tree. Therefore, this commit changes the variable to specify a
custom protoc executable to CAFFE2_CUSTOM_PROTOC_EXECUTABLE, and
forces the use of the bundled libprotobuf when it is specified.
The result is that we now EITHER specify a custom protoc (as required
for cross-compilation where protoc must be compiled for the host and
libprotobuf for the target architecture) and use libprotobuf from the
Caffe2 tree, OR use system protobuf.
If system protobuf cannot be found, we fall back to building protoc
and libprotobuf in tree and packaging it as part of the Caffe2 build
artifacts.
Closes https://github.com/caffe2/caffe2/pull/1328
Differential Revision: D6032836
Pulled By: pietern
fbshipit-source-id: b75f8dd88412f02c947dc81ca43f7b2788da51e5
Summary:
Latest version of Gloo takes care of MPI_Init/MPI_Finalize for us, so
this commit removes handling that from caffe2/contrib/gloo. It also
imports CMake NCCL module changes from Gloo to stay consistent and
allow setting NCCL_INCLUDE_DIR and NCCL_LIB_DIR separately.
Closes https://github.com/caffe2/caffe2/pull/1295
Reviewed By: dzhulgakov
Differential Revision: D5979364
Pulled By: pietern
fbshipit-source-id: 794b00b0a445317c30a13cc8f0f4dc38e590cc77
Summary:
This would allow one to debug with asan. Known problems:
- only works with new -fsanitizer=address option.
- not tested on clang.
It's turned off in default so existing builds won't be affected.
Closes https://github.com/caffe2/caffe2/pull/1299
Differential Revision: D5987034
Pulled By: Yangqing
fbshipit-source-id: de29cd3b84edaed5db73e33f8f759c5c3271b5b7
Summary:
Useful for figuring out with people which version they built with. We can just ask for --caffe2_version gflag or get core.build_options from python.
Also adds CMAKE_INSTALL_RPATH_USE_LINK_PATH - without it wasn't building on my Mac. How should it be tested?
Closes https://github.com/caffe2/caffe2/pull/1271
Reviewed By: bddppq
Differential Revision: D5940750
Pulled By: dzhulgakov
fbshipit-source-id: 45b4c94f67e79346a10a65b34f40fd258295dad1
Summary:
This brings proper versioning in Caffe2: instead of manual version macros, this puts the version information in CMake (replacing the TODO bwasti line) and uses macros.h.in to then generate the version in the C++ header.
A few misc updates:
- Removed the mac os rpath, verified on local macbook that it is no longer needed.
- Misc updates for caffe2 ready:
- Mapped cmake/Cuda.cmake with gloo's setting.
- upstreamed third_party/nccl so it builds with cuda 9.
- Separated the Caffe2 cpu dependencies and cuda dependencies
- now libCaffe2_CPU.so do not depend on any cuda libs.
- caffe2 python extensions now depend on cpu and gpu separately too.
- Reduced the number of unused functions in Utils.cmake
Closes https://github.com/caffe2/caffe2/pull/1256
Reviewed By: dzhulgakov
Differential Revision: D5899210
Pulled By: Yangqing
fbshipit-source-id: 36366e47366c3258374d646cf410b5f49f95767b
Summary:
The problem:
Building caffe2 fails because the installed directory contains "anaconda".
The cause:
Compiling Gloo will generate a new config.h file in the binary folder.
If we put the original config.h in front, the compiler will complain "Expected GLOO_USE_CUDA to be defined".
~~~Switch the positions of the include folders can solve the problem.~~~
Function caffe2_include_directories in cmake/Utils.cmake is a little bit hacky. If the directory contains "anaconda", it will append the new include directory after existing include path. Otherwise it will insert the directory before the path. So in the first case, the directories are inserted in order, and in the latter one, they are inserted reversely.
The solution:
See the commit.
pietern #1121
Closes https://github.com/caffe2/caffe2/pull/1258
Reviewed By: Yangqing
Differential Revision: D5907167
Pulled By: houseroad
fbshipit-source-id: 2cb3916e7e0313ebc3be3d1666bfa14bbf479607
Summary:
This operator allows the use of Torch's underlying TH libraries (TH, THC, THNN, and THCUNN)
through the ATen tensor library. Use of the operator is described in the README.
The operator itself is generated from ATen's Declarations.yaml file which describes its public API.
Closes https://github.com/caffe2/caffe2/pull/1235
Reviewed By: dzhulgakov
Differential Revision: D5876944
Pulled By: zdevito
fbshipit-source-id: b558e8563a5e82a0e6278705a4a359bd7df4e70a
Summary:
After this, windows should be all green.
Closes https://github.com/caffe2/caffe2/pull/1228
Reviewed By: bwasti
Differential Revision: D5888328
Pulled By: Yangqing
fbshipit-source-id: 98fd39a4424237f2910df69c8609455d7af3ca34
Summary:
If Caffe2 used the packaged NCCL version then the Gloo build will try
to use it as well. To make sure the NCCL build has completed we need
to add an explicit dependency between the two.
Another subtle change here is that we add the PROJECT_BINARY_DIR to
the include path, since that is where the generated <gloo/config.h>
resides. Without this path Caffe2 includes the empty config.h from the
source tree.
Closes https://github.com/caffe2/caffe2/pull/1170
Differential Revision: D5779002
Pulled By: pietern
fbshipit-source-id: 9bc0d41f01a9b0f023d71bc4dee128a77eec1712
Summary:
Here is the buggy behavior which this change fixes:
* On the first configure with CMake, a system-wide benchmark installation is not found, so we use the version in `third_party/` ([see here](https://github.com/caffe2/caffe2/blob/v0.8.1/cmake/Dependencies.cmake#L98-L100))
* On installation, the benchmark sub-project installs its headers to `CMAKE_INSTALL_PREFIX` ([see here](https://github.com/google/benchmark/blob/4bf28e611b/src/CMakeLists.txt#L41-L44))
* On a rebuild, CMake searches the system again for a benchmark installation (see https://github.com/caffe2/caffe2/issues/916 for details on why the first search is not cached)
* CMake includes `CMAKE_INSTALL_PREFIX` when searching the system ([docs](https://cmake.org/cmake/help/v3.0/variable/CMAKE_SYSTEM_PREFIX_PATH.html))
* Voila, a "system" installation of benchmark is found at `CMAKE_INSTALL_PREFIX`
* On a rebuild, `-isystem $CMAKE_INSTALL_PREFIX/include` is added to every build target ([see here](https://github.com/caffe2/caffe2/blob/v0.8.1/cmake/Dependencies.cmake#L97)). e.g:
cd /caffe2/build/caffe2/binaries && ccache /usr/bin/c++ -I/caffe2/build -isystem /caffe2/third_party/googletest/googletest/include -isystem /caffe2/install/include -isystem /usr/include/opencv -isystem /caffe2/third_party/eigen -isystem /usr/include/python2.7 -isystem /usr/lib/python2.7/dist-packages/numpy/core/include -isystem /caffe2/third_party/pybind11/include -isystem /usr/local/cuda/include -isystem /caffe2/third_party/cub -I/caffe2 -I/caffe2/build_host_protoc/include -fopenmp -std=c++11 -O2 -fPIC -Wno-narrowing -O3 -DNDEBUG -o CMakeFiles/split_db.dir/split_db.cc.o -c /caffe2/caffe2/binaries/split_db.cc
This causes two issues:
1. Since the headers and libraries at `CMAKE_INSTALL_PREFIX` have a later timestamp than the built files, an unnecessary rebuild is triggered
2. Out-dated headers from the install directory are used during compilation, which can lead to strange build errors (which can usually be fixed by `rm -rf`'ing the install directory)
Possible solutions:
* Stop searching the system for an install of benchmark, and always use the version in `third_party/`
* Cache the initial result of the system-wide search for benchmark, so we don't accidentally pick up the installed version later
* Hack CMake to stop looking for headers and libraries in the installation directory
This PR is an implementation of the first solution. Feel free to close this and fix the issue in another way if you like.
Closes https://github.com/caffe2/caffe2/pull/1112
Differential Revision: D5761750
Pulled By: Yangqing
fbshipit-source-id: 2240088994ffafdb6eedb3626d898b505a4ba564
Summary:
Use HINTS instead of PATHS for find_library so that you can specify
-DNCCL_ROOT_DIR and it will use this NCCL installation regardless of
what else is installed on your system. Also add a path hint to include
the default base path for NCCL 2 libraries.
Closes https://github.com/caffe2/caffe2/pull/1152
Reviewed By: Yangqing
Differential Revision: D5740053
Pulled By: pietern
fbshipit-source-id: 43f0908a63e8a9b90320dece0bbb558827433b48
Summary:
I ran into an issue where a subset of packages were found in the
Anaconda path. This path also contained includes for other packages
and the Anaconda path inadvertently took precendence over the intended
include path. The new `caffe2_include_directories` helper is a hacky
attempt to "fix" this by deprioritizing Anaconda paths in the hope
that intended include paths are searched before Anaconda.
Closes https://github.com/caffe2/caffe2/pull/1121
Reviewed By: Yangqing
Differential Revision: D5701819
Pulled By: pietern
fbshipit-source-id: 908284cd4ea6c8167774e4e3fcc4dc0ca8a23110
Summary:
The PATHS suggestion to find_library is searched after everything
else. By using HINTS, it searches CUDNN_ROOT_DIR much earlier, avoiding
potential conflicts with other paths that have the CuDNN header.
Closes https://github.com/caffe2/caffe2/pull/1122
Reviewed By: Yangqing
Differential Revision: D5701822
Pulled By: pietern
fbshipit-source-id: 3f15757701aff167e7ae2a3e8a4ccf5d96763a0c
Summary:
Using file(WRITE) caused the file to be rewritten for every CMake
reconfigure, which was causing unnecessary full rebuilds of the project
even when no source files changed.
The new strategy has the added benefit of enforcing that the macros.h file
is always generated correctly. When the main project relies on this
header for macro definitions (instead of relying on add_definitions()),
we can be more confident that the project will build correctly when used
as a library (which is the whole point of the macros.h file).
Upsides:
* No more unnecessary rebuilds
* Higher confidence that the project will compile properly as a third-party library
Downsides:
* Developers need to add an entry to `macros.h.in` whenever they would have added a new definition with `add_definitions()`
Closes https://github.com/caffe2/caffe2/pull/1103
Differential Revision: D5680367
Pulled By: Yangqing
fbshipit-source-id: 4db29c28589efda1b6a3f5f88752e3984260a0f2
Summary:
I discovered this while investigating more build-caching issues like https://github.com/caffe2/caffe2/pull/1103.
> If a relative path is given it is interpreted relative to the value of the CMAKE_INSTALL_PREFIX variable.
https://cmake.org/cmake/help/v3.0/command/install.html
This is a non-functional change - it just makes the code a bit easier to read. I verified locally that the resulting install directories are identical.
Closes https://github.com/caffe2/caffe2/pull/1111
Differential Revision: D5677328
Pulled By: Yangqing
fbshipit-source-id: 9bb1bfe85fc0bc54a9b7ce33cc31e45ea061d21e
Summary:
As part of the cuda 9 move we have decided to deprecate the cnmem path
as it seems to be superceded by cub if one needs a memory pool.
Closes https://github.com/caffe2/caffe2/pull/1104
Differential Revision: D5647672
Pulled By: Yangqing
fbshipit-source-id: 988af5bf63e24efa1b631fd91ddb58e798ffc5c6
Summary:
This PR replaces PR #464. It requires C+11 support using the
new CMake variables (`CMAKE_CXX_STANDARD`, `CMAKE_CXX_STANDARD_REQUIRED`,
etc.) when CMake is version 3.1 or above. Otherwise, if CMake is older
(e.g. Ubuntu 14.04) it falls back to using the -std=c++11 flag and
issues a warning.
This PR is based on the comment from Yangqing:
https://github.com/caffe2/caffe2/pull/464#issuecomment-305376923
The corresponding line in cmake/MiscCheck.cmake is removed in order to
reduce redundancy. Another option would be to move the C++11 logic to MiscCheck.cmake.
Closes https://github.com/caffe2/caffe2/pull/1027
Differential Revision: D5590646
Pulled By: Yangqing
fbshipit-source-id: 11ac63fbeaab7a1da02115549e214f9c529f1873
Summary:
While I was trying to make a quick oss cmakefile, I found that some of the
ios source files are out of sync with the most code changes. This diff should
fix the issues.
I manually ran cmake on the oss side with scripts/build_ios.sh to make sure
things pass.
Reviewed By: ajtulloch
Differential Revision: D5582265
fbshipit-source-id: 2636d353d32fcd8fb7087385b9bbed8476e33e74
Summary:
Adds support for the CUDA 9 toolkit.
Includes new fp16 data type fixes, and changes to warp-synchronous programming. Also updates CUB third-party repo for CUDA 9 support.
Closes https://github.com/caffe2/caffe2/pull/853
Differential Revision: D5548507
Pulled By: Yangqing
fbshipit-source-id: c7fd2edb623f2aa8c67b9a1000efc8f71e6832ab
Summary:
I successfully built caffe2 using MSVC 2015 and the Ninja Generator. I use vcpkg to build glfags, glog, lmdb and protobuf. Here is my build procedure:
1. Install vcpkg and set it up according to vcpkg docs
2. Install dependencies
```
$> vcpkg install gflags glog lmdb protobuf eigen3 --triplet x64-windows-static
```
3. Run CMake with this batch file
```Batch
setlocal
if NOT DEFINED VCPKG_DIR ( echo "Please defined VCPKG_DIR" && exit /b 1 )
if NOT DEFINED CMAKE_BUILD_TYPE set CMAKE_BUILD_TYPE=Release
if NOT DEFINED BUILD_DIR set BUILD_DIR=build_%CMAKE_BUILD_TYPE%
if NOT DEFINED USE_CUDA set USE_CUDA=OFF
call "%VS140COMNTOOLS%\..\..\VC\vcvarsall.bat" amd64
if NOT EXIST %BUILD_DIR% (mkdir %BUILD_DIR%)
pushd %BUILD_DIR%
set CMAKE_GENERATOR=Ninja
set ZLIB_LIBRARY=%VCPKG_DIR%\installed\x64-windows-static\lib\zlib.lib
cmake -G"%CMAKE_GENERATOR%" ^
-DBUILD_SHARED_LIBS=OFF ^
-DCMAKE_VERBOSE_MAKEFILE=1 ^
-DBUILD_TEST=OFF ^
-DBUILD_SHARED_LIBS=OFF ^
-DCMAKE_BUILD_TYPE=%CMAKE_BUILD_TYPE% ^
-DUSE_CUDA=%USE_CUDA% ^
-DZLIB_LIBRARY:FILEPATH="%ZLIB_LIBRARY%" ^
-DVCPKG_TARGET_TRIPLET=x64-windows-static ^
-DVCPKG_APPLOCAL_DEPS:BOOL=OFF ^
-DCMAKE_TOOLCHAIN_FILE:FILEPATH=%VCPKG_DIR%\scripts\buildsystems\vcpkg.cmake ^
-DPROTOBUF_PROTOC_EXECUTABLE:FILEPATH=%VCPKG_DIR%\installed\x64-windows-static\tools\protoc.exe ^
..\
ninja
popd
endlocal
```
Closes https://github.com/caffe2/caffe2/pull/880
Differential Revision: D5497384
Pulled By: Yangqing
fbshipit-source-id: e0d81d3dbd3286ab925eddef0e6fbf99eb6375a5
Summary:
libpthreadpool is needed during the linking stage and is missing when user chooses to use an external nnpack installation (from system libraries).
Fixes GitHub issue #459.
Detailed discussion on [this comment](https://github.com/caffe2/caffe2/issues/459#issuecomment-308831547).
Closes https://github.com/caffe2/caffe2/pull/808
Differential Revision: D5430318
Pulled By: Yangqing
fbshipit-source-id: 5e10332fb01e54d8360bb929c1a82b0eef580bbb
Summary:
MKL on windows works with this change. Tested with MKL 2017 Update 3 (https://software.intel.com/en-us/articles/intel-math-kernel-library-intel-mkl-2017-release-notes).
Should fix#544
With MKL 2017 Update 3 #514 should not happen too.
Note: I used Anaconda which ships with its own MKL, so I had to make sure that the MKL 2017 Update 3 version was loaded by replacing the .dll in the `%AnacondaPrefix%\Library\bin` folder. Otherwise, numpy would load it's own version and I would have all sorts of missing procedures errors. Now that the same version is available through `conda` this is easily fixed with `conda install mkl==2017.0.3`
Closes https://github.com/caffe2/caffe2/pull/929
Differential Revision: D5429664
Pulled By: Yangqing
fbshipit-source-id: eaa150bab563ee4ce8348faee1624ac4af477513
Summary:
I closed https://github.com/caffe2/caffe2/pull/736 because one of these variables should be used after all.
Here's how C1 uses this variable: https://github.com/BVLC/caffe/blob/rc5/cmake/Targets.cmake#L116
Without this fix, there is a race condition in the parallel build leading to this error:
```
make[2]: *** No rule to make target `../third_party/NNPACK/lib/libnnpack.a', needed by `caffe2/libCaffe2_CPU.so'.
```
Closes https://github.com/caffe2/caffe2/pull/737
Differential Revision: D5211794
Pulled By: Yangqing
fbshipit-source-id: 9e368f09b01edaf86252727adc6f6cc40d244e29
Summary:
This is a little excessive:
```
CMake Warning at cmake/Dependencies.cmake:201 (find_package):
By not providing "FindEigen3.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "Eigen3", but
CMake did not find one.
Could not find a package configuration file provided by "Eigen3" with any
of the following names:
Eigen3Config.cmake
eigen3-config.cmake
Add the installation prefix of "Eigen3" to CMAKE_PREFIX_PATH or set
"Eigen3_DIR" to a directory containing one of the above files. If "Eigen3"
provides a separate development package or SDK, be sure it has been
installed.
Call Stack (most recent call first):
CMakeLists.txt:72 (include)
```
Closes https://github.com/caffe2/caffe2/pull/729
Differential Revision: D5183059
Pulled By: Yangqing
fbshipit-source-id: d17d5d06a50abb50f9978d022ddc4918e991079d
