CLOGS User Manual

At present CLOGS is only supported with GCC on GNU/Linux and with Visual C++ 2010 on Windows. The code for the library itself is portable, but some aspects of the build system and test suite require porting.

It has been tested on an NVIDIA GeForce 480 GTX, an AMD Radeon HD 6790 and on a CPU using the AMD APP SDK. It is expected to work on any OpenCL implementation but this is untested, and it is unlikely to have optimal performance.

The following dependencies are required to build CLOGS:

CLOGS uses the waf build system. The build system is included in the distribution, so you do not need to download it separately.

The first step is to configure the build. This is done by running

$ python waf configure

This will check that the necessary headers can be found. OpenCL headers are not always installed in the normal include paths. You can explicitly specify the location for them by running

$ python waf configure --cl-headers=path

Note that path should be the directory containing the CL directory. If other libraries also need to be added to the include or link paths, you should use compiler-specific environment variables, such as INCLUDE and LIBPATH for MSVC or CPATH and LIBRARY_PATH for GCC.

You can control where CLOGS will be installed by using --prefix, as well as the other standard autoconf options (run waf --help for a list).

The configuration will automatically detect whether doxygen and xsltproc are present. However, you can disable them with --without-doxygen and --without-xsltproc.

If you need to debug into CLOGS, you can pass --variant=debug at configuration time to create a debug build, or --variant=optimized to create an optimized build with debug symbols.

Installation is done by running python waf install. Unless you have changed the installation directory, you will probably need to be root to do this, and you will also need to run ldconfig afterwards on a GNU system.

waf also supports a --destdir option that allows the entire directory tree to be placed into a subdirectory rather than the root of the filesystem. This is intended for use with package management tools.

After installing, it is necessary to tune CLOGS for your OpenCL devices. See the next section for details.

CLOGS uses autotuning to choose good tuning parameters for each device. This helps with performance, but unfortunately also means that you need to run the autotuner before using CLOGS. This is done simply by running clogs-tune. You should do this on a system that is otherwise idle, to avoid skewing the results.

This can take a long time, particularly for CPU devices. If you have OpenCL CPU devices but don't intend to use them with CLOGS, you can save time by passing --cl-gpu to clogs-tune to only tune for GPU devices.

You need to do autotuning when you first install CLOGS or when you do an upgrade. You will also need to do autotuning when adding a new OpenCL device. The previous tuning is remembered, so this will only cause new configurations to be tuned. If for some reason you need to re-do tuning, you can pass --force to clogs-tune.

On POSIX systems, the results of autotuning are stored in $HOME/.clogs/cache. On Windows, they are stored in the local application data directory.

We start with a simple example showing how to do a scan (prefix sum) operation. This operation is typically combined with an algorithm that produces a variable amount of output per work item, to allocate positions in an output buffer. In a first pass, each work-item writes the number of output elements to a buffer, which we'll call counts. A prefix sum is then run over counts, which replaces each value with the sum of all values strictly before it. The second pass of the algorithm then uses the values in counts as the initial positions to start writing output to another buffer.

In the sample code, we assume that an OpenCL context, device and command queue have already been created using the C++ bindings, and that the counts array has already been written.

#include <clogs/clogs.h>
...
clogs::Scan scanner(context, device, clogs::TYPE_UINT);
...
scanner.enqueue(queue, dataBuffer, numElements);

The above code first allocates an object that can be used to perform scans. The constructor handles compilation of the internal kernels and allocation of internal storage. These objects are quite slow to create and should be reused where possible.

The last line enqueues to queue the kernel launches needed to perform the scan. There are several optional parameters we have not shown. These include a vector of events to wait for, and an output parameter which returns an event that is signaled when the scan is complete. These work in the same way as the other enqueuing functions in the OpenCL C++ API, and allow scans to be combined with other kernel launches in a dependency graph.

Now let us look at a slightly more complex sorting example. Suppose we have unsigned 20-bit keys (stored in uints), and float4 values. In CLOGS, keys and values must be held in separate buffers (which reduces overall bandwidth), so let us say that keys are in keys and values in values. Furthermore, let us suppose that the vector wait contains a list of events for work that will generate the keys and values, and that we want an event that will be signaled when the sort is complete. Then the code we want is

#include <clogs/clogs.h>
...
clogs::Radixsort sorter(context, device, clogs::TYPE_UINT, clogs::Type(clogs::TYPE_FLOAT, 4));
sorter.enqueue(queue, keys, values, numElements, 20, &wait, &event);

Notice that we used clogs::Type to specify the type of the values. This is a C++ type that encodes an OpenCL built-in type. In this case we specified the base type (float) and vector length (4). We could also do this for the key type, but it was not necessarily since there is an implicit conversion from clogs::TYPE_UINT to clogs::Type.

Also notice that we explicitly specified how many bits are used in the key. This parameter is optional (passing zero is the same as not passing it), but specifying it may reduce the number of passes and hence improve performance.

Both of the classes we can covered so far have additional features, which are described in the reference manual.

For a complete example of using the API, refer to tools/clogs-benchmark.cpp, which is a tool for benchmarking the performance of CLOGS when sorting. This tool is also installed when you install CLOGS, so you can obtain estimates of sorting performance from the command-line.

The classes in this API (clogs::Scan and clogs::Radixsort) store internal state that is used by the enqueued work. There are two limitations on reentrance:

The source of CLOGS is compiled with __CL_ENABLE_EXCEPTIONS defined, so any errors caused internally will be thrown as a cl::Error. Additionally, the reference documentation lists some higher-level conditions which will be signaled by throwing cl::Error (for example, if numElements was too large and would have overflowed the buffer).

The other type of exception that may occur is clogs::InternalError. This is only thrown for unexpected errors with the implementation, such as when the source for one of the kernels fails to compile. If autotuning has not been done, a clogs::CacheError (a subclass of clogs::InternalError) is thrown.

Each object allocated through the API allocates a small amount of OpenCL memory, whose size depends only on the arguments to the constructor. Additionally, clogs::Radixsort allocates temporary buffers as part of enqueue to hold partially-sorted copies of the keys and values.

For some uses, it is desirable to avoid repeatedly allocating and freeing these temporary buffers, and so it is possible for the user to specify buffers to use by calling setTemporaryBuffers (see the reference documentation for details).

The scanning and sorting objects are non-copyable, to avoid accidently triggering expensive copies of OpenCL objects. They need to be passed by reference.

The event returned by the various enqueue commands is suitable for event ordering, but it does not work well with OpenCL event profiling functions to determine how much time is spent on the GPU. For this purpose, one should call setEventCallback on the clogs::Radixsort or clogs::Scan object. The registered callback will be called once for each CL command enqueued, passing the associated event. Note that the callback is called during the enqueue call, rather than when the event completes; it is up to you to defer querying the profiling information until the event is complete.