back to main page

Documentation

This document describes in detail the black box optimization competition and its software interface.

Introduction

The black box optimization competition challenges its participants to achieve as good as possible solution to unknown optimization problems through a black-box interface within a given budget of objective function evaluations.

The competition is organized into tracks, which again consist of problems. Each problem is defined by the dimension d of its search space Ω, which for all problems takes the form Ω = [0,1]d. For each problem the participant has a predefined budget B of evaluations f(x) of an unknown function f at his disposal. Hence f can be evaluated in B different points x ∈ Ω. The goal is to obtain an as low as possible value f(x) for the best point found (single-objective case), or to dominate the largest possible share of the objective space if f is vector-valued (multi-objective optimization). This setting is known as black box optimization.

Each participant is identified by an account (user name), which is protected with a password. User accounts are created by the competition organizers on request. Write to blackboxcompetition@gmail.com and be sure to include your name, affiliation and your preferred user name. Accounts can be managed through a web interface, which is accessible through the login box on the top right of the main page.

Besides the actual competition track there exists special tracks named "trial" and "trialMO". These tracks consist of dummy problems which are, in number and dimension (but not necessarily in their other characteristics), comparable to competition tracks. These tracks should be used to thoroughly test optimization software before switching to a competition track. The special property of the trial tracks is that the optimization can be repeated many times, since the number of objective function evaluations can be reset through the web interface. Note that a user can solve each problem in a competition track only once. Hence, it must be made double sure on the trial tracks that the optimization software works exactly as expected.

Web Interface

The web interface offers the following functionality:

Architecture

The overall software architecture of the black box optimization competition is based on a client/server model. On the hardware level there exists a server machine that collects data, while the actual optimization tasks are running on client machines at the participant's side. This means in particular that the competition does not provide computing resources to participants. Instead participants use any computing resources at their disposal to participate in the competition. The only requirement is that clients are connected to the internet so that results can be communicated to the competition server.

On the software level participants do not get in touch with the competition server directly. Instead their optimization software talks to the server through a dynamic library (dll on windows, so (shared object) on linux, and dylib on MacOS). This library is provided as a binary file. It exports a well-defined interface, and otherwise truly is a black box to participants. Its interface functions are explained in detail in the next section. The maybe most important of these is the evaluate(...) function, which is the actual black box optimization interface.

Participants are expected to execute their preferred black box optimization algorithm on each problem in a competition track. However, the program implementing the algorithm must organize a few things before it can get started, e.g., log into the system, select a problem for optimization, perform proper error handling, etc. This program can be written in any programming language capable of invoking the dynamic library or opening a TCP/IP socket connection. Example programs (implementing random search and the classic downhill simplex method) are provided in C, C++, Matlab/Octave, Python, and Java. Socket-based access requires a standalone proxy program that comes with the competition software collection. It allows to access all library functions transparently with an ASCII-based protocol through the socket connection, however, with increased latency as compared to direct library calls (described below).

Dynamic Library Setup

The client (user) program most connect to the dynamic link library that is at the heart of the black box optimization competition software. The library is interfaced in plain C. It consists of two files, the actual library (bbcomp.dll on Windows, libbbcomp.so on Linux, and libbbcomp.dylib on MacOS), and an accompanying header file bbcomplib.h. The easiest way to learn how to connect to this library is to have a look at the example clients (included in the software). The client's functionality is explained below.

The following additional information may be useful, e.g., when accessing the library from a programming language not covered by the examples. In most of these cases you should get away with writing minimal glue code, probably at least partly written in C. The header file bbcomplib.h provides four macros: DLL_DECLARATIONS, DLL_LOAD, DLL_LOAD_STATUS, and DLL_CLOSE. They are supposed to be used as follows:

#include <bbcomplib.h>
DLL_DECLARATIONS

int main()
{
  DLL_LOAD
  if (DLL_LOAD_STATUS < 0)
  {
    printf("error loading library\n");
    exit(EXIT_FAILURE);
  }
  ...your actual code goes here...
  DLL_CLOSE
}

After DLL_LOAD and before DLL_CLOSE, and as long as DLL_DECLARATIONS is in scope, the functions exported by the dynamic library can be called. What happens behind the scenes is that DLL_DECLARATIONS declares (global) variables, which happen to be function pointers to the library's interface functions. DLL_LOAD initialized these pointers. The DLL_LOAD_STATUS variable contains a negative value after DLL_LOAD if initialization failed, and a positive value otherwise. DLL_CLOSE performs cleanup operations.

Library Functions

This section acts as a reference documentation for the library. It exports a plain C-language interface. The interface should hence be accessible from many programming languages.

Note:

The library is stateful and not thread safe. Hence it cannot be used by more than one optimizer running in one thread per process at a time. However, it can be accessed by multiple processes per machine and even from multiple machines, which allows for massive parallelism. We recommend users to limit themselves to at most 10 simultaneous connections.


int configure(int history, const char* logfilepath);

Set global configuration options. This call is optional, however, it can only occur before login.

The function returns 0 on error.

Note: It is strongly recommend to leave logging enabled if possible. The logging mechanism safeguards against failures of all kinds, such as network problems, power outage, program crashes, and many more. Recall once more that you have only a single trial.


int login(const char* username, const char* password);

Login to the system. This call results in communication with the competition server. It validates the user's credentials. Multiple concurrent logins with the same credentials are possible to allow for parallel evaluation of problems on different cores/machines, but not from multiple threads within the same process. The account becomes a persistent property of the library state; subsequent function calls refer to this account. The function returns 0 on error.


int numberOfTracks();

Return the number N of tracks available to the user. Valid track indices range from 0 to N-1. The function returns 0 on error.


const char* trackName(int trackindex);

Return the name of the track. Tracks are identified by their name, not by their index (which may change depending on user rights). The function returns a NULL pointer on error.


int setTrack(const char* trackname);

Select a track. Subsequent function calls refer to this track. The function returns 0 on error.


int numberOfProblems();

Return the number of problems M in the current track. Valid problem IDs range from 0 to M-1. The function returns 0 on error.


int setProblem(int problemID);

Select a problem within the chosen track for optimization. Subsequent function calls refer to this problem. The function returns 0 on error.


int dimension();

Return the dimension of the search space of the current problem. The function returns 0 on error.


int numberOfObjectives();

Return the number of objectives, i.e., the number of real values returned by the evaluate function. For (standard) single-objective optimization this function always returns 1. For multi-objective optimization this function returns a value greater than one. The function returns 0 on error.


int budget();

Return the evaluation budget for the current problem. This is the total available budget, not the remaining budget (which can be obtained by subtracting the result of the evaluations function). Calling evaluate more than this number of times on the current problem will fail. The function returns 0 on error.


int evaluations();

Return the number of evaluations so far for the current problem. The function returns -1 on error.


int evaluate(double* point, double* value);

Evaluate the objective function.

This call results in communication with the competition server. The function returns 0 on error.

Note:

This function fails if the predefined budget of function evaluations is used up.

Note:

It is assumed that all variables are within the unit hypercube, i.e., the box constraint
for all i:  0 ≤ point[i] ≤ 1
is respected. Evaluation of an infeasible point, i.e., violation of the constraint results in an error; the objective function is not evaluated.


int history(int index, double* point, double* value);

Recover a previous evaluation from the log file. Note: if logging was disabled through configure or if the log file was removed or corrupted then this function will fail.

The function returns 0 on error.


const char* errorMessage();

This function returns a human readable error message for the previous error.

The Socket-based Proxy

In some programming environments it is non-trivial to access a dynamic library and its C-language interface. In this case an alternative access method is offered through a standalone executable acting as a proxy for the dynamic library. This proxy forwards the C-language interface to a client through a socket connection. This approach is used, e.g., for the Java frontend. The default port is 7000, which can be changed through a command line option so that multiple proxies can run on a single machine.

The protocol is line-based. A line is defined as a sequence of ASCII characters in the range 32 to 126, ending with code 10 (CR). The sequence 13+10 (LF+CR) is not recognized and results in an error. Each line received by the proxy is interpreted as a function call in the format

functionname(param1,param2,...,paramN)

There are four data types:

All functions return one or multiple values, also collected in one line. The C-function return value comes first, followed by "return arguments", separated with commas. Example communication:

sender: login("joe","secret")
 proxy: 1
sender: setTrack("trial")
 proxy: 1
sender: setProblem(0)
 proxy: 1
sender: dimension()
 proxy: 3
sender: numberOfObjectives()
 proxy: 1
sender: evaluate([0.92241 0.1817 0.76203])
 proxy: 1,3.4419
sender: history(0)
 proxy: 1,[0.92241 0.1817 0.76203],3.4419
sender: history(10000)
 proxy: 0
sender: errorMessage()
 proxy: "index out of range"
sender: cheat()
 proxy: ERROR,"unknown function 'cheat'"
sender: login(malformed,request,,,)
 proxy: ERROR,"failed to parse request parameters"

Note the difference between the evaluate library function and this example. The value parameter is actually an output of the function, hence it is not passed as a parameter but rather is returned as a second value after the success indicator return code "1". A similar consideration applies to the history function. Its first parameter is the time index. The parameters point and value are return values, hence they are not passed as parameters.

In case of a protocol violation the proxy returns the special sequence

ERROR,"this is a human readable error message" .

Note that this mechanism does not replace the role of the errorMessage function. It is only applied to communication errors between client and proxy.

Example Clients

This section describes the working principle of the example clients (available in various programming languages) and how they invoke the library functions.

Note:

All example clients should run out-of-the-box. They use a demonstration account called demoaccount. Note that this account cannot participate in the competition. It is limited to the "trial" and "trialMO" tracks. Error messages are expected if the client is run before the corresponding track opens or after it closes.

The example clients are intended to serve as templates and starting points for the implementation of actual competition entries. The example clients in all language execute the same sequence of commands. This sequence is discussed in the sequel for the dummy clients performing pure random search.

The first action is to call the configure function. This step is optional. It allows to define settings related to the evaluation history.

The first essential step is the login call. Simply provide you username and password to this function. It makes the dynamic library setup a socket connection to the competition server. This connection will remain open for the duration of the use of the library by the present process. The participant remains logged in for this duration, and all subsequent calls implicitly refer to the logged in participant. Multiple such connections can coexist per participant so that one participant can use multiple cores to solve problems in parallel.

Mostly for completeness there exist the functions numberOfTracks and trackName. They allow to obtain the list of tracks available to the participant.

A track is selected with the setTrack function. The track can be either hard-coded or obtained from the above list.

The function numberOfProblems returns the number of problems in the selected track.

A problem is selected with the setProblem function. This problem is now ready for optimization.

The functions dimension, numberOfObjectives, and budget return properties of the selected problem, while evaluations returns the number of evaluations of the objective function on this problem so far.

Once these problem characteristics have been obtained the demonstration program starts an extremely simple optimization strategy: random search with i.i.d. uniform sampling. Each attempt to solve the problem consists of sampling a search point x uniformly at random from the search space. This point is then evaluated by calling evaluate.

Now assume that the client program has crashed, say, due to a power outage. Then it would be fatal to lose the evaluations so far, since they are limited by a budget. The library helps out in this case by providing the history of previous evaluations at any time (unless this feature has been disabled by a call to configure, see above). The demo client program implements an extremely simple recovery strategy: when it detects that the optimization was run partially then instead of drawing samples it requests historic samples through the history function. This allows the algorithm to repeat its steps and to continue from exactly where it was at the time of the program crash. Of course, since pure random search is stateless there is no state to recover, however, the example program demonstrates the technique in principle by outputting newly obtained as well as historic samples.

After the optimization loop the program checks whether the budget was indeed used up. This check is only for demonstration purposes.

Finally the program terminates, which results in closing the connection to the competition server. Alternatively the program could continue to optimize other problems or even problems from other tracks. A typical setup is to add a loop over all problems of a track. Alternatively this loop can be external to the program, e.g., to take advantage of parallel hardware. However, for parallel processing take this note into account.

Known Problems

The following problems and bugs are known. All known bugs are fixed in the most recent version of the software, hence this list is mostly for historical completeness. If you encounter any of these problems then be sure to update to the most recent version of the BBComp software.