Reading a Campaign into Metrics

To extract pieces of data from the campaign of experiments you ran and feed it into Metrics, you need to use the Scalpel module of Metrics. Scalpel stands for “extraCt dAta of exPeriments from softwarE Logs” (sCAlPEL).

A campaign is basically read using the following:

from metrics.scalpel import read_campaign
my_campaign, my_configuration = read_campaign('path/to/campaign/file', log_level='WARNING')

Currently, two types of files can be given as input to Scalpel:

a JSON file containing a serialized form of the campaign (when you have already loaded your campaign in Metrics, and saved it for later use), or
a YAML file describing how to extract data from the campaign you ran.

In the first case, there is almost nothing to do, as the JSON file generated by Metrics already contains all the data needed by Scalpel (and the returned configuration will thus be None). In the second case, the following sections give more details on how to write a configuration file that describes your campaign (the returned configuration will be an object representation of this description).

Additionally, as you can see in the example above, a log_level parameter may be specified to the function read_campaign(). This allows to configure the minimum level of the parsing events that should be logged. Available levels are, from the lowest to the highest:

'TRACE'
'DEBUG'
'INFO'
'WARNING' (this is the default level)
'ERROR'

Parsing events are regularly logged by Scalpel to trace the extraction it performs, mostly for debugging purposes. For instance, activating a lower logging level may allow to identify why some data is missing for a particular experiment. However, it should be noted that Scalpel may become particularly verbose when doing so, which may affect performance. This is why this feature should only be activated for debugging purposes.

Metadata of the Campaign

In the YAML file, you first need to give elementary information about the campaign, such as its name and the date on which it has been run.

name: my-awesome-campaign
date: 2020-11-17

This information is used to identify your campaign, and is particularly interesting for the traceability of your experiments.

The YAML file must also contain the experimental setup on which the campaign took place, as in the following example:

setup:
  os: Linux CentOS 7 (x86_64)
  cpu: Intel XEON X5550 (2.66 GHz, 8 MB cache)
  gpu: Nvidia GeForce 256 SDR
  ram: 32GB
  timeout: 1800
  memout: 1024

Note that, for the setup description, only timeout and memout are required. The other values may be displayed in the reports generated by Metrics for reproducibility purposes.

Description of the Campaign Files

Scalpel is able to parse a wide variety of files that contain the output of the experiments you ran during your campaign. All the information describing the source of your campaign must be given in the source section of your YAML configuration file. The main key of this section is path, which lists the file(s) containing the data to extract.

source:
  path:
    - path/to/first/file
    - path/to/second/file

This key declares the list of the files (either regular files or directories, depending on the format of your campaign) that Scalpel will parse. Note that all files must have the same format.

All these files will be parsed sequentially, and their content will be merged into a single campaign. If these files represent distinct parts of your campaign (e.g., each file contains the result of a different experiment-ware), you may be interested in the extraction of metadata from the name of the file, described here.

If you only have one file containing all the results of your campaign, you may avoid the use of a list, and simply write the path of the file as the value for path:

source:
  path: path/to/single/file

In the following subsections, we present what you must add to the source field to configure Scalpel for parsing your campaign, depending on its format.

Parsing a CSV File

The CSV (Comma-Separated Values) format is often used to store experimental data. It is mainly a tabular format, which has an (optional) header line giving the titles of the column. Each of the remaining lines corresponds to the data collected during an experiment.

Depending on the variant, columns may be separated by:

a comma (,), giving the default csv format,
a semicolon (;), giving the csv2 format, or
a tabulation (\t), giving the table format.

To specify that your campaign is in one of these formats, you need to add the following to your YAML configuration file:

source:
  path: path/to/my/file.csv
  format: csv

Actually, the format may be omitted in this example, as the extension of the file already tells Scalpel that the file is in the (classical) csv format. Similarly, if you specify as path the files path/to/my/file.csv2 or path/to/my/file.table, you may omit the format, as Scalpel will infer that such files use the csv2 and table formats, respectively.

You may also have more “exotic” CSV-like files, which do not use a standard separator or quote character (by default, " is used as quote character). If this is the case, you may describe them by adding the following keys in the source section:

source:
  quote-char: "%"
  separator: "|"

In the example above, the quote character is % and the columns are separated by the character |.

Finally, you may have a header for your CSV file, or not. By default, the first line is considered as a header line, and is used to identify the values parsed in the other lines as experimental data. If you do not have a header line, add the following key:

source:
  has-header: false

In this case, values will be identified by the index of the corresponding column, as a string (starting from "0"). Note that, in this case, Scalpel’s naming convention cannot be followed. As such, do not forget to specify the mapping of the columns in the text file to fit Scalpel’s needs (see below for more details). You must also do so as long as the name of the columns in your CSV files do not fit Scalpel’s expectations.

Parsing an “Evaluation” File

If you are interested in analyzing the results of a campaign run with the so-called “Evaluation” platform (such as, for instance, the results of the XCSP’19 competition, we provide a parser to read the “results of individual jobs as text file” provided by this platform (as the one of the XCSP’19 competition, available here).

To do so, specify the following in your YAML configuration file:

source:
  path: path/to/result/file.txt
  format: evaluation

As this platform does not use in general the same naming convention as that of Scalpel, do not forget to specify the mapping of the columns in the text file to fit Scalpel’s needs (see below for more details).

Parsing a “Reverse” CSV File

We call a CSV file “reverse” when each line in this file corresponds to an input, and the columns to the different statistics collected for the experiment-wares run during the campaign. Here is an example of such a file:

xp-ware-a,xp-ware-b,xp-ware-c
0.01,0.02,0.03

In this example, we consider a campaign that run three experiment-wares, namely xp-ware-a, xp-ware-b and xp-ware-c. Each column is by default interpreted as the CPU time of the corresponding experiment, as this is the only statistic required for an experiment. Also, note that no input is specified in this example. This is tolerated, as each line in such a format maps to exactly one input. However, we strongly recommend specifying the name of the input file, especially because it makes easier the interpretation of the experimental results and their reproducibility.

A more complete example of a “reverse” CSV file is given below:

input,xp-ware-a.cpu_time,xp-ware-a.memory,xp-ware-b.cpu_time,xp-ware-b.memory,xp-ware-c.cpu_time,xp-ware-c.memory
input-a,0.01,10,0.02,20,0.03,30

Here, we collect more statistics, as we consider both the cpu_time and memory needed for an experiment. Those names are used to identify the corresponding statistics in the representation of the experiment. In the example above, the experiment-ware and the statistics identifiers are separated with a dot (.), which is the default. If you want to specify a different separator, you can specify it in the YAML configuration as follows (make sure not to use the same separator as for the columns):

source:
  title-separator: "!"

To configure how a reverse CSV file is parsed, you can also use the same properties as those used in classical CSV file (see the previous section), and specify one of the formats reverse-csv, reverse-csv2 or reverse-table (using the same naming convention as before).

Parsing Raw Data from a File Hierarchy

If you have gathered the output of your experiment-wares in a directory, Scalpel can explore the file hierarchy rooted at this directory and extract all relevant data for you. We support three different kinds of file hierarchies, which are described below.

Note that, by default, Scalpel does not follow symlinks when exploring a file hierarchy. For each of the configurations below, you may alter this behavior by adding the following to the source section of the YAML file:

source:
  follow-symlinks: true

One File per Experiment

In this case, the file hierarchy being explored is supposed to contain exactly one (regular) file per experiment. You can configure Scalpel to consider such a file hierarchy using the following description:

source:
  path: /path/to/my-experiment-directory
  format: one-file

Let us consider an example to illustrate how Scalpel extracts data based on this configuration. Suppose that the file hierarchy to explore has the following form:

my-experiment-directory
    + experiment-a.log
    + experiment-b.log
    ` more-experiments
        + experiment-c.log
        ` experiment-d.log

Here, Scalpel will recursively explore the whole file hierarchy, and will parse all regular files, provided that these files are specified in the data section of the YAML configuration file (see the dedicated documentation here for more details). Each file experiment-a.log, experiment-b.log, experiment-c.log and experiment-d.log will be considered as the output of a single experiment.

Note that these files may have common formats (such as JSON, XML or CSV) or may also be the raw output of the solver. More details on how to retrieve relevant information from these files are given here.

Multiple Files per Experiment

In this case, the file hierarchy being explored is supposed to contain a set of (regular) files per experiment. The name of the files (without their extensions) will be used to identify each experiment. You can configure Scalpel to consider such a file hierarchy using the following description:

source:
  path: /path/to/my-experiment-directory
  format: multi-files

Let us consider an example to illustrate how Scalpel extracts data based on this configuration. Suppose that the file hierarchy to explore has the following form:

my-experiment-directory
    + experiment-a.out
    + experiment-a.err
    + experiment-b.out
    + experiment-b.err
    ` more-experiments
        + experiment-c.out
        + experiment-c.err
        + experiment-d.out
        ` experiment-d.err

Here, Scalpel will recursively explore the whole file hierarchy, and will parse all regular files, provided that these files are specified in the data section of the YAML configuration file (see the dedicated documentation here for more details). In this case, the files experiment-a.out and experiment-a.err, for instance, will be considered as outputs of the same experiment (they are both named experiment-a).

Note that these files may have common formats (such as JSON, XML or CSV) or may also be the raw output of the solver. More details on how to retrieve relevant information from these files are given here.

One Directory per Experiment

In this case, the file hierarchy being explored is supposed to have one directory that contain the output files of each experiment. The name of the files inside this directory may be arbitrary (and even the same from one experiment to another). You can configure Scalpel to consider such a file hierarchy using the following description:

source:
  path: /path/to/my-experiment-directory
  format: dir

Let us consider an example to illustrate how Scalpel extracts data based on this configuration. Suppose that the file hierarchy to explore has the following form:

my-experiment-directory
    + experiment-a
    |   + stdout
    |   + stderr
    + experiment-b
    |   + stdout
    |   + stderr
    ` more-experiments
        + experiment-c
        |   + stdout
        |   + stderr
        ` experiment-d
            + stdout
            + stderr

Here, Scalpel will recursively explore the whole file hierarchy, and will consider each directory containing regular files as an experiment. All the regular files contained in this directory will thus be considered as outputs of the corresponding experiments, as long as these files are specified in the data section of the YAML configuration file (see the dedicated documentation here for more details). For instance, the stdout and stderr files in the directory experiment-a will be considered as output files of the experiment experiment-a, and will thus be used together to extract relevant information for this experiment.

Note that these files may have common formats (such as JSON, XML or CSV) or may also be the raw output of the solver. More details on how to retrieve relevant information from these files are given here.

Parsing Unsupported Formats

When developing Scalpel, we tried to think about as many campaign formats as possible. However, it may happen that you need to parse a campaign that uses a format that is not recognized (yet) by Scalpel. If this is the case you may write your own parser. by extending the class CampaignParser. This class must define a constructor taking as argument a CampaignParserListener and a ScalpelConfiguration. To give you ideas on how to write such a parser, you may have a look to the source of our parsers.

Then, add the class of your parser to your YAML configuration file as follows:

source:
  parser: my.completely.specified.AwesomeParser

Scalpel will dynamically instantiate your parser, and will then use it to parse the campaign. To make this possible, you will need to import your AwesomeParser before invoking read_campaign(), to make sure that this class will be reachable.

Remark

If you need to parse a campaign that uses an unsupported format, do not hesitate to submit an issue, with an example of what you want to parse. We will provide you some advices for writing your own parser.

We may also add a new feature to Scalpel by supporting this format, either by writing a parser or by integrating yours if you agree to contribute and submit a pull request.

Identifying Successful Experiments

When analyzing experimental results, it is often useful to identify which experiments are successful and which are not. By default, an experiment is considered as successful when it ended within the time limit. However, you may also want to perform additional checks to make sure that an experiment succeeded (for instance, by checking that the output of the experiment is correct).

To do so, you may add to your YAML configuration file an is-success filter that allows to make such checks, as in the following example:

source:
  is-success:
    - ${success}
    - ${valueA} == ${valueB} or {valueC} == 0
    - ${result} in ['CORRECT', 'CORRECT TOO']

Let us describe the syntax of the filter in the example above. First, is-success defines a list of conjunctively interpreted Boolean expressions. These expressions are themselves disjunctions of predicates.

Each predicate has to contain at least one variable, delimited using ${...}. Such a variable corresponds to the identifier of an experimental data read for a given experiment (for instance, the cpu_time of the experiment).

If the predicate contains only the variable (such as ${success}), then this variable is interpreted as a Boolean value. Otherwise, the predicate can use any comparison operator (among <, <=, ==, !=, >=, >) to compare the variable with either a literal value (which can be a Boolean value, an integer, a float number or a string) or another variable. A predicate can also check that a variable is either contained in a list of values (either literal values or variables) or contains a value (either a literal value or a variable) using the in operator. Lists are delimited using [...].

Remark

It is worth noting that Scalpel itself does not use is-success to filter data, in the sense that even failed experiments are included in the campaign it builds.

Instead, Scalpel passes this filter on to Wallet, so that the drawn figures only take into account successful experiments.

Description of the Data to Extract

In order to extract data from the files of your campaign, you need to provide a description of their content. In the following, we describe how to write such a description.

Extracting Data from Raw Files

If your experiment-ware produces raw output, and you want Scalpel to parse it, you can describe how to extract data from the corresponding files (which can be given using wildcards or relative paths) by providing regular expressions, as in the following example:

data:
  raw-data:
    - log-data: cpu_time
      file: "*.out"
      regex: 'overall runtime: (\d+.\d+) seconds'
      group: 1

In this case, when Scalpel reads a file with extension .out, it looks for a line that matches the specified regular expression, and extracts the cpu_time of the experiment from the group 1 (i.e. (\d+.\d+)) in this regular expression. In this case, the group could be omitted, as the value 1 is the default.

To make easier the description of raw data, Scalpel also recognizes so-called simplified patterns, as illustrated in the following example:

data:
  raw-data:
    - log-data: cpu_time
      file: "*.out"
      pattern: "overall runtime: {real} seconds"

Observe that, here, pattern is used in place of regex, and that the group (\d+.\d+) used in the previous example is replaced by {real}. This syntax allows to use one of the different symbols used to represent common data, and to avoid worrying about whitespaces (in a simplified pattern, any whitespace is interpreted as a sequence of whitespace characters).

Scalpel can interpret the following symbols.

{integer} for a (possibly signed) integer,
{real} for a real number,
{boolean} for a Boolean value (true or false, case-insensitive),
{word} for a word (i.e., a sequence of letters, digits and _), and
{any} for any sequence of characters (not greedy).

If the same line contains multiple relevant data, you can extract them by giving names to the groups you specified (in this case, the value for log-data may be omitted).

data:
  raw-data:
    - file: "*.out"
      pattern: "runtime: {real} seconds (cpu), {real} seconds (wallclock)"
      groups:
        cpu_time: 1
        wall_time: 2

Note that it is not possible to mix regular expressions and simplified patterns.

Extracting Data from File Names

Depending on your setting, you may need to extract relevant information from the name of the file to parse (for instance, the name of the experiment-ware or that of the input). This can be achieved through file-name-meta, as in the following example:

data:
  file-name-meta:
    pattern: "{any}_{any}.log"
    groups:
      experiment_ware: 1
      input: 2

As for log-data, you may choose to use either regular expressions (regex) or a simplified pattern. The fields in groups are used to name the groups identifying relevant data.

For instance, if the file my-xp-ware_my-input.log, the group 1 matches with my-xp-ware, which is thus identified as the experiment_ware, while the group 2 matches with my-input, which is thus identified as the input.

As file hierarchies are explored through the file system, the paths of the files that are encountered during this exploration are system-dependent (in particular, the file separator may vary from one system to another). Scalpel is able to dynamically adapt file separators used in the pattern specified as file-name-meta to ensure cross-platform compatibility for your configuration. To make sure that this compatibility is applied, you must always use / as file separator (even if it is not that of your system).

Extracting Data from Common Formats

If your output files use a common format (as JSON, CSV or XML), you do not need to use raw-data to extract their value. Instead, you just need to specify the name of such files as follows (wildcards and relative paths are supported).

data:
  data-files:
    - "*.json"
    - "output.xml"

Note that Scalpel will be able to extract data from such files by inferring automatically identifiers for the data it extracts. In the case of CSV files, the identifiers that will be used is inferred based on the header of the file.

For JSON and XML files, a “dotted” notation will be used. For example, consider the following JSON output:

{
  "experiment": {
    "runtime": 123.4,
    "value": [24, 27, 42, 51, 1664]
  }
}

Scalpel will automatically identify the runtime as experiment.runtime and the list of values as experiment.value. The same identifiers are inferred for the following XML output:

<experiment runtime="123.4">
  <value>24</value>
  <value>27</value>
  <value>42</value>
  <value>51</value>
  <value>1664</value>
</experiment>

By default, all keys stored in a JSON or XML file are extracted by Scalpel, and stored in the internal representation of the campaign. This may be memory consuming, in particular if there are some keys that you do not need. To discard such keys, you may specify them in the field ignored-data in your YAML configuration (this may actually be applied to any key defined by the campaign). For instance, the snippet below allows to discard the list experiment.value described in the two examples above.

data:
  ignored-data:
    - experiment.value

If needed, you can also configure the parser to use for reading data from data-files, as in the following example:

data:
  data-files:
    - name: "*.json"
      format: json
      name-as-prefix: true
    - name: "*.csv"
      format: csv
      has-header: false
      separator: " "
      name-as-prefix: true
    - name: "*.txt"
      parser: my.completely.specified.AwesomeParser

Observe in the example above that CSV files may be configured as for CSV campaigns (the same fields are used to describe the format of the file).

For each data-file, you can also set name-as-prefix to true, so that each field in the file will be prefixed by the name of the file, using a dotted notation. This is particularly useful whe the same key appears in different files.

Moreover, it is also possible to specify a custom parser, provided you give the completely specified name of this class. This parser must extend CampaignOutputParser, and its constructor must take as input a CampaignParserListener, a ScalpelConfiguration, the path of the file to parse and its name.

Finally, you may face some cases where the wildcards you use for declaring data files (or even log-data) are too generic. To ignore some files that still match these wildcards, you may specify the files to ignore with the field ignored-files (wildcards and relative paths are supported). For example, the snippet below allows to ignore some JSON files.

data:
  ignored-files:
    - "*ignore*.json"

Mapping Data to Scalpel’s Expectations

When parsing an experiment, Scalpel expects to find the required information to describe the result of this experiment. The identifier of such data is thus crucial to allow Scalpel to build consistent experiments. This is in particular true for the identifiers:

experiment_ware, which is the experiment-ware run in a given experiment,
input, which is the input on which the experiment-ware has been run, and
cpu_time, which is the runtime of the experiment.

If these identifiers are not specified in your campaign files (for instance, you have a CSV file in which the header does not use these names), you need to tell Scalpel how to map your experimental data to the expected identifiers. This can be achieved by specifying a mapping as in the following example:

data:
  mapping:
    experiment_ware:
      - program
      - options
    cpu_time: runtime
    input: file
    file: path

In this example, we have that, for each experiment, the data read as runtime will be interpreted as cpu_time and file as input.

Note that, for experiment_ware, two identifiers are specified. In this case, the data read as program and options will be concatenated (in this order) to build up the identifier of the experiment-ware. Moreover, if this experiment-ware does not exist yet, an object representation of this experiment-ware will be instantiated, using program and options has two additional fields.

Finally, observe that path is mapped to file, which is itself mapped to input. In this case, a recursive mapping is actually applied on path, which will be eventually interpreted as input while parsing the campaign. Recursive mapping is the recommended approach for mapping several identifiers to the same key.

Remark

This mapping is mainly designed to map custom identifiers to Metrics’ naming conventions. However, you can also use this mapping to rename other data (especially when their identifiers are automatically inferred by Scalpel), or to group together data that are separated in your campaign files.

Adding Default Values

Sometimes, it may happen that some data are missing in your experiment files, either because some experiment-wares did not output them correctly, or did not have enough time to output them within the time limit. This may be a problem if this data is required by Scalpel. For such data, you may provide default values as follows:

data:
  default-values:
    cpu_time: 1800

In this example, we have set the default cpu_time to the same value as the time limit (note that this is done by default by Scalpel). You may set default values for any key of the campaign, and even for “partial keys” (i.e., those that are part of a mapping).

Additional Information About the Campaign

When collecting data about a campaign, you may want to add relevant information that do not appear in the files produced during the execution of your experiments regarding its settings. This section presents how you can describe the experiment-wares and inputs you used for your experiments.

Description of the Experiment-Wares

Optionally, you may provide a description of the experiment-wares (i.e., the software programs you ran during your campaign). By default, experiment-wares are automagically instantiated when encountered during the parsing of your campaign files.

However, you may want to specify additional data w.r.t. the programs you experimented (for instance, the version of the software, the command line options passed to the program that was executed, etc.).

As such data may not appear in your campaign files, you can specify them in the YAML configuration as follows:

experiment-wares:
  - name: my-awesome-xpware
    version: 0.1.0
    command-line: ./my-awesome-xpware -o option
  - name: my-great-xpware
    commit-sha: abcd1234
    command-line: ./my-great-xpware -v value

When you specify information about experiment-wares, only their names are required. The name of an experiment-ware must uniquely identify this experiment-ware in the campaign, and must match the one that Scalpel will extract from your campaign files. For all other information you specify, you may use any key you want to identify this information.

Also, note that you are not required to use the same keys for all experiment-wares. You may also omit experiment-wares for which you do not need more information than those mentioned in the campaign files: these experiment-wares will simply be discovered when parsing the files.

Moreover, you may simply specify the list of the experiment-wares used in the campaign:

experiment-wares:
  - my-awesome-xpware
  - my-great-xpware

Doing so is rarely useful, as the name of the experiment-wares must necessarily be mentioned in the campaign files, and thus will be discovered during their parsing. However, this may be helpful to remind you that some experiments are missing, for instance, if you do not have run all experiment-wares yet.

Description of the Inputs

As for experiment-wares, you may want to add data about the inputs of your experiments. This is achieved by defining an input-set in your YAML configuration file, and giving it a proper name, as in the following example:

input-set:
  name: my-awesome-input-set
  type: list
  files:
    - path: path/to/instanceA.cnf
      family: F1
    - path: path/to/instanceB.cnf
      family: F2

In this example, files allows to list all the inputs you used in your experiments. As for experiment-wares, you may specify as many data as you want for your inputs. You may also use different keys for these data, and omit input files when you do not need to add more information than that provided in the campaign files. The only required key is path.

In the example above, observe that the type list is declared, to specify that all relevant information are specified in files.

Another possible type is file-list, if you only list the path of the files (in which case, you do not need to specify the path key). You may also use file if this list is written in a separate file (one path per line), in which case the files must give the list of the files to read.

Finally, you may also specify a hierarchy type, in which case Scalpel will explore a file hierarchy to find all input files from the file hierarchy rooted at the directory specified in files, as in the following example:

input-set:
  name: my-awesome-input-set
  type: hierarchy
  extensions: ".cnf"
  files: /path/to/my/benchmarks
  file-name-meta:
    pattern: /path/to/my/benchmarks/{any}/{any}.cnf
    groups:
      family: 1
      name: 2

Note that, in this example, the input files that are considered are those stored in the file hierarchy rooted at the directory /path/to/my/benchmarks, and having a .cnf extension (you may also specify a list of extensions if you have more than one).

Also, observe that a file-name-meta section is specified, with the same syntax as that described here. It allows extracting relevant information from the name of each input.

Both extensions and file-name-meta are taken into account for any type of input-sets.