Skip to main content

CSV Reports

Since v3.77.0

Gwen will generate CSV reports for you when you include results in the format option.

By default, the following CSV files will be generated in the reports/results subdirectory of your configured output directory, but you can completely reconfigure this and create your own CSVs to suit your requirements:

  • Feature level results:
    • feature-results-PASSED.csv - A log of all passed feature results
    • feature-results-FAILED.csv - A log of all failed feature results including the failed reason
    • feature-results-ALL.csv - A log of all feature results (passed or failed) including the failed reason (for failed)
  • Scenario level results:
    • scenario-results-PASSED.csv - A log of all passed scenario results
    • scenario-results-FAILED.csv - A log of all failed scenario results including the failed reason
    • scenario-results-ALL.csv - A log of all scenario results (passed or failed) including the failed reason (for failed)

Configuration​

Results files​

CSV files can be configured within the gwen.reports.results.files element in your settings file.

<fileId> {
  file = "<filename>" 
  scope = "[scope]"
  status = "[status]"
  fields = [
    { name = "<fieldName 1>", ref = "[reference 1]", optional = [optionality 1], excludes = "[excludeFields 1]"}
    { name = "<fieldName 2>", ref = "[reference 2]", optional = [optionality 2], excludes = "[excludeFields 2]" }
    ..
    { name = "<fieldName N>", ref = "[reference N]", optional = [optionality N], excludes = "[excludeFields N]" }
  ]
}

Where:

  • <fileId> - (Mandatory) Is an arbitatry Id for the file element

  • <filename> - (Mandatory) - Is the name or path to the target CSV file

    • If a file name only is specified (without a prefixed path), then the named file will be written to the reports/results subdirectory within your configured output directory
    • If the file name is prefixed with a path, then the named file will be written to that location

  • [scope] - (Optional) is either:

    • Feature - To log the results of all features to the CSV file as they complete.

      • or Feature: <name> - To log the results of a specifically named Feature only, where <name> is the name of the Feature to log the results for. Alternatively, omit the scope attribute and place the @Results annotation directly on the Feature itself in the feature or meta file to make it more explict and avoid potential name clashes (preferred approach).

    • Rule - To log the results of all rules to the CSV file as they complete.

      • or Rule: <name> - To log the results of a specifically named Rule only, where <name> is the name of the Rule to log the results for. Alternatively, omit the scope attribute and place the @Results annotation directly on the Rule itself in the feature or meta file to make it more explict and avoid potential name clashes (preferred approach).

    • Scenario - To log the results of all scenarios to the CSV file as they complete.

      • or Scenario: <name> - To log the results of a specifically named Scenario only, where <name> is the name of the Scenario to log the results for. Alternatively, omit the scope attribute and place the @Results annotation directly on the Scenario itself in the feature or meta file to make it more explict and avoid potential name clashes (preferred approach).

    • Examples - To log the results of all examples to the CSV file as they complete.

      • or Examples: <name> - To log the results of a specifically named Examples only, where <name> is the name of the Examples to log the results for. Alternatively, omit the scope attribute and place the @Results annotation directly on the Examples itself in the feature or meta file to make it more explict and avoid potential name clashes (preferred approach).

    • StepDef - To log the results of all StepDefs to the CSV file as they complete.

      • or StepDef: <name> - To log the results of a specifically named StepDef only, where <name> is the name of the StepDef to log the results for. Alternatively, omit the scope attribute and place the @Results annotation directly on the StepDef itself in the feature or meta file to make it more explict and avoid potential name clashes (preferred approach).

    • Omit the scope to not log any results automatically; for when you want to explicitly control the scope and where the logging will occur using a @Results annotation

  • [status] - (Optional) is either:

    • Passed - To log only passed results to the CSV file
    • Failed - To log only failed results to the CSV file
    • Or blank to log passed or failed results to the CSV file
  • fields - (Mandatory) - Contains an array of fields as described below

Results fields​

CSV fields are configured as arrays witihin the fields attribute and have the following structure:

{ name = "<fieldName>", ref = "[reference]", optional = [optionality], excludes = "[excludeFields]" }

Where:

  • <fieldName> - (Mandatory) - Is the name of the field as it will appear in generated CSV

  • [reference] - (Optional) - Can be one of:

    • The name of a varaible in scope containing the value to bind
    • If blank, will use the value bound to the variable of same name as <fieldName>

    • If <fieldName> is set to *, then:

      • This reference must specify a path to the CSV template file containing the names of all fields to bind in the first row (and the names of such fields will also be used as the binding references)
      • Can also be set to the special reference $<gwen.options.dataFile> which will resolve to the input CSV (data feed) file passed to Gwen via the -i|--input-data <file> CLI option. In the case all the fields present in the input file will be included and resolved by name.

  • [optionality] - (Optional) - Can be either of:

    • true - (Default) - To make this field mandatory. Will result in an undefined binding error being raised at runtime if the field binding does does not exist (i.e: is not in scope) at the time the record is written.
    • false - To make this field optional. Will resolve to blank at runtime if the field binding does does not exist (i.e: is not in scope) at the time the record is written.

  • [excludeFields] - (Optional) - Comma separated list of fields to exclude when <fieldName> is set to *:

    • All fields in the template file (referred to by [reference]) that match those in the excludes list will be excluded

Default configuration​

In this configuration, fields are uniquely defined upfront and reused across file definitions.

gwen {
  report {
    results {
      fields {
        feature {
          status = [
            { field = "EVAL_STATUS", ref = "gwen.feature.eval.status.keyword.upperCased" }
            { field = "EVAL_STARTED", ref = "gwen.feature.eval.started" }
            { field = "EVAL_FINISHED", ref = "gwen.feature.eval.finished" }
          ]
          details = [
            { field = "FEATURE_FILE", ref = "gwen.feature.file.path" }
            { field = "FEATURE_NAME", ref = "gwen.feature.displayName" }
          ]
          duration = [
            { field = "EVAL_DURATION", ref = "gwen.feature.eval.duration" }
          ]
          message = [
            { field = "EVAL_MESSAGE", ref = "gwen.feature.eval.status.message" }
          ]
        }
        scenario {
          status = [
            { field = "EVAL_STATUS", ref = "gwen.scenario.eval.status.keyword.upperCased" }
            { field = "EVAL_STARTED", ref = "gwen.scenario.eval.started" }
            { field = "EVAL_FINISHED", ref = "gwen.scenario.eval.finished" }
          ]
          details = [
            ${gwen.report.results.fields.feature.details}
            { field = "SCENARIO_NAME", ref = "gwen.scenario.displayName" }
          ]
          duration = [
            { field = "EVAL_DURATION", ref = "gwen.scenario.eval.duration" }
          ]
          message = [
            { field = "EVAL_MESSAGE", ref = "gwen.scenario.eval.status.message" }
          ]
        }
        input {
            data = [
            { field = "*", ref = "$<gwen.options.dataFile>" }
          ]
        }
      }
      files {
        feature {
          passed {
            file = "feature-results-PASSED.csv"
            scope = "Feature"
            status = "Passed"
            fields = [ 
              ${gwen.report.results.fields.feature.status}
              ${gwen.report.results.fields.feature.details}
              ${gwen.report.results.fields.feature.duration}
              ${gwen.report.results.fields.input.data}
            ]
          }
          failed {
            file = "feature-results-FAILED.csv"
            scope = "Feature"
            status = "Failed"
            fields = [ 
              ${gwen.report.results.fields.feature.status}
              ${gwen.report.results.fields.feature.details}
              ${gwen.report.results.fields.feature.duration}
              ${gwen.report.results.fields.input.data}
              ${gwen.report.results.fields.feature.message}
            ]
          }
          all {
            file = "feature-results-ALL.csv"
            scope = "Feature"
            fields = [ 
              ${gwen.report.results.fields.feature.status}
              ${gwen.report.results.fields.feature.details}
              ${gwen.report.results.fields.feature.duration}
              ${gwen.report.results.fields.input.data}
              ${gwen.report.results.fields.feature.message}
            ]
          }
        }
        scenario {
          passed {
            file = "scenario-results-PASSED.csv"
            scope = "Scenario"
            status = "Passed"
            fields = [
              ${gwen.report.results.fields.scenario.status}
              ${gwen.report.results.fields.scenario.details}
              ${gwen.report.results.fields.scenario.duration}
              ${gwen.report.results.fields.input.data}
            ]
          }
          failed {
            file = "scenario-results-FAILED.csv"
            scope = "Scenario"
            status = "Failed"
            fields = [
              ${gwen.report.results.fields.scenario.status}
              ${gwen.report.results.fields.scenario.details}
              ${gwen.report.results.fields.scenario.duration}
              ${gwen.report.results.fields.input.data}
              ${gwen.report.results.fields.scenario.message}
            ]
          }
          all {
            file = "scenario-results-ALL.csv"
            scope = "Scenario"
            fields = [
              ${gwen.report.results.fields.scenario.status}
              ${gwen.report.results.fields.scenario.details}
              ${gwen.report.results.fields.scenario.duration}
              ${gwen.report.results.fields.input.data}
              ${gwen.report.results.fields.scenario.message}
            ]
          }
        }
      }
    }
  }
}

Results Annotation​

Used to log results to a file at a specific Feature, Rule, Scenario, StepDef, Scenaro Outline, or Examples node level.

Syntax:​

@Results('<fileId>')

Where:

  • <fileId> - Is a configured file Id under the gwen.reports.results.files settings element.

Example:​

The configuration for my.csv might be defiend as follows in your settings file. Note that the scope attribute is deliberately absent here, since we will using the Results annotation to specify where the scope will be.

gwen.conf 
gwen {
  ..
  reports {
    results {
      files {
        my {
          csv {
            file = "file.csv"
            fields = [
              { field = "status",   ref = "gwen.scenario.eval.status.keyword.upperCased" }
              { field = "name",     ref = "my name" }
              { field = "job",      ref = "my job" }
              { field = "duration", ref = "gwen.scenario.eval.duration" }
            ]
          }
        }
      }
    }
  }
  ..
}

Specifying the @Results('my.csv') annoation on a scenario, would then log to results to the configured file when the scenario completes. The scope of the my.csv file would then be bound to this Scenario.

my.feature 
Feature: My feature

  @Results('my.csv')
  Scenario: My scenario
    Given my name is "Gwen"
      And my job is "Automation"
      ..

And the generated results file would then contain:

file.csv 
status,name,job,duration
PASSED,Gwen,Automation,1ms

When used on a Scenario Outline, each Examples scenario would be logged.

my.feature 
Feature: My feature

  @Results('my.csv')
  Scenario Outline: My scenario
    Given my name is <name>
      And my job is <job>
    Examples:
      | name    | job        |
      | Gwen    | Automation |
      | Stacey  | Agent      |
      ..

The generated results file would then contain:

file.csv 
status,name,job,duration
PASSED,Gwen,Automation,1ms
PASSED,Gwen,Automation,2ms
..

The @Results annotation can also be used on Scenario Outlies in conjunction with the @Examples annotation for the same effect.

my.meta 
Feature: My meta

  @Examples('data-records.csv`)
  @Results('my.csv')
  Scenario Outline: My scenario
    Given my name is <name>
      And my job is <job>
      ..

Logging Results​

With the above settings in place, CSV files containing evaluated results will automatiacally be logged when you launch Gwen to execute with the -f results format option.

Example

Executing features in the gwen/features/todo folder and log CSV reports.

To generate CSV files only

yarn gwen -b -f results gwen/features/todo

To generate HTML reports and CSV files

yarn  -b f html,results gwen/features/todo

Gwen CLI

Output
  • The CSV results logged to:
    • output/reports/results/
  • The HTML report will be generated at
    • output/reports/index.html.