_relative_error

Metric functions for relating to relative error.

Classes

RelativeError	Computes the relative error between two scalar values.
QuantileRelativeError	Computes the quantile of the empirical relative error.
MedianRelativeError	Computes the median relative error.
HighRelativeErrorFraction	Computes the fraction of groups with relative error above a threshold.

class RelativeError(output, column=None, *, name=None, description=None, baselines=None)

Bases: tmlt.analytics.metrics._base.ScalarMetric

Computes the relative error between two scalar values.

Note

This is only available on a paid version of Tumult Analytics. If you would like to hear more, please contact us at info@tmlt.io.

How it works:

1. The algorithm takes as input two single-row tables: one representing the differentially private (DP) output and the other representing the baseline output.

   DP Table (dp): This table contains the output data generated by a differentially private mechanism.

   Baseline Table (baseline): This table contains the output data generated by a non-private or baseline mechanism. It serves as a reference point for comparison with the DP output.

   The scalar values are retrieved from these single-row dataframes. Both values are expected to be numeric (either integers or floats). If not, the algorithm raises a ValueError.

2. The algorithm computes the relative error. Relative error is calculated as the absolute difference between the corresponding values in the DP and baseline outputs to the value in the baseline using the formula \(abs(dp - baseline) / baseline\). If baseline is zero, it returns infinity for non-zero differences (\(∞\)) and zero for zero differences (\(0\)).

Example

>>> dp_df = spark.createDataFrame(pd.DataFrame({"A": [5]}))
>>> dp_outputs = {"O": dp_df}
>>> baseline_df = spark.createDataFrame(pd.DataFrame({"A": [5]}))
>>> baseline_outputs = {"O": baseline_df}

>>> metric = RelativeError(output="O")
>>> result = metric.compute_for_baseline(dp_outputs, baseline_outputs)
>>> result
0.0
>>> metric.format(result)
'0.0'

Parameters

• output (str) –

• column (Optional[str]) –

• name (Optional[str]) –

• description (Optional[str]) –

• baselines (Optional[List[str]]) –

__init__(output, column=None, *, name=None, description=None, baselines=None)

Constructor.

Parameters

• output (strstr) – The output to compute the metric for.

• column (str | NoneOptional[str] (default: None)) – The column to compute the relative error over. If the given output has only one column, this argument may be omitted.

• name (str | NoneOptional[str] (default: None)) – A name for the metric.

• description (str | NoneOptional[str] (default: None)) – A description of the metric.

• baselines (List[str] | NoneOptional[List[str]] (default: None)) – The name of the baseline program(s) used for the error report. If None, use all baselines specified as custom baseline and baseline options on tuner class. If no baselines are specified on tuner class, use default baseline. If a string, use only that baseline. If a list, use only those baselines.

format(value)

Returns a string representation of this object.

compute_on_scalar(dp_value, baseline_value)

Computes metric value from DP and baseline values.

property output

Returns the name of the run output or view name.

Return type

str

property column

Returns the name of the value column, if it is set.

Return type

Optional[str]

compute_for_baseline(dp_outputs, baseline_outputs, unprotected_inputs=None, program_parameters=None)

Returns the metric value given the DP outputs and the baseline outputs.

Parameters

• dp_outputs (Dict[str, pyspark.sql.DataFrame]) –

• baseline_outputs (Dict[str, pyspark.sql.DataFrame]) –

• unprotected_inputs (Optional[Dict[str, pyspark.sql.DataFrame]]) –

• program_parameters (Optional[Dict[str, Any]]) –

Return type

Any

property name

Returns the name of the metric.

Return type

str

property description

Returns the description of the metric.

Return type

str

property baselines

Returns the baselines used for the metric.

Return type

Optional[Union[str, List[str]]]

__call__(dp_outputs, baseline_outputs, unprotected_inputs=None, program_parameters=None)

Computes the given metric on the given DP and baseline outputs.

Parameters

• dp_outputs (Dict[str, pyspark.sql.DataFrame]) – The differentially private outputs of the program.

• baseline_outputs (Dict[str, Dict[str, pyspark.sql.DataFrame]]) – The outputs of the baseline programs.

• unprotected_inputs (Optional[Dict[str, pyspark.sql.DataFrame]]) – Optional public dataframes used in error computation.

• program_parameters (Optional[Dict[str, Any]]) – Optional program specific parameters used in error computation.

Return type

List[tmlt.analytics.metrics.MetricOutput]

class QuantileRelativeError(output, quantile, measure_column, join_columns, *, name=None, description=None, baselines=None)

Bases: tmlt.analytics.metrics._base.JoinedOutputMetric

Computes the quantile of the empirical relative error.

Note

This is only available on a paid version of Tumult Analytics. If you would like to hear more, please contact us at info@tmlt.io.

How it works:

1. The algorithm takes as input two tables: one representing the differentially private (DP) output and the other representing the baseline output.

   DP Table (dp): This table contains the output data generated by a differentially private mechanism.

   Baseline Table (baseline): This table contains the output data generated by a non-private or baseline mechanism. It serves as a reference point for comparison with the DP output.

   The algorithm includes error handling to ensure the validity of the input data. It checks for the existence and numeric type of the measure_column.

   The algorithm performs an inner join between the DP and baseline tables based on join_columns to produce the combined dataframe. This join must be one-to-one, with each row in the DP table matches exactly one row in the baseline table, and vice versa. This ensures that there is a direct correspondence between the DP and baseline outputs for each entity, allowing for accurate comparison.

2. After performing the join, the algorithm computes the relative error for each group. Relative error is calculated as the absolute difference between the corresponding values in the DP and baseline outputs to the value in the baseline using the formula \(abs(dp - baseline) / baseline\). If baseline is zero, it returns infinity for non-zero differences (\(∞\)) and zero for zero differences (\(0\)).

3. The algorithm then calculates the n-th quantile of the relative error across all groups.

   The algorithm handles cases where the quantile computation may result in an empty column, returning a NaN (not a number) value in such scenarios.

   Note

   • Provided algorithm assumes a one-to-one join scenario.

   • Nulls in the measure columns are dropped because the metric cannot handle null values, and the absolute error computation requires valid numeric values in both columns.

Example

>>> dp_df = spark.createDataFrame(
...     pd.DataFrame(
...         {
...             "A": ["a1", "a2", "a3"],
...             "X": [50, 110, 100]
...         }
...     )
... )
>>> dp_outputs = {"O": dp_df}
>>> baseline_df = spark.createDataFrame(
...     pd.DataFrame(
...         {
...             "A": ["a1", "a2", "a3"],
...             "X": [100, 100, 100]
...         }
...     )
... )
>>> baseline_outputs = {"O": baseline_df}

>>> metric = QuantileRelativeError(
...     output="O",
...     quantile=0.5,
...     measure_column="X",
...     join_columns=["A"]
... )
>>> metric.quantile
0.5
>>> metric.join_columns
['A']
>>> result = metric.compute_for_baseline(dp_outputs, baseline_outputs)
>>> result
0.1
>>> metric.format(result)
'0.10'

Methods

quantile()	Returns the quantile.
measure_column()	Returns name of the column to compute the quantile of relative error over.
format()	Returns a string representation of this object.
compute_on_joined_output()	Computes quantile relative error value from combined dataframe.
output()	Returns the name of the run output or view name.
join_columns()	Returns the name of the join columns.
check_join_key_uniqueness()	Check if the join keys uniquely identify rows in the joined DataFrame.
compute_for_baseline()	Computes metric value.
name()	Returns the name of the metric.
description()	Returns the description of the metric.
baselines()	Returns the baselines used for the metric.
__call__()	Computes the given metric on the given DP and baseline outputs.

Parameters

• output (str) –

• quantile (float) –

• measure_column (str) –

• join_columns (List[str]) –

• name (Optional[str]) –

• description (Optional[str]) –

• baselines (Optional[List[str]]) –

__init__(output, quantile, measure_column, join_columns, *, name=None, description=None, baselines=None)

Constructor.

Parameters

• output (strstr) – The output to compute the metric for.

• quantile (floatfloat) – The quantile to calculate (between 0 and 1).

• measure_column (strstr) – The column to compute the quantile of relative error over.

• join_columns (List[str]List[str]) – Columns to join on.

• name (str | NoneOptional[str] (default: None)) – A name for the metric.

• description (str | NoneOptional[str] (default: None)) – A description of the metric.

• baselines (List[str] | NoneOptional[List[str]] (default: None)) – The name of the baseline program(s) used for the error report. If None, use all baselines specified as custom baseline and baseline options on tuner class. If no baselines are specified on tuner class, use default baseline. If a string, use only that baseline. If a list, use only those baselines.

property quantile

Returns the quantile.

Return type

float

property measure_column

Returns name of the column to compute the quantile of relative error over.

Return type

str

format(value)

Returns a string representation of this object.

compute_on_joined_output(joined_output)

Computes quantile relative error value from combined dataframe.

Parameters

joined_output (pyspark.sql.DataFrame) –

property output

Returns the name of the run output or view name.

Return type

str

property join_columns

Returns the name of the join columns.

Return type

List[str]

check_join_key_uniqueness(joined_output)

Check if the join keys uniquely identify rows in the joined DataFrame.

Parameters

joined_output (pyspark.sql.DataFrame) –

compute_for_baseline(dp_outputs, baseline_outputs, unprotected_inputs=None, program_parameters=None)

Computes metric value.

Parameters

• dp_outputs (Dict[str, pyspark.sql.DataFrame]) –

• baseline_outputs (Dict[str, pyspark.sql.DataFrame]) –

• unprotected_inputs (Optional[Dict[str, pyspark.sql.DataFrame]]) –

• program_parameters (Optional[Dict[str, Any]]) –

property name

Returns the name of the metric.

Return type

str

property description

Returns the description of the metric.

Return type

str

property baselines

Returns the baselines used for the metric.

Return type

Optional[Union[str, List[str]]]

__call__(dp_outputs, baseline_outputs, unprotected_inputs=None, program_parameters=None)

Computes the given metric on the given DP and baseline outputs.

Parameters

• dp_outputs (Dict[str, pyspark.sql.DataFrame]) – The differentially private outputs of the program.

• baseline_outputs (Dict[str, Dict[str, pyspark.sql.DataFrame]]) – The outputs of the baseline programs.

• unprotected_inputs (Optional[Dict[str, pyspark.sql.DataFrame]]) – Optional public dataframes used in error computation.

• program_parameters (Optional[Dict[str, Any]]) – Optional program specific parameters used in error computation.

Return type

List[tmlt.analytics.metrics.MetricOutput]

class MedianRelativeError(output, measure_column, join_columns, *, name=None, description=None, baselines=None)

Bases: QuantileRelativeError

Computes the median relative error.

Equivalent to QuantileRelativeError with quantile = 0.5.

Note

This is only available on a paid version of Tumult Analytics. If you would like to hear more, please contact us at info@tmlt.io.

Example

>>> dp_df = spark.createDataFrame(
...     pd.DataFrame(
...         {
...             "A": ["a1", "a2", "a3"],
...             "X": [50, 110, 100]
...         }
...     )
... )
>>> dp_outputs = {"O": dp_df}
>>> baseline_df = spark.createDataFrame(
...     pd.DataFrame(
...         {
...             "A": ["a1", "a2", "a3"],
...             "X": [100, 100, 100]
...         }
...     )
... )
>>> baseline_outputs = {"O": baseline_df}

>>> metric = MedianRelativeError(
...     output="O",
...     measure_column="X",
...     join_columns=["A"]
... )
>>> metric.quantile
0.5
>>> metric.join_columns
['A']
>>> result = metric.compute_for_baseline(dp_outputs, baseline_outputs)
>>> result
0.1
>>> metric.format(result)
'0.10'

Methods

quantile()	Returns the quantile.
measure_column()	Returns name of the column to compute the quantile of relative error over.
format()	Returns a string representation of this object.
compute_on_joined_output()	Computes quantile relative error value from combined dataframe.
output()	Returns the name of the run output or view name.
join_columns()	Returns the name of the join columns.
check_join_key_uniqueness()	Check if the join keys uniquely identify rows in the joined DataFrame.
compute_for_baseline()	Computes metric value.
name()	Returns the name of the metric.
description()	Returns the description of the metric.
baselines()	Returns the baselines used for the metric.
__call__()	Computes the given metric on the given DP and baseline outputs.

Parameters

• output (str) –

• measure_column (str) –

• join_columns (List[str]) –

• name (Optional[str]) –

• description (Optional[str]) –

• baselines (Optional[List[str]]) –

__init__(output, measure_column, join_columns, *, name=None, description=None, baselines=None)

Constructor.

Parameters

• output (strstr) – The output to compute the metric for.

• measure_column (strstr) – The column to compute the median of relative error over.

• join_columns (List[str]List[str]) – Columns to join on.

• name (str | NoneOptional[str] (default: None)) – A name for the metric.

• description (str | NoneOptional[str] (default: None)) – A description of the metric.

• baselines (List[str] | NoneOptional[List[str]] (default: None)) – The name of the baseline program(s) used for the error report. If None, use all baselines specified as custom baseline and baseline options on tuner class. If no baselines are specified on tuner class, use default baseline. If a string, use only that baseline. If a list, use only those baselines.

property quantile

Returns the quantile.

Return type

float

property measure_column

Returns name of the column to compute the quantile of relative error over.

Return type

str

format(value)

Returns a string representation of this object.

compute_on_joined_output(joined_output)

Computes quantile relative error value from combined dataframe.

Parameters

joined_output (pyspark.sql.DataFrame) –

property output

Returns the name of the run output or view name.

Return type

str

property join_columns

Returns the name of the join columns.

Return type

List[str]

check_join_key_uniqueness(joined_output)

Check if the join keys uniquely identify rows in the joined DataFrame.

Parameters

joined_output (pyspark.sql.DataFrame) –

compute_for_baseline(dp_outputs, baseline_outputs, unprotected_inputs=None, program_parameters=None)

Computes metric value.

Parameters

• dp_outputs (Dict[str, pyspark.sql.DataFrame]) –

• baseline_outputs (Dict[str, pyspark.sql.DataFrame]) –

• unprotected_inputs (Optional[Dict[str, pyspark.sql.DataFrame]]) –

• program_parameters (Optional[Dict[str, Any]]) –

property name

Returns the name of the metric.

Return type

str

property description

Returns the description of the metric.

Return type

str

property baselines

Returns the baselines used for the metric.

Return type

Optional[Union[str, List[str]]]

__call__(dp_outputs, baseline_outputs, unprotected_inputs=None, program_parameters=None)

Computes the given metric on the given DP and baseline outputs.

Parameters

• dp_outputs (Dict[str, pyspark.sql.DataFrame]) – The differentially private outputs of the program.

• baseline_outputs (Dict[str, Dict[str, pyspark.sql.DataFrame]]) – The outputs of the baseline programs.

• unprotected_inputs (Optional[Dict[str, pyspark.sql.DataFrame]]) – Optional public dataframes used in error computation.

• program_parameters (Optional[Dict[str, Any]]) – Optional program specific parameters used in error computation.

Return type

List[tmlt.analytics.metrics.MetricOutput]

class HighRelativeErrorFraction(output, relative_error_threshold, measure_column, join_columns, *, name=None, description=None, baselines=None)

Bases: tmlt.analytics.metrics._base.JoinedOutputMetric

Computes the fraction of groups with relative error above a threshold.

Note

This is only available on a paid version of Tumult Analytics. If you would like to hear more, please contact us at info@tmlt.io.

How it works:

1. The algorithm takes as input two tables: one representing the differentially private (DP) output and the other representing the baseline output.

   DP Table (dp): This table contains the output data generated by a differentially private mechanism.

   Baseline Table (baseline): This table contains the output data generated by a non-private or baseline mechanism. It serves as a reference point for comparison with the DP output.

   The algorithm includes error handling to ensure the validity of the input data. It checks for the existence and numeric type of the measure_column.

   The algorithm performs an inner join between the DP and baseline tables based on join_columns to produce the combined dataframe. This join must be one-to-one, with each row in the DP table matches exactly one row in the baseline table, and vice versa. This ensures that there is a direct correspondence between the DP and baseline outputs for each entity, allowing for accurate comparison.

2. After performing the join, the algorithm computes the relative error for each group. Relative error is calculated as the absolute difference between the corresponding values in the DP and baseline outputs to the value in the baseline using the formula \(abs(dp - baseline) / baseline\). If baseline is zero, it returns infinity for non-zero differences (\(∞\)) and zero for zero differences (\(0\)).

3. Next, the algorithm filters the relative error dataframe to include only those data points where the relative error exceeds a specified threshold (relative_error_threshold). This threshold represents the maximum allowable relative error for a data point to be considered within acceptable bounds.

4. Finally, the algorithm then calculates the high relative error fraction by dividing the count of data points with relative errors exceeding the threshold by the total count of data points in the dataframe.

   The algorithm handles cases where the resulting dataframe after relative error computation is empty (i.e., it contains no data points), returning a NaN (not a number) value in such scenarios.

   Note

   • Provided algorithm assumes a one-to-one join scenario.

   • Nulls in the measure columns are dropped because the metric cannot handle null values, and the absolute error computation requires valid numeric values in both columns.

Example

>>> dp_df = spark.createDataFrame(
...     pd.DataFrame(
...         {
...             "A": ["a1", "a2", "a3"],
...             "X": [50, 110, 100]
...         }
...     )
... )
>>> dp_outputs = {"O": dp_df}
>>> baseline_df = spark.createDataFrame(
...     pd.DataFrame(
...         {
...             "A": ["a1", "a2", "a3"],
...             "X": [100, 100, 100]
...         }
...     )
... )
>>> baseline_outputs = {"O": baseline_df}

>>> metric = HighRelativeErrorFraction(
...     output="O",
...     measure_column="X",
...     relative_error_threshold=0.25,
...     join_columns=["A"]
... )
>>> metric.relative_error_threshold
0.25
>>> metric.join_columns
['A']
>>> result = metric.compute_for_baseline(dp_outputs, baseline_outputs)
>>> result
0.3333333333333333
>>> metric.format(result)
'0.33'

Methods

relative_error_threshold()	Returns the relative error threshold.
measure_column()	Returns name of the column to compute the relative error over.
format()	Returns a string representation of this object.
compute_on_joined_output()	Computes high relative error fraction from combined dataframe.
output()	Returns the name of the run output or view name.
join_columns()	Returns the name of the join columns.
check_join_key_uniqueness()	Check if the join keys uniquely identify rows in the joined DataFrame.
compute_for_baseline()	Computes metric value.
name()	Returns the name of the metric.
description()	Returns the description of the metric.
baselines()	Returns the baselines used for the metric.
__call__()	Computes the given metric on the given DP and baseline outputs.

Parameters

• output (str) –

• relative_error_threshold (float) –

• measure_column (str) –

• join_columns (List[str]) –

• name (Optional[str]) –

• description (Optional[str]) –

• baselines (Optional[str]) –

__init__(output, relative_error_threshold, measure_column, join_columns, *, name=None, description=None, baselines=None)

Constructor.

Parameters

• output (strstr) – The output to compute the metric for.

• relative_error_threshold (floatfloat) – The threshold for the relative error.

• measure_column (strstr) – The column to compute relative error over.

• join_columns (List[str]List[str]) – Columns to join on.

• name (str | NoneOptional[str] (default: None)) – A name for the metric.

• description (str | NoneOptional[str] (default: None)) – A description of the metric.

• baselines (str | NoneOptional[str] (default: None)) – The name of the baseline program(s) used for the error report. If None, use all baselines specified as custom baseline and baseline options on tuner class. If no baselines are specified on tuner class, use default baseline. If a string, use only that baseline. If a list, use only those baselines.

property relative_error_threshold

Returns the relative error threshold.

Return type

float

property measure_column

Returns name of the column to compute the relative error over.

Return type

str

format(value)

Returns a string representation of this object.

compute_on_joined_output(joined_output)

Computes high relative error fraction from combined dataframe.

Parameters

joined_output (pyspark.sql.DataFrame) –

property output

Returns the name of the run output or view name.

Return type

str

property join_columns

Returns the name of the join columns.

Return type

List[str]

check_join_key_uniqueness(joined_output)

Check if the join keys uniquely identify rows in the joined DataFrame.

Parameters

joined_output (pyspark.sql.DataFrame) –

compute_for_baseline(dp_outputs, baseline_outputs, unprotected_inputs=None, program_parameters=None)

Computes metric value.

Parameters

• dp_outputs (Dict[str, pyspark.sql.DataFrame]) –

• baseline_outputs (Dict[str, pyspark.sql.DataFrame]) –

• unprotected_inputs (Optional[Dict[str, pyspark.sql.DataFrame]]) –

• program_parameters (Optional[Dict[str, Any]]) –

property name

Returns the name of the metric.

Return type

str

property description

Returns the description of the metric.

Return type

str

property baselines

Returns the baselines used for the metric.

Return type

Optional[Union[str, List[str]]]

__call__(dp_outputs, baseline_outputs, unprotected_inputs=None, program_parameters=None)

Computes the given metric on the given DP and baseline outputs.

Parameters

• dp_outputs (Dict[str, pyspark.sql.DataFrame]) – The differentially private outputs of the program.

• baseline_outputs (Dict[str, Dict[str, pyspark.sql.DataFrame]]) – The outputs of the baseline programs.

• unprotected_inputs (Optional[Dict[str, pyspark.sql.DataFrame]]) – Optional public dataframes used in error computation.

• program_parameters (Optional[Dict[str, Any]]) – Optional program specific parameters used in error computation.

Return type

List[tmlt.analytics.metrics.MetricOutput]