metric#

from tmlt.tune import metric
tmlt.tune.metric(name, description=None, grouping_columns=None, measure_column=None, empty_value=None)#

Decorator to define a generic Metric.

This is the most generic custom metric decorator. It is typically easier to use either the joined_output_metric() or the single_output_metric() decorator instead, whenever feasible.

The decorated function must have the following parameters:

  • dp_outputs: a dictionary of DataFrames containing the program’s outputs.

  • baseline_outputs: a dictionary mapping baseline names to dictionaries of output DataFrames.

It may also have the following optional parameters:

  • result_column_name: if the function returns a DataFrame, the metric results should be in a column with this name

  • unprotected_inputs: A dictionary containing the program’s unprotected inputs.

  • parameters: A dictionary containing the program’s parameters.

If the metric does not have grouping columns, the function must return a numeric value, a boolean, or a string. If the metric has grouping columns, then it must return a DataFrame. This DataFrame should contain the grouping columns, and exactly one additional column containing the metric value for each group. This column’s type should be numeric, boolean, or string.

To use the built-in metrics in addition to this custom metric, you can separately specify metrics class variable.

Parameters:

  • name (str) – A name for the metric.

  • description (Optional[str]) – A description of the metric.

  • grouping_columns (Optional[List[str]]) – If specified, the metric should group the outputs by the given columns, and calculate the metric for each group.

  • measure_column (Optional[str]) – If specified, the column in the outputs to measure.

  • empty_value (Optional[Any]) – If all dp and baseline outputs are empty, the metric will return this value.

>>> from tmlt.analytics import Session
>>> from tmlt.tune import MedianAbsoluteError
>>> from pyspark.sql import DataFrame
>>> from typing import Dict

>>> class Program(SessionProgram):
...     class ProtectedInputs:
...         protected_df: DataFrame
...     class UnprotectedInputs:
...         unprotected_df: DataFrame
...     class Outputs:
...         output_df: DataFrame
...     def session_interaction(self, session: Session):
...         return {"output_df": dp_output}
>>> class Tuner(SessionProgramTuner, program=Program):
...     @metric(name="custom_metric")
...     @staticmethod
...     def custom_metric(
...         dp_outputs: Dict[str, DataFrame],
...         baseline_outputs: Dict[str, Dict[str, DataFrame]]
...     ):
...         # If the program has unprotected inputs and/or parameters, the custom
...         #  metric method can take them as an argument.
...         ...
...     metrics = [
...         MedianAbsoluteError(
...             output="output_df",
...             join_columns=["join_column"],
...             measure_column="Y"
...         ),
...     ]  # You can mix custom and built-in metrics.
On this page