tuner#

Interface for tuning SessionPrograms.

Warning

SessionProgramTuner is intended to be used for tuning DP programs. It does not provide any privacy guarantees. It is recommended to use synthetic or historical data for tuning instead of the data that will be used in production.

The SessionProgramTuner class is an abstract base class that defines the interface for tuning SessionPrograms. To tune a specific program, users should subclass SessionProgramTuner, passing their SessionProgram as the program class argument.

>>> class Program(SessionProgram):
...     class ProtectedInputs:
...         protected_df: DataFrame
...     class Outputs:
...         b_sum: DataFrame
...     class Parameters:
...         low: int
...         high: int
...     def session_interaction(self, session: Session):
...         low = self.parameters["low"]
...         high = self.parameters["high"]
...         a_values = KeySet.from_dict({"a": ["x", "y"]})
...         sum_query = QueryBuilder("protected_df").groupby(a_values).sum("b", low, high)
...         b_sum = session.evaluate(sum_query, self.privacy_budget)
...         return {"b_sum": b_sum}
>>> class Tuner(SessionProgramTuner, program=Program):
...     baseline_options = {
...         "use_clamping_bounds": NoPrivacySession.Options(
...             enforce_clamping_bounds=True
...         ),
...         "ignore_clamping_bounds": NoPrivacySession.Options(
...             enforce_clamping_bounds=False
...         ),
...     }
...
...     @baseline("custom_baseline")
...     def no_clamping_bounds_baseline(protected_inputs: Dict[str, DataFrame]) -> Dict[str, DataFrame]:
...         df = protected_inputs["protected_df"]
...         sum_value = df.groupBy("a").agg(sf.sum("b").alias('b_sum'))
...         return {"b_sum": sum_value}
...
...     @metric(name="root_mean_squared_error", output="b_sum", join_columns=["a"])
...     def compute_rmse(
...         grouped_output: DataFrame, output_column_name: str
...     ):
...         err = sf.col("b_sum_dp") - sf.col("b_sum_baseline")
...         rmse = grouped_output.agg(sf.sqrt(sf.sum(sf.pow(err, sf.lit(2)))).alias("rmse"))
...         return rmse.collect()[0]["rmse"]
...
...     metrics = [
...         MedianRelativeError(
...             "b_sum",
...             measure_column="b_sum",
...             join_columns=["a"],
...             baselines=list(baseline_options.keys()) + ["custom_baseline"],
...         ),
...     ] # This is required to use the built-in metrics

Just like a SessionProgram, once a subclass of SessionProgramTuner is defined, it can be instantiated using the automatically-generated builder for that class. Unlike a SessionProgram, you can pass Tunable objects to the builder methods instead of concrete values.

>>> protected_df = spark.createDataFrame([("x", 2), ("y", 4)], ["a", "b"])
>>> tuner = (
...     Tuner.Builder()
...     .with_privacy_budget(Tunable("budget"))
...     .with_private_dataframe("protected_df", protected_df, AddOneRow())
...     .with_parameter("low", 0)
...     .with_parameter("high", Tunable("high"))
...     .build()
... )

The outputs() method can be used to run the program to get the outputs of the DP and baseline programs.

>>> dp_outputs, baseline_outputs = (
...     tuner.outputs({"budget": PureDPBudget(1), "high": 1})
... )

The error_report() method on the tuner can be used to run the program to get the DP and baseline outputs as well as the metrics defined in the Tuner class.

>>> tuner.error_report({"budget": PureDPBudget(1), "high": 1}).show()  
Error report ran with budget PureDPBudget(epsilon=1) and the following tunable parameters:
budget: PureDPBudget(epsilon=1)
high: 1
and the following additional parameters:
low: 0

Metric results:
+---------+-------------------------+------------------------+-------------------------------------------------------+
|   Value | Metric                  | Baseline               | Description                                           |
+=========+=========================+========================+=======================================================+
|    0    | mre                     | use_clamping_bounds    | Median relative error for column b_sum of table b_sum |
+---------+-------------------------+------------------------+-------------------------------------------------------+
|    0.5  | mre                     | ignore_clamping_bounds | Median relative error for column b_sum of table b_sum |
+---------+-------------------------+------------------------+-------------------------------------------------------+
|    0.5  | mre                     | custom_baseline        | Median relative error for column b_sum of table b_sum |
+---------+-------------------------+------------------------+-------------------------------------------------------+
|    0    | root_mean_squared_error | use_clamping_bounds    | User-defined metric (no description)                  |
+---------+-------------------------+------------------------+-------------------------------------------------------+
|    3.16 | root_mean_squared_error | ignore_clamping_bounds | User-defined metric (no description)                  |
+---------+-------------------------+------------------------+-------------------------------------------------------+
|    3.16 | root_mean_squared_error | custom_baseline        | User-defined metric (no description)                  |
+---------+-------------------------+------------------------+-------------------------------------------------------+

Functions#

`baseline()`	Decorator to define a custom baseline method for `SessionProgramTuner`.
`metric()`	Decorator to define a custom metric method for `SessionProgram`.
`view()`	Views of the output table to be used across metrics in place of program outputs.

baseline(name)#

Decorator to define a custom baseline method for SessionProgramTuner.

To use the “default” baseline in addition to this custom baseline, you need to separately specify “default”: NoPrivacySession.Options() in baseline_options class variable.

Parameters: name (str) – A name for the custom baseline.

>>> from tmlt.analytics.session import Session

>>> class Program(SessionProgram):
...     class ProtectedInputs:
...         protected_df: DataFrame
...     class UnprotectedInputs:
...         unprotected_df: DataFrame
...     class Outputs:
...         output_df: DataFrame
...     def session_interaction(self, session: Session):
...         ...
>>> class Tuner(SessionProgramTuner, program=Program):
...     @baseline("custom_baseline")
...     def custom_baseline(
...         protected_inputs: Dict[str, DataFrame],
...     ) -> Dict[str, DataFrame]:
...         ...
...     @baseline("another_custom_baseline")
...     def another_custom_baseline(
...         self,
...         protected_inputs: Dict[str, DataFrame],
...         unprotected_inputs: Dict[str, DataFrame],
...     ) -> Dict[str, DataFrame]:
...         # If the program has unprotected inputs or parameters, the custom
...         # baseline method can take them as an argument.
...         ...
...     baseline_options = {
...         "default": NoPrivacySession.Options()
...     }  # This is required to keep the default baseline

metric(name, output, join_columns, grouping_columns=None, description=None, baselines=None, indicator_column_name=None, join_how='inner')#

Decorator to define a custom metric method for SessionProgram.

The decorated metric must have the following parameters: grouped_output: A GroupedData produced by joining the DP and baseline output on the specified columns, and grouping by the specified columns. output_column_name: If the metric returns a table, the value of the metric should be in a column with this name.

The following parameters are allowed: self: A reference to the CustomGroupedMetric class created for this metric. grouping_columns: The columns the output is grouped by. unprotected_inputs: The program’s unprotected inputs. Not allowed if the program has none. program_parameters: The program’s parameters. Not allowed if the program has none.

Alternatively, you can use CustomGroupedMetric directly.

To use the built-in metrics in addition to this custom metric, you can separately specify metrics class variable.

Parameters

name (str) – A name for the metric.
description (Optional[str]) – A description of the metric.
output (str) – The output to compute the metric for.
join_columns (List[str]) – The list of output columns to use to join the dp and baseline output.
grouping_columns (Optional[List[str]]) – If specified, calculate the metric per group when the data is grouped by these columns.
baselines (Optional[Union[str, List[str]]]) – The name of the baseline program(s) used for the error report. If None, use all baselines specified as custom baseline and baseline options on tuner class. If no baselines are specified on tuner class, use default baseline. If a string, use only that baseline. If a list, use only those baselines.
indicator_column_name (Optional[str]) – If specified, we will add a column with the specified name to the joined data that contains either “dp”, “baseline”, or “both” to indicate where the values in the row came from.
join_how (str) – The type of join to perform. Should be one of “left”, “right”, “inner” or “outer”.

>>> from tmlt.analytics.session import Session
>>> from tmlt.analytics.metrics import AbsoluteError
>>> from pyspark.sql import GroupedData

>>> class Program(SessionProgram):
...     class ProtectedInputs:
...         protected_df: DataFrame
...     class UnprotectedInputs:
...         unprotected_df: DataFrame
...     class Outputs:
...         output_df: DataFrame
...     def session_interaction(self, session: Session):
...         return {"output_df": dp_output}
>>> class Tuner(SessionProgramTuner, program=Program):
...     @metric(name="custom_metric",
...         output="output_df",
...         join_columns=["join_column"]
...     )
...     def custom_metric(
...         grouped_output: GroupedData, output_column_name: str
...     ):
...         # If the program has unprotected inputs and/or parameters, the custom
...         #  metric method can take them as an argument.
...         ...
...     metrics = [
...         AbsoluteError(output="output_df", column="Y"),
...     ]  # You can mix custom and built-in metrics.

view(name)#

Views of the output table to be used across metrics in place of program outputs.

Parameters: name (str) – A name for the output view.

>>> from tmlt.analytics.session import Session
>>> from tmlt.analytics.metrics import RelativeError

>>> class Program(SessionProgram):
...     class ProtectedInputs:
...         protected_df: DataFrame
...     class UnprotectedInputs:
...         unprotected_df: DataFrame
...     class Outputs:
...         output_df: DataFrame
...     def session_interaction(self, session: Session):
...         ...
>>> class Tuner(SessionProgramTuner, program=Program):
...     @view("output_view")
...     def custom_view1(
...         outputs: Dict[str, DataFrame],
...     ) -> DataFrame:
...         ...
...     @view("another_output_view")
...     def custom_view2(
...         self,
...         outputs: Dict[str, DataFrame],
...         unprotected_inputs: Dict[str, DataFrame],
...     ) -> DataFrame:
...         # If the program has unprotected inputs or parameters, the view method
...         # can take them as an argument.
...         ...
...     metrics = [
...         RelativeError("output_view", column="a_sum"),
...     ] # The view can be used instead of output when metric is defined

Classes#

`SessionProgramTuner`	Base class for defining an object to tune inputs to a `SessionProgram`.
`Tunable`	Named placeholder for a single input to a `Builder`.
`ErrorReport`	Output of a single error report run.
`MultiErrorReport`	Output of an error report run across multiple input combinations.
`UnprotectedInput`	An unprotected input that was used for an `ErrorReport`.
`ProtectedInput`	A protected input that was used for an `ErrorReport`.

class SessionProgramTuner(builder)#

Base class for defining an object to tune inputs to a SessionProgram.

Note

This is only available on a paid version of Tumult Analytics. If you would like to hear more, please contact us at info@tmlt.io.

SessionProgramTuners should not be directly constructed. Instead, users should create a subclass of SessionProgramTuner, then construct their SessionProgramTuner using the auto-generated Builder attribute of the subclass.

Parameters: builder (SessionProgramTuner) –

class Builder#

The builder for a specific subclass of SessionProgramTuner.

with_private_dataframe(source_id, dataframe, protected_change)#

Add a tunable private dataframe to the builder.

Parameters

source_id (str) –
dataframe (Union[pyspark.sql.DataFrame, Tunable]) –
protected_change (Union[tmlt.analytics.protected_change.ProtectedChange, Tunable]) –

Return type

SessionProgramTuner

with_public_dataframe(source_id, dataframe)#

Add a tunable public dataframe to the builder.

Parameters

source_id (str) –
dataframe (Union[pyspark.sql.DataFrame, Tunable]) –

Return type

SessionProgramTuner

with_parameter(name, value)#

Set the value of a parameter.

Parameters

name (str) –
value (Any) –

build()#

Returns an instance of the matching SessionProgramTuner subtype.

Return type: SessionProgramTuner

with_id_space(id_space)#

Adds an identifier space.

This defines a space of identifiers that map 1-to-1 to the identifiers being protected by a table with the AddRowsWithID protected change. Any table with such a protected change must be a member of some identifier space.

Parameters: id_space (str) –

with_privacy_budget(privacy_budget)#

Set the privacy budget for the object being built.

Parameters: privacy_budget (Union[tmlt.analytics.privacy_budget.PrivacyBudget, Tunable]) –

baseline_options :Optional[Union[Dict[str, tmlt.analytics.no_privacy_session.NoPrivacySession.Options], tmlt.analytics.no_privacy_session.NoPrivacySession.Options]]#

Configuration for how baseline outputs are computed.

By default, a SessionProgramTuner computes both the DP outputs and the baseline outputs for a SessionProgram to compute metrics. The baseline outputs are computed by calling the session_interaction() method with a NoPrivacySession. The baseline_options attribute allows you to override the default options for the NoPrivacySession used to compute the baseline. You can also specify multiple configurations to compute the baselines with different options. When multiple baseline configurations are specified, the metrics are computed with respect to each of the baseline configurations (unless specified otherwise in the metric definitions).

To override the default baseline options (see Options), you can set this to an Options object.

If you want to specify multiple baseline configurations, you can set this to a dictionary mapping baseline names to Options.

metrics :Optional[List[tmlt.analytics.metrics.Metric]]#: A list of metrics to compute in each error_report.

program :Type[tmlt.analytics.program.SessionProgram]#: A subclass of SessionProgram to be tuned.

__init__(builder)#

Constructor.

Warning

This constructor is not intended to be used directly. Use the automatically generated builder instead. It can be accessed using the Builder attribute of the subclass.

Parameters: builder (tmlt.analytics.tuner._tuner.SessionProgramTuner.Builder) –

property tunables#

Returns a list of tunable inputs associated with this tuner.

Return type: List[Tunable]

outputs(tunable_values=None)#

Computes all outputs for a single run.

Parameters: tunable_values (Optional[Dict[str, Any]]) – A dictionary mapping names of Tunables to concrete values to use for this run. Every Tunable used in building this tuner must have a value in this dictionary. This can be None only if no Tunables were used.
Return type: Tuple[Dict[str, pyspark.sql.DataFrame], Dict[str, Dict[str, pyspark.sql.DataFrame]]]

error_report(tunable_values=None)#

Computes DP outputs, baseline outputs, and metrics for a single run.

Parameters: tunable_values (Optional[Dict[str, Any]]) – A dictionary mapping names of Tunables to concrete values to use for this error report. Every Tunable used in building this tuner must have a value in this dictionary. This can be None only if no Tunables were used.
Return type: tmlt.analytics.tuner._error_report.ErrorReport

multi_error_report(tunable_values_list)#

Runs the error_report for each set of values for the Tunables.

Parameters: tunable_values_list (List[Dict[str, Any]]) –
Return type: tmlt.analytics.tuner._error_report.MultiErrorReport

class Tunable#

Named placeholder for a single input to a Builder.

Note

This is only available on a paid version of Tumult Analytics. If you would like to hear more, please contact us at info@tmlt.io.

When a Tunable is passed to a Builder, it is replaced with the concrete values for the tunable parameter when building SessionProgram s inside of methods like error_report() and multi_error_report().

name :str#: Name of the tunable parameter.

class ErrorReport#

Output of a single error report run.

Note

This is only available on a paid version of Tumult Analytics. If you would like to hear more, please contact us at info@tmlt.io.

This class is not intended to be constructed directly. Instead, it is returned by the error_report() method.

Attributes#
`tunable_values`	The values of the tunable parameters used for this error report.
`parameters`	The non-tunable parameters used for this error report.
`protected_inputs`	The protected inputs used for this error report.
`unprotected_inputs`	The unprotected inputs used for this error report.
`privacy_budget`	The privacy budget used for this error report.
`dp_outputs`	The differentially private outputs of the program.
`baseline_outputs`	The outputs of the baseline program.
`metric_results`	The metrics computed on the outputs of the dp and baseline programs.

Methods#
`result_dataframes()`	Return the result dataframes for each metric.
`format()`	Return a string representation of this object.
`show()`	Prints the error report in a nicely-formatted, human-readable way.

tunable_values :Dict[str, Any]#: The values of the tunable parameters used for this error report.

parameters :Dict[str, Any]#: The non-tunable parameters used for this error report.

protected_inputs :Dict[str, ProtectedInput]#: The protected inputs used for this error report.

unprotected_inputs :Dict[str, UnprotectedInput]#: The unprotected inputs used for this error report.

privacy_budget :tmlt.analytics.privacy_budget.PrivacyBudget#: The privacy budget used for this error report.

dp_outputs :Dict[str, pyspark.sql.DataFrame]#: The differentially private outputs of the program.

baseline_outputs :Dict[str, Dict[str, pyspark.sql.DataFrame]]#: The outputs of the baseline program.

metric_results :List[tmlt.analytics.metrics._base.MetricResult]#: The metrics computed on the outputs of the dp and baseline programs.

result_dataframes()#

Return the result dataframes for each metric.

Return type: Dict[str, tmlt.analytics.metrics._base.MetricResultDataframe]

format()#: Return a string representation of this object.

show()#: Prints the error report in a nicely-formatted, human-readable way.

class MultiErrorReport(reports)#

Output of an error report run across multiple input combinations.

Note

This is only available on a paid version of Tumult Analytics. If you would like to hear more, please contact us at info@tmlt.io.

This class is not intended to be constructed directly. Instead, it is returned by the multi_error_report() method.

Parameters: reports (List[ErrorReport]) –

__init__(reports)#

Constructor.

Warning

This class is not intended to be constructed directly. Instead, it is returned by the multi_error_report() method.

Parameters: reports (List[ErrorReport]List[ErrorReport]) – An error report for each run.

property reports#

Return the error reports.

Return type: List[ErrorReport]

__iter__()#

Return an iterator over the error reports.

Return type: Iterator[ErrorReport]

result_dataframes()#

Return the result dataframes for each metric combined across runs.

Each metric produces a separate dataframe. The dataframes are keyed by the metric names.

Return type: Dict[str, pandas.DataFrame]

dataframe()#

Return a dataframe representation of the error reports.

The dataframe will have a row for each error report (run) and a column for each tunable and metric. If grouped metrics are specified, the dataframe will also have a row for the grouping columns.

If not all metrics have the same (or no) grouping column, this will throw an error.

Return type: pandas.DataFrame

class UnprotectedInput#

Bases: NamedTuple

An unprotected input that was used for an ErrorReport.

name :str#: The name of the input.

dataframe :pyspark.sql.DataFrame#: A dataframe containing the unprotected data used for the report.

class ProtectedInput#

Bases: NamedTuple

A protected input that was used for an ErrorReport.

Warning

Note that normally ProtectedInputs are treated as sensitive and would not accessible to the user except through the Session API to avoid violating differential privacy. But these error reports are not differentially private, and for this reason it is highly recommended to avoid using sensitive data in error reports, and to instead use synthetic data or other non-sensitive data.

For these reasons, the protected inputs used in error reports are attached to the outputs for your convenience, but it is ultimately your responsibility to ensure that truly sensitive data is not used inappropriately.

name :str#: The name of the input.

dataframe :pyspark.sql.DataFrame#: A dataframe containing the protected data used for the report.

protected_change :tmlt.analytics.protected_change.ProtectedChange#: What changes to the protected data the Session should protect.

Tumult Analytics Pro

tuner#

Functions#

Classes#