query_expr#

Building blocks of the Analytics query language. Not for direct use.

Defines the QueryExpr class, which represents expressions in the Analytics query language. QueryExpr and its subclasses should not be directly constructed or deconstructed by most users; interfaces such as tmlt.analytics.query_builder.QueryBuilder to create them and tmlt.analytics.session.Session to consume them provide more user-friendly features.

Data#

Row#

Type alias for dictionary with string keys.

Classes#

CountMechanism

Possible mechanisms for the count() aggregation.

CountDistinctMechanism

Enumerating the possible mechanims used for the count_distinct aggregation.

SumMechanism

Possible mechanisms for the sum() aggregation.

AverageMechanism

Possible mechanisms for the average() aggregation.

VarianceMechanism

Possible mechanisms for the variance() aggregation.

StdevMechanism

Possible mechanisms for the stdev() aggregation.

QueryExpr

A query expression, base class for relational operators.

PrivateSource

Loads the private source.

Rename

Returns the dataframe with columns renamed.

Filter

Returns the subset of the rows that satisfy the predicate.

Select

Returns a subset of the columns.

Map

Applies a map function to each row of a relation.

FlatMap

Applies a flat map function to each row of a relation.

JoinPrivate

Returns the join of two private tables.

JoinPublic

Returns the join of a private and public table.

AnalyticsDefault

Default values for each type of column in Analytics.

ReplaceNullAndNan

Returns data with null and NaN expressions replaced by a default.

ReplaceInfinity

Returns data with +inf and -inf expressions replaced by defaults.

DropNullAndNan

Returns data with rows that contain null or NaN value dropped.

DropInfinity

Returns data with rows that contain +inf/-inf dropped.

GroupByCount

Returns the count of each combination of the groupby domains.

GroupByCountDistinct

Returns the count of distinct records in each groupby domain value.

GroupByQuantile

Returns the quantile of a column for each combination of the groupby domains.

GroupByBoundedSum

Returns the bounded sum of a column for each combination of groupby domains.

GroupByBoundedAverage

Returns bounded average of a column for each combination of groupby domains.

GroupByBoundedVariance

Returns bounded variance of a column for each combination of groupby domains.

GroupByBoundedSTDEV

Returns bounded stdev of a column for each combination of groupby domains.

QueryExprVisitor

A base class for implementing visitors for QueryExpr.

class CountMechanism#

Bases: enum.Enum

Possible mechanisms for the count() aggregation.

Currently, the count() aggregation uses an additive noise mechanism to achieve differential privacy.

DEFAULT#

The framework automatically selects an appropriate mechanism. This choice might change over time as additional optimizations are added to the library.

LAPLACE#

Double-sided geometric noise is used.

GAUSSIAN#

The discrete Gaussian mechanism is used. Not compatible with pure DP.

name(self)#

The name of the Enum member.

value(self)#

The value of the Enum member.

class CountDistinctMechanism#

Bases: enum.Enum

Enumerating the possible mechanims used for the count_distinct aggregation.

Currently, the count_distinct() aggregation uses an additive noise mechanism to achieve differential privacy.

DEFAULT#

The framework automatically selects an appropriate mechanism. This choice might change over time as additional optimizations are added to the library.

LAPLACE#

Double-sided geometric noise is used.

GAUSSIAN#

The discrete Gaussian mechanism is used. Not compatible with pure DP.

name(self)#

The name of the Enum member.

value(self)#

The value of the Enum member.

class SumMechanism#

Bases: enum.Enum

Possible mechanisms for the sum() aggregation.

Currently, the sum() aggregation uses an additive noise mechanism to achieve differential privacy.

DEFAULT#

The framework automatically selects an appropriate mechanism. This choice might change over time as additional optimizations are added to the library.

LAPLACE#

Laplace and/or double-sided geometric noise is used, depending on the column type.

GAUSSIAN#

The discrete Gaussian mechanism is used for integer sums. Not compatible with floating-point sums or pure DP.

name(self)#

The name of the Enum member.

value(self)#

The value of the Enum member.

class AverageMechanism#

Bases: enum.Enum

Possible mechanisms for the average() aggregation.

Currently, the average() aggregation uses an additive noise mechanism to achieve differential privacy.

DEFAULT#

The framework automatically selects an appropriate mechanism. This choice might change over time as additional optimizations are added to the library.

LAPLACE#

Laplace and/or double-sided geometric noise is used, depending on the column type.

GAUSSIAN#

The discrete Gaussian mechanism is used for integer averages. Not compatible with floating-point averages or pure DP.

name(self)#

The name of the Enum member.

value(self)#

The value of the Enum member.

class VarianceMechanism#

Bases: enum.Enum

Possible mechanisms for the variance() aggregation.

Currently, the variance() aggregation uses an additive noise mechanism to achieve differential privacy.

DEFAULT#

The framework automatically selects an appropriate mechanism. This choice might change over time as additional optimizations are added to the library.

LAPLACE#

Laplace and/or double-sided geometric noise is used, depending on the column type.

GAUSSIAN#

The discrete Gaussian mechanism is used for integer variance. Not compatible with floating-point averages or pure DP.

name(self)#

The name of the Enum member.

value(self)#

The value of the Enum member.

class StdevMechanism#

Bases: enum.Enum

Possible mechanisms for the stdev() aggregation.

Currently, the stdev() aggregation uses an additive noise mechanism to achieve differential privacy.

DEFAULT#

The framework automatically selects an appropriate mechanism. This choice might change over time as additional optimizations are added to the library.

LAPLACE#

Laplace and/or double-sided geometric noise is used, depending on the column type.

GAUSSIAN#

The discrete Gaussian mechanism is used for integer stdev. Not compatible with floating-point averages or pure DP.

name(self)#

The name of the Enum member.

value(self)#

The value of the Enum member.

class QueryExpr#

Bases: abc.ABC

A query expression, base class for relational operators.

In most cases, QueryExpr should not be manipulated directly, but rather created using tmlt.analytics.query_builder.QueryBuilder and then consumed by tmlt.analytics.session.Session. While they can be created and modified directly, this is an advanced usage and is not recommended for typical users.

QueryExpr are organized in a tree, where each node is an operator which returns a relation.

abstract accept(self, visitor)#

Dispatch methods on a visitor based on the QueryExpr type.

Parameters

visitor (QueryExprVisitor) –

Return type

Any

class PrivateSource#

Bases: QueryExpr

Loads the private source.

source_id :str#

The ID for the private source to load.

accept(self, visitor)#

Visit this QueryExpr with visitor.

Parameters

visitor (QueryExprVisitor) –

Return type

Any

class Rename#

Bases: QueryExpr

Returns the dataframe with columns renamed.

child :QueryExpr#

The QueryExpr to apply Rename to.

column_mapper :Dict[str, str]#

The mapping of old column names to new column names.

This mapping can contain all column names or just a subset. If it contains a subset of columns, it will only rename those columns and keep the other column names the same.

accept(self, visitor)#

Visit this QueryExpr with visitor.

Parameters

visitor (QueryExprVisitor) –

Return type

Any

class Filter#

Bases: QueryExpr

Returns the subset of the rows that satisfy the predicate.

child :QueryExpr#

The QueryExpr to filter.

predicate :str#

A string of SQL expression specifying the filter to apply to the data.

For example, the string “A > B” matches rows where column A is greater than column B.

accept(self, visitor)#

Visit this QueryExpr with visitor.

Parameters

visitor (QueryExprVisitor) –

Return type

Any

class Select#

Bases: QueryExpr

Returns a subset of the columns.

child :QueryExpr#

The QueryExpr to apply the select on.

columns :List[str]#

The columns to select.

accept(self, visitor)#

Visit this QueryExpr with visitor.

Parameters

visitor (QueryExprVisitor) –

Return type

Any

class Map#

Bases: QueryExpr

Applies a map function to each row of a relation.

child :QueryExpr#

The QueryExpr to apply the map on.

f :Callable[[Row], Row]#

The map function.

schema_new_columns :tmlt.analytics._schema.Schema#

The expected schema for new columns produced by f.

augment :bool#

Whether to keep the existing columns.

If True, schema = old schema + schema_new_columns, otherwise only keeps the new columns (schema = schema_new_columns).

accept(self, visitor)#

Visit this QueryExpr with visitor.

Parameters

visitor (QueryExprVisitor) –

Return type

Any

__eq__(self, other)#

Returns true iff self == other.

This uses the bytecode of self.f and other.f to determine if the two functions are equal.

Parameters

other (object) –

Return type

bool

class FlatMap#

Bases: QueryExpr

Applies a flat map function to each row of a relation.

child :QueryExpr#

The QueryExpr to apply the flat map on.

f :Callable[[Row], List[Row]]#

The flat map function.

max_num_rows :int#

The enforced limit on number of rows from each f(row).

schema_new_columns :tmlt.analytics._schema.Schema#

The expected schema for new columns produced by f.

If the schema_new_columns has a grouping_column, that means this FlatMap produces a column that must be grouped by eventually. It also must be the only column in the schema.

augment :bool#

Whether to keep the existing columns.

If True, schema = old schema + schema_new_columns, otherwise only keeps the new columns (schema = schema_new_columns).

accept(self, visitor)#

Visit this QueryExpr with visitor.

Parameters

visitor (QueryExprVisitor) –

Return type

Any

__eq__(self, other)#

Returns true iff self == other.

This uses the bytecode of self.f and other.f to determine if the two functions are equal.

Parameters

other (object) –

Return type

bool

class JoinPrivate#

Bases: QueryExpr

Returns the join of two private tables.

Before performing the join, each table is truncated based on the corresponding TruncationStrategy. For a more detailed overview of JoinPrivate’s behavior, see join_private().

child :QueryExpr#

The QueryExpr to join with right operand.

right_operand_expr :QueryExpr#

The QueryExpr for private source to join with.

truncation_strategy_left :tmlt.analytics.truncation_strategy.TruncationStrategy.Type#

Truncation strategy to be used for the left table.

truncation_strategy_right :tmlt.analytics.truncation_strategy.TruncationStrategy.Type#

Truncation strategy to be used for the right table.

join_columns :Optional[List[str]]#

The columns used for joining the tables, or None to use all common columns.

accept(self, visitor)#

Visit this QueryExpr with visitor.

Parameters

visitor (QueryExprVisitor) –

Return type

Any

class JoinPublic#

Bases: QueryExpr

Returns the join of a private and public table.

child :QueryExpr#

The QueryExpr to join with public_df.

public_table :Union[pyspark.sql.DataFrame, str]#

A DataFrame or public source to join with.

join_columns :Optional[List[str]]#

The columns used for joining the tables, or None to use all common columns.

accept(self, visitor)#

Visit this QueryExpr with visitor.

Parameters

visitor (QueryExprVisitor) –

Return type

Any

__eq__(self, other)#

Returns true iff self == other.

For the purposes of this equality operation, two dataframes are equal if they contain the same data, in any order.

Calling this on a JoinPublic that includes a very large dataframe could take a long time or consume a lot of resources, and is not recommended.

Parameters

other (object) –

Return type

bool

class AnalyticsDefault#

Default values for each type of column in Analytics.

INTEGER = 0#

The default value used for integers (0).

DECIMAL = 0.0#

The default value used for floats (0).

VARCHAR =#

The default value used for VARCHARs (the empty string).

DATE#

The default value used for dates (datetime.date.fromtimestamp(0)).

See fromtimestamp().

TIMESTAMP#

The defauLt value used for timestamps (datetime.datetime.fromtimestamp(0)).

See fromtimestamp().

class ReplaceNullAndNan#

Bases: QueryExpr

Returns data with null and NaN expressions replaced by a default.

Note that after a ReplaceNullAndNan query has been performed for a column, Analytics will raise an error if you use a KeySet for that column that contains null values.

child :QueryExpr#

The QueryExpr to replace null/NaN values in.

replace_with :Mapping[str, Union[int, float, str, datetime.date, datetime.datetime]]#

New values to replace with, by column.

If this dictionary is empty, all columns will be changed, with values replaced by a default value for each column’s type (see the AnalyticsDefault class variables).

accept(self, visitor)#

Visit this QueryExpr with visitor.

Parameters

visitor (QueryExprVisitor) –

Return type

Any

class ReplaceInfinity#

Bases: QueryExpr

Returns data with +inf and -inf expressions replaced by defaults.

child :QueryExpr#

The QueryExpr to replace +inf and -inf values in.

replace_with :Dict[str, Tuple[float, float]]#

New values to replace with, by column. The first value for each column will be used to replace -infinity, and the second value will be used to replace +infinity.

If this dictionary is empty, all columns of type DECIMAL will be changed, with infinite values replaced with a default value (see the AnalyticsDefault class variables).

accept(self, visitor)#

Visit this QueryExpr with visitor.

Parameters

visitor (QueryExprVisitor) –

Return type

Any

class DropNullAndNan#

Bases: QueryExpr

Returns data with rows that contain null or NaN value dropped.

Note that after a DropNullAndNan query has been performed for a column, Analytics will raise an error if you use a KeySet for that column that contains null values.

child :QueryExpr#

The QueryExpr in which to drop nulls/NaNs.

columns :List[str]#

Columns in which to look for nulls and NaNs.

If this list is empty, all columns will be looked at - so if any column contains a null or NaN value that row will be dropped.

accept(self, visitor)#

Visit this QueryExpr with visitor.

Parameters

visitor (QueryExprVisitor) –

Return type

Any

class DropInfinity#

Bases: QueryExpr

Returns data with rows that contain +inf/-inf dropped.

child :QueryExpr#

The QueryExpr in which to drop +inf/-inf.

columns :List[str]#

Columns in which to look for and infinite values.

If this list is empty, all columns will be looked at - so if any column contains an infinite value, that row will be dropped.

accept(self, visitor)#

Visit this QueryExpr with visitor.

Parameters

visitor (QueryExprVisitor) –

Return type

Any

class GroupByCount#

Bases: QueryExpr

Returns the count of each combination of the groupby domains.

child :QueryExpr#

The QueryExpr to measure.

groupby_keys :tmlt.analytics.keyset.KeySet#

The keys of the resulting aggregated data.

output_column :str = count#

The name of the column to store the counts in.

mechanism :CountMechanism#

Choice of noise mechanism.

By DEFAULT, the framework automatically selects an appropriate mechanism.

accept(self, visitor)#

Visit this QueryExpr with visitor.

Parameters

visitor (QueryExprVisitor) –

Return type

Any

class GroupByCountDistinct#

Bases: QueryExpr

Returns the count of distinct records in each groupby domain value.

child :QueryExpr#

The QueryExpr to measure.

groupby_keys :tmlt.analytics.keyset.KeySet#

The keys of the resulting aggregated data.

columns_to_count :Optional[List[str]]#

The columns that are compared when determining if two rows are distinct.

If empty, will count all distinct rows.

output_column :str = count_distinct#

The name of the column to store the distinct counts in.

mechanism :CountDistinctMechanism#

Choice of noise mechanism.

By DEFAULT, the framework automatically selects an appropriate mechanism.

accept(self, visitor)#

Visit this QueryExpr with visitor.

Parameters

visitor (QueryExprVisitor) –

Return type

Any

class GroupByQuantile#

Bases: QueryExpr

Returns the quantile of a column for each combination of the groupby domains.

If the column to be measured contains null, NaN, or positive or negative infinity, those values will be dropped (as if dropped explicitly via DropNullAndNan and DropInfinity) before the quantile is calculated.

child :QueryExpr#

The QueryExpr to measure.

groupby_keys :tmlt.analytics.keyset.KeySet#

The keys of the resulting aggregated data.

measure_column :str#

The column to compute the quantile over.

quantile :float#

The quantile to compute (between 0 and 1).

low :float#

The lower bound for clamping the measure_column.

high :float#

The upper bound for clamping the measure_column.

output_column :str = quantile#

The name of the column to store the quantiles in.

accept(self, visitor)#

Visit this QueryExpr with visitor.

Parameters

visitor (QueryExprVisitor) –

Return type

Any

class GroupByBoundedSum#

Bases: QueryExpr

Returns the bounded sum of a column for each combination of groupby domains.

If the column to be measured contains null, NaN, or positive or negative infinity, those values will be dropped (as if dropped explicitly via DropNullAndNan and DropInfinity) before the sum is calculated.

child :QueryExpr#

The QueryExpr to measure.

groupby_keys :tmlt.analytics.keyset.KeySet#

The keys of the resulting aggregated data.

measure_column :str#

The column to compute the sum over.

low :float#

The lower bound for clamping the measure_column.

high :float#

The upper bound for clamping the measure_column.

output_column :str = sum#

The name of the column to store the sums in.

mechanism :SumMechanism#

Choice of noise mechanism.

By DEFAULT, the framework automatically selects an appropriate mechanism.

accept(self, visitor)#

Visit this QueryExpr with visitor.

Parameters

visitor (QueryExprVisitor) –

Return type

Any

class GroupByBoundedAverage#

Bases: QueryExpr

Returns bounded average of a column for each combination of groupby domains.

If the column to be measured contains null, NaN, or positive or negative infinity, those values will be dropped (as if dropped explicitly via DropNullAndNan and DropInfinity) before the average is calculated.

child :QueryExpr#

The QueryExpr to measure.

groupby_keys :tmlt.analytics.keyset.KeySet#

The keys of the resulting aggregated data.

measure_column :str#

The column to compute the average over.

low :float#

The lower bound for clamping the measure_column.

high :float#

The upper bound for clamping the measure_column.

output_column :str = average#

The name of the column to store the averages in.

mechanism :AverageMechanism#

Choice of noise mechanism.

By DEFAULT, the framework automatically selects an appropriate mechanism.

accept(self, visitor)#

Visit this QueryExpr with visitor.

Parameters

visitor (QueryExprVisitor) –

Return type

Any

class GroupByBoundedVariance#

Bases: QueryExpr

Returns bounded variance of a column for each combination of groupby domains.

If the column to be measured contains null, NaN, or positive or negative infinity, those values will be dropped (as if dropped explicitly via DropNullAndNan and DropInfinity) before the variance is calculated.

child :QueryExpr#

The QueryExpr to measure.

groupby_keys :tmlt.analytics.keyset.KeySet#

The keys of the resulting aggregated data.

measure_column :str#

The column to compute the variance over.

low :float#

The lower bound for clamping the measure_column.

high :float#

The upper bound for clamping the measure_column.

output_column :str = variance#

The name of the column to store the variances in.

mechanism :VarianceMechanism#

Choice of noise mechanism.

By DEFAULT, the framework automatically selects an appropriate mechanism.

accept(self, visitor)#

Visit this QueryExpr with visitor.

Parameters

visitor (QueryExprVisitor) –

Return type

Any

class GroupByBoundedSTDEV#

Bases: QueryExpr

Returns bounded stdev of a column for each combination of groupby domains.

If the column to be measured contains null, NaN, or positive or negative infinity, those values will be dropped (as if dropped explicitly via DropNullAndNan and DropInfinity) before the standard deviation is calculated.

child :QueryExpr#

The QueryExpr to measure.

groupby_keys :tmlt.analytics.keyset.KeySet#

The keys of the resulting aggregated data.

measure_column :str#

The column to compute the standard deviation over.

low :float#

The lower bound for clamping the measure_column.

high :float#

The upper bound for clamping the measure_column.

output_column :str = stdev#

The name of the column to store the stdev in.

mechanism :StdevMechanism#

Choice of noise mechanism.

By DEFAULT, the framework automatically selects an appropriate mechanism.

accept(self, visitor)#

Visit this QueryExpr with visitor.

Parameters

visitor (QueryExprVisitor) –

Return type

Any

class QueryExprVisitor#

Bases: abc.ABC

A base class for implementing visitors for QueryExpr.

Methods#

visit_private_source()

Visit a PrivateSource.

visit_rename()

Visit a Rename.

visit_filter()

Visit a Filter.

visit_select()

Visit a Select.

visit_map()

Visit a Map.

visit_flat_map()

Visit a FlatMap.

visit_join_private()

Visit a JoinPrivate.

visit_join_public()

Visit a JoinPublic.

visit_replace_null_and_nan()

Visit a ReplaceNullAndNan.

visit_replace_infinity()

Visit a ReplaceInfinity.

visit_drop_null_and_nan()

Visit a DropNullAndNan.

visit_drop_infinity()

Visit a DropInfinity.

visit_groupby_count()

Visit a GroupByCount.

visit_groupby_count_distinct()

Visit a GroupByCountDistinct.

visit_groupby_quantile()

Visit a GroupByQuantile.

visit_groupby_bounded_sum()

Visit a GroupByBoundedSum.

visit_groupby_bounded_average()

Visit a GroupByBoundedAverage.

visit_groupby_bounded_variance()

Visit a GroupByBoundedVariance.

visit_groupby_bounded_stdev()

Visit a GroupByBoundedSTDEV.

abstract visit_private_source(self, expr)#

Visit a PrivateSource.

Parameters

expr (PrivateSource) –

Return type

Any

abstract visit_rename(self, expr)#

Visit a Rename.

Parameters

expr (Rename) –

Return type

Any

abstract visit_filter(self, expr)#

Visit a Filter.

Parameters

expr (Filter) –

Return type

Any

abstract visit_select(self, expr)#

Visit a Select.

Parameters

expr (Select) –

Return type

Any

abstract visit_map(self, expr)#

Visit a Map.

Parameters

expr (Map) –

Return type

Any

abstract visit_flat_map(self, expr)#

Visit a FlatMap.

Parameters

expr (FlatMap) –

Return type

Any

abstract visit_join_private(self, expr)#

Visit a JoinPrivate.

Parameters

expr (JoinPrivate) –

Return type

Any

abstract visit_join_public(self, expr)#

Visit a JoinPublic.

Parameters

expr (JoinPublic) –

Return type

Any

abstract visit_replace_null_and_nan(self, expr)#

Visit a ReplaceNullAndNan.

Parameters

expr (ReplaceNullAndNan) –

Return type

Any

abstract visit_replace_infinity(self, expr)#

Visit a ReplaceInfinity.

Parameters

expr (ReplaceInfinity) –

Return type

Any

abstract visit_drop_null_and_nan(self, expr)#

Visit a DropNullAndNan.

Parameters

expr (DropNullAndNan) –

Return type

Any

abstract visit_drop_infinity(self, expr)#

Visit a DropInfinity.

Parameters

expr (DropInfinity) –

Return type

Any

abstract visit_groupby_count(self, expr)#

Visit a GroupByCount.

Parameters

expr (GroupByCount) –

Return type

Any

abstract visit_groupby_count_distinct(self, expr)#

Visit a GroupByCountDistinct.

Parameters

expr (GroupByCountDistinct) –

Return type

Any

abstract visit_groupby_quantile(self, expr)#

Visit a GroupByQuantile.

Parameters

expr (GroupByQuantile) –

Return type

Any

abstract visit_groupby_bounded_sum(self, expr)#

Visit a GroupByBoundedSum.

Parameters

expr (GroupByBoundedSum) –

Return type

Any

abstract visit_groupby_bounded_average(self, expr)#

Visit a GroupByBoundedAverage.

Parameters

expr (GroupByBoundedAverage) –

Return type

Any

abstract visit_groupby_bounded_variance(self, expr)#

Visit a GroupByBoundedVariance.

Parameters

expr (GroupByBoundedVariance) –

Return type

Any

abstract visit_groupby_bounded_stdev(self, expr)#

Visit a GroupByBoundedSTDEV.

Parameters

expr (GroupByBoundedSTDEV) –

Return type

Any