infer_stypes#

sdm.infer_stypes(table: pa.Table | pd.DataFrame | cudf.DataFrame, overrides: Mapping[str, StypeLike] | None = None, *, text: Literal['off', 'infer', 'drop'] = 'off', id: Literal['off', 'infer', 'drop'] = 'off', unsupported: Literal['error', 'warn', 'drop'] = 'error') → dict[str, StypeLike]#

Infer semantic types from raw data statistics.

Semantic types are inferred as follows:

  • Integer, floating-point, and decimal columns are inferred as numerical.

  • String, boolean and dictionary-encoded columns are inferred as categorical.

  • Datetime columns are inferred as datetime.

Optionally, infer the following semantic types based on best-effort heuristics:

  • String columns are inferred as text if they contain at least 200 distinct string values, a distinct non-null value ratio of at least 5%, and an average of at least 3 words per distinct value.

  • Integer or (non-dictionary) string columns are inferred as id if its name contains "id" as a whole word (e.g., "user_id", "userId", "id", but not "solid" or "covid").

Parameters:

  • table (pa.Table | pd.DataFrame | cudf.DataFrame) – A pandas.DataFrame, pyarrow.Table, or cudf.DataFrame.

  • overrides (Mapping[str, StypeLike] | None) – Optional semantic type overrides by column name.

  • text (Literal['off', 'infer', 'drop']) – The text detection policy. "off" disables text column detection. "infer" includes inferred text columns. "drop" omits inferred text columns.

  • id (Literal['off', 'infer', 'drop']) – The id detection policy. "off" disables id column detection. "infer" includes inferred id columns. "drop" omits inferred id columns.

  • unsupported (Literal['error', 'warn', 'drop']) – How to handle unsupported dtypes. "error" raises a TypeError. "warn" emits a warning and omits the column. "drop" omits the column silently.

Returns:

Dictionary mapping column names to inferred semantic type.

Return type:

dict[str, StypeLike]