Conversation
Update develop
… types. Todos to update the preprocessing method to distinguish between categorical, numerical, and boolean features based on configuration. Added TODOs for implementing separate pipelines for categorical and numerical processing. Included plans to protect relevant features based on configuration.
Scaler is a wrap-up estimator for any scaler used that can passthrough encoded features. Imputer is a wrap-up estimator covering simple imputer and categorical imputer. Categorical econder one-hot-encodes categorical features.
Updated the scaler to use `scale_categorical_features` instead of `scale_encoded_features` for clarity. The imputer was expanded to separately handle numerical and categorical features, introducing dedicated imputers for each and enhancing column-specific processing. Imputer is not tested yet and logging is missing.
Replaced sklearn's StandardScaler and SimpleImputer with custom Scaler and Imputer classes. Improved CategoricalEncoder to handle missing columns gracefully and separate numerical and categorical processing. Added validation with `check_is_fitted` in data_preprocessor.py ensure transformers are correctly applied.
…`CategoricalEncoder` with detailed docstrings, improved handling of numerical and categorical features, and fixed feature name consistency issues. Data preprocessing pipeline successfully tested.
- Update Imputer to drop non-declared categorical features and log details.. - Improve DataPreprocessor docstring and fix scaler_type parameter name. - Validate scaler_type in Scaler initialization. - Update FaultDetector to ensure preprocessor is fitted before use and log warnings for non-numeric data.
The class is an alternative imputer based on ffill stratey up to a set limit of samples. Data is previously resmpled to 1 minute by default. Remaining nans ar dropped. Added 'median' strategy to simple imputer. Added the condition that only one type of imputer can be present in the steps config. Refatoring of other preprocessor steps. Transform method of DataPreprocessor now returns the transformed object as it is returned by the sklear pipeline transform method, i.e. without explicitly converting to dataframe.
…d categorical features and harmonized data types in numerical columns converting numerical columns, which might inlcude booleans, into float. Updated FaultDetector docstrings for clarity.
…pdated .gitignore to include generated documentation files, and added TODO comments in FaultDetector for handling of encoded categorical features as conditions.
- Enhanced `TestDataPreprocessorPipeline` to include tests for a preprocessor with a categorical encoder and forward fill imputer. - Updated `TestForwardFillImputer` to validate fitting with non-declared categorical features and improved handling of NaN values. - Introduced `TestScaler` to cover various scenarios including fitting, transforming, and inverse transforming with both standard and min-max scalers, as well as handling of categorical features, negative values, outliers, and constant columns.
…eps and other advanced parameters. Refactored base_config.yaml to reflect changes in the data_preprocessor.
…d categorical columns
…ions are now resolved properly even if the declared name is a substring on any column in data, which is the case of encoded categorical featrues. Improve clarity and logging for available, missing and surviving conditions during preprocessing.
…FaultDetector comments. Improved handling of available protected features and clarified potential issues with conditional feature name substrings.
…ng pipeline - Changed autoencoder name to ConditionalAutoencoder in advanced_config.yaml. - Added comprehensive tests for preprocessing pipeline, including dtype conversion, expected columns, NaN handling, and integration with FaultDetector. - Ensured protected features are retained during preprocessing and validated conditional feature handling.
Unit tests: - Fix TestComprehensivePipelineFlow.setUp: use iloc instead of positional .loc slices (rejected by newer pandas on DatetimeIndex). - Fix inverted protected-features check in DataPreprocessor.fit and the corresponding test assertion: iterate over protected features, not output columns, so normal columns are no longer flagged as dropped. - Adjust test_pipeline_inverse_transform to compare against the transformed row count, since ForwardFillImputer drops rows with remaining NaNs. Categorical feature handling: - CategoricalEncoder, Imputer, ForwardFillImputer now match categorical_features by exact name (they run pre-encoding); Scaler keeps substring matching to cover one-hot encoded columns. - Imputer.inverse_transform: preserve the input index instead of overwriting it with the fit-time index (fixes wrong row alignment at inference). - CategoricalEncoder.inverse_transform: use categorical_columns (the fitted names) instead of categorical_features, consistent with get_feature_names_out. - DataPreprocessor._build_from_steps_spec: deepcopy the steps spec so name normalization and step_name assignment no longer mutate the caller's config. - Scaler.inverse_transform: add the same column-order validation as transform. - CategoricalEncoder.transform: report only the actually-missing columns in the KeyError message, and guard non-DataFrame input with a TypeError. Other: - Promote non-declared categorical features dropped log messages from INFO to WARNING across CategoricalEncoder, Imputer, and ForwardFillImputer. - Document ffill_limit as a time delta (count x freq) in ForwardFillImputer. - Update docstrings to reflect exact-name vs substring matching.
ForwardFillImputer: - Replace resampling-based ffill with time-based ffill: remove asfreq() resampling, preserve original (possibly irregular) timestamps. - Replace freq + int ffill_limit with a single Timedelta ffill_limit (string or pd.Timedelta). Forward-fill all NaNs, then invalidate fills where elapsed time from the last valid observation exceeds the threshold. - Drop PeriodIndex support (only DatetimeIndex/TimedeltaIndex now). - Drop all-NaN columns in fit and log a warning, preventing dropna from wiping the entire dataset when a column is unusable. Imputer: - Guard numerical_imputer.fit/transform when no numerical columns are present (categorical-only data no longer crashes). - Drop all-NaN columns in fit and log a warning, preventing the SimpleImputer silent-skip that caused a column-count mismatch on transform. Tests: - Update ffill_imputer tests for Timedelta-based ffill_limit. - Update test_all_nan_column_handling to assert the column is dropped. - Enable test_transform_with_only_categorical (was TODO/commented out). - Update test_fit_with_all_nan_column to assert the column is dropped and transform succeeds.
Fix RST formatting issues in docstrings (numbered lists, bullet alignment, indentation) that caused Sphinx build warnings. Re-enable run_apidoc in conf.py with templatedir, and strip 'energy_fault_detector.' prefix from auto-generated API page titles for a cleaner sidebar. Exclude sklearn boilerplate methods (set_fit_request, set_output, __sklearn_is_fitted__, etc.) from autodoc via exclude-members and a high-priority skip-member handler. Document generate_quickstart_config by un-excluding config/ __init__.py and using :no-index: to prevent duplicate Config object warnings.
…nd dead code - Emit canonical 'scaler' step name with 'scaler_type' param in generate_quickstart_config() and _data_preprocessor_params_to_steps() instead of unregistered 'standard_scaler'/'minmax_scaler' (BUG #1) - Add _migrate_legacy_scaler_names() to DataPreprocessor that auto-converts old names with a DeprecationWarning showing the correct config format - Fix tune() mutating the user's Config object by using a local variable - Remove unreachable dead code in QuantileThresholdSelector.fit - Add empty-data guard in preprocess_train_data raising ValueError - Set OneHotEncoder handle_unknown='ignore' in CategoricalEncoder and log warning for unseen categories instead of crashing - Fix tests that passed for the wrong reason (unknown-step-name error masked the intended singleton/duplicate-scaler checks) - Add test_legacy_scaler_names_migrated - Update notebook configs and docs to use canonical scaler name
- Add `_handle_duplicate_index` to FaultDetectionModel base class: drops true duplicates (same index + same values), raises ValueError for contradicting duplicates (same index, different values) - Replace blanket duplicate-index rejection in FaultDetector with the new helper, applied in preprocess_train_data and predict - Update ForwardFillImputer docstrings to explain drop_duplicates removes exact duplicates created by forward-filling - Add TestHandleDuplicateIndex covering DataFrame and Series variants - Silence decoder-not-loaded warning (many models don't save the decoder)
- Update Imputer to drop non-declared categorical features and log details.. - Improve DataPreprocessor docstring and fix scaler_type parameter name. - Validate scaler_type in Scaler initialization. - Update FaultDetector to ensure preprocessor is fitted before use and log warnings for non-numeric data.
…d categorical features and harmonized data types in numerical columns converting numerical columns, which might inlcude booleans, into float. Updated FaultDetector docstrings for clarity.
…pdated .gitignore to include generated documentation files, and added TODO comments in FaultDetector for handling of encoded categorical features as conditions.
Fix RST formatting issues in docstrings (numbered lists, bullet alignment, indentation) that caused Sphinx build warnings. Re-enable run_apidoc in conf.py with templatedir, and strip 'energy_fault_detector.' prefix from auto-generated API page titles for a cleaner sidebar. Exclude sklearn boilerplate methods (set_fit_request, set_output, __sklearn_is_fitted__, etc.) from autodoc via exclude-members and a high-priority skip-member handler. Document generate_quickstart_config by un-excluding config/ __init__.py and using :no-index: to prevent duplicate Config object warnings.
Replace version-specific Zenodo DOIs (14958989, 15846963) with the concept DOI 10958774 across docs, source, and notebooks so links always point to the latest version. The concept ID resolves correctly via Zenodo's REST API, so download_zenodo_data and Care2CompareDataset now default to it. Fix broken evaluate_event kwargs in the CARE to Compare notebook (event_start_id/event_end_id -> event_start/event_end), switch all examples to index_column='time_stamp', and update the data directory name to CARE_To_Compare.
- Model Finetuning: get_formatted_event_dataset -> load_and_format_event_dataset - Create new model classes: get_dataset_for_event -> load_event_dataset; migrate data_preprocessor config from deprecated params style to steps - Hyperparameter Optimization: save_model -> save_models (plural); eventwise_f_score_beta -> reliability_beta; remove ineffective decay_rate sampling (decay_steps was never set)
…move redundant config - Add concise module-level docstrings to all subpackage __init__.py files so the API reference and code navigation describe each submodule's purpose at a glance, without duplicating the class lists Sphinx generates. - fault_detector: capture declared conditional features before the preprocessor expands categorical condition names to one-hot columns, so the available/missing checks operate on the original names. - registration: drop the stale 'data_preprocessor' from the class_type docstring (DataTransformer steps are registered in DataPreprocessor, not the global registry). - docs/sequence_models: rename section headings from 'Seq2One/Seq2Seq models' to the more readable 'Sequence-to-one/Sequence-to-sequence models'. - docs/index: reorder the 'Models and methods' toctree entries. - Remove energy_fault_detector/advanced_config.yaml: it was neither shipped (package-data lists only base_config.yaml and c2c_options.yaml) nor used by code, and had diverged from the canonical docs/examples/advanced_config.yaml shown in configuration.rst.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merges the latest
developupdates intomain. It introduces an overhaul of the data preprocessing pipeline to support categorical features, alongside documentation updates and example notebook fixes.Included Changes
Details
Preprocessing Pipeline Overhaul (#30)
CategoricalEncoder,Imputer,ForwardFillImputer, andScaler.DataPreprocessor(NaN-introducing steps -> ColumnSelector -> ... -> Imputer -> CategoricalEncoder -> Scaler -> TimestampTransformer).FaultDetectorto handle encoded categorical columns and improved logging.advanced_config.yamland fixed various Sphinx build warnings.Documentation & Maintenance (#31)