Spaces:

datavision-ai
/

Datavision

Running

Datavision / backend /ml /ml_code_generator.py

DataVision CI/CD Bot

release: clean production build for HuggingFace Space

09801ca 1 day ago

277 kB

	"""
	📦 ML CODE GENERATOR - Dynamic Project ZIP Export
	===================================================

	Generates a downloadable ZIP file containing a complete, runnable Python
	ML project BASED ON THE USER'S ACTUAL TRAINING. Dynamically generates
	code matching the exact modes, algorithms, features, and task type
	that were used during AutoML training.

	All scripts use real column names, actual model types, and genuine
	feature metadata from the user's training session.
	"""

	import io
	import json
	import zipfile
	import logging
	import base64
	from pathlib import Path
	from typing import Dict, Any, Optional, List

	logger = logging.getLogger(__name__)


	API_SERVER_HTML_TEMPLATE = """<!DOCTYPE html>
	<html>
	<head>
	<title>DataVision ML API</title>
	<style>
	body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif; max-width: 800px; margin: 40px auto; padding: 0 20px; background: #f9fafb; color: #111827; }
	h1 { color: #4F46E5; } h2 { color: #1f2937; border-bottom: 2px solid #e5e7eb; padding-bottom: 8px; }
	.card { background: white; border-radius: 12px; padding: 24px; margin: 16px 0; box-shadow: 0 1px 3px rgba(0,0,0,0.1); }
	button { background: #4F46E5; color: white; border: none; padding: 12px 32px; border-radius: 8px; font-size: 16px; cursor: pointer; margin-top: 12px; }
	button:hover { background: #4338CA; }
	#result { margin-top: 20px; padding: 16px; background: #f0fdf4; border-radius: 8px; display: none; }
	</style>
	</head>
	<body>
	<h1>🚀 DataVision ML API</h1>
	<div class="card">
	<h2>Model Info</h2>
	<p>🏆 <b>Model:</b> {{ model_name }} \| 🎯 <b>Target:</b> {{ target }} \| 📊 <b>Task:</b> {{ task_type_val }} \| 📁 <b>Features:</b> {{ raw_feature_info\|length }}</p>
	<ul>
	{% for k, v in metrics.items() %}
	<li><b>{{ k }}</b>: {{ "%.4f"\|format(v) if v is float else v }}</li>
	{% endfor %}
	</ul>
	</div>
	<div class="card">
	<h2>🔮 Make a Prediction</h2>
	<form id="predForm">
	{% for col, info in raw_feature_info.items() %}
	<div style="margin:6px 0">
	<label style="display:inline-block;width:260px;font-weight:500">{{ col }}</label>
	{% if info.type == 'categorical' %}
	<select name="{{ col }}" style="padding:6px 10px;border:1px solid #d1d5db;border-radius:6px;width:214px">
	{% for o in info.options[:20] %}
	<option value="{{ o }}">{{ o }}</option>
	{% endfor %}
	</select>
	{% else %}
	{% set placeholder = (info.min\|string + ' - ' + info.max\|string) if info.min else '' %}
	<input name="{{ col }}" type="text" value="{{ info.default }}" placeholder="{{ placeholder }}" style="padding:6px 10px;border:1px solid #d1d5db;border-radius:6px;width:200px">
	{% endif %}
	</div>
	{% endfor %}
	<button type="submit">Predict</button>
	</form>
	<div id="result"></div>
	</div>
	<div class="card"><h2>📡 API Usage</h2>
	<pre style="background:#1f2937;color:#e5e7eb;padding:16px;border-radius:8px;overflow-x:auto">
	POST /predict
	Content-Type: application/json

	{"feature1": val, "feature2": val}

	GET /health — Health check
	GET /model-info — Model metadata</pre></div>
	<script>
	document.getElementById("predForm").onsubmit=function(e){
	e.preventDefault();
	var fd=new FormData(this);
	var data={};
	fd.forEach(function(v,k){ data[k]=isNaN(v)\|\|v===""?v:parseFloat(v) });
	fetch("/predict",{ method:"POST",headers:{"Content-Type":"application/json"},body:JSON.stringify(data) })
	.then(function(r){ return r.json() })
	.then(function(j){
	var h="<h3>🎯 Prediction: "+j.prediction+"</h3>";
	if(j.confidence)h+="<p>📊 Confidence: "+(j.confidence*100).toFixed(1)+"%</p>";
	if(j.probabilities){
	h+="<p>Probabilities:</p><ul>";
	var ent=Object.entries(j.probabilities).sort(function(a,b){ return b[1]-a[1] });
	for(var i=0;i<ent.length;i++)h+="<li><b>"+ent[i][0]+"</b>: "+(ent[i][1]*100).toFixed(1)+"%</li>";
	h+="</ul>"
	}
	document.getElementById("result").innerHTML=h;
	document.getElementById("result").style.display="block"
	});
	};
	</script>
	</body>
	</html>"""



	def generate_code_zip(
	model_path: Path,
	metadata: Dict[str, Any],
	cleaned_data_path: Optional[Path] = None,
	dl_model_path: Optional[Path] = None,
	charts_data: Optional[Dict[str, str]] = None,
	) -> io.BytesIO:
	"""
	Generate a complete Python ML project as a ZIP file.
	Only includes scripts for modes the user actually trained.
	All code uses real column names, real model types, real metrics.

	Args:
	model_path: Path to model.pkl
	metadata: Training metadata dict
	cleaned_data_path: Path to cleaned_data.csv
	dl_model_path: Path to deep_learning_model.pkl
	charts_data: Dict of chart_name -> base64 data URI strings from active_charts.json

	IMPORTANT: Extracts actual hyperparameters from the trained model
	to ensure generated code reproduces EXACT same results.
	"""
	import pickle

	buf = io.BytesIO()

	# =========================================================================
	# CRITICAL: Load actual model to extract EXACT hyperparameters
	# =========================================================================
	actual_model_params = {}
	actual_model_class = None
	production_engineer = None
	scaler = None
	label_encoders = {}
	target_encoder = None
	model_state = None # Initialize so it's available for clean pkl creation later

	if model_path and model_path.exists():
	try:
	with open(model_path, 'rb') as f:
	model_state = pickle.load(f)

	# Extract the actual trained model
	if model_state is not None:
	actual_model = model_state.get('model')
	if actual_model is not None and hasattr(actual_model, 'get_params'):
	actual_model_params = actual_model.get_params()
	actual_model_class = type(actual_model).__name__
	logger.info(f"Extracted hyperparameters from {actual_model_class}: {list(actual_model_params.keys())}")

	# Extract preprocessing objects
	production_engineer = model_state.get('production_engineer')
	scaler = model_state.get('scaler')
	label_encoders = model_state.get('label_encoders', {})
	target_encoder = model_state.get('target_encoder')

	except Exception as e:
	logger.warning(f"Could not load model for hyperparameter extraction: {e}")

	# Extract REAL training info from metadata
	target_column = metadata.get('target_column', 'target')
	feature_columns = metadata.get('feature_columns', [])
	task_type = metadata.get('task_type', 'classification')
	best_mode = metadata.get('best_mode', 'traditional')
	modes_trained = metadata.get('modes_trained', ['traditional'])
	best_overall = metadata.get('best_overall') or {}
	best_model_name = best_overall.get('name', metadata.get('model_name', 'Unknown'))
	best_metrics = best_overall.get('metrics', metadata.get('metrics', {}))
	feature_metadata = metadata.get('feature_metadata') or []
	data_summary = metadata.get('data_summary') or {}
	results_per_mode = metadata.get('results_per_mode') or {}
	leaderboard = metadata.get('leaderboard') or []
	nlp_text_column = metadata.get('primary_text_col', metadata.get('nlp_text_column', metadata.get('text_column', '')))

	# Classify features by type from feature_metadata
	numeric_features = []
	categorical_features = []
	text_features = []
	for fm in feature_metadata:
	name = fm.get('name', '')
	ftype = fm.get('type', 'numeric')
	if ftype == 'numeric':
	numeric_features.append(fm)
	elif ftype == 'categorical':
	categorical_features.append(fm)
	elif ftype == 'text':
	text_features.append(fm)

	# Build config object with ALL real data INCLUDING actual model params
	config = {
	"target_column": target_column,
	"feature_columns": feature_columns,
	"task_type": task_type,
	"best_mode": best_mode,
	"modes_trained": modes_trained,
	"best_model_name": best_model_name,
	"metrics": _sanitize_for_json(best_metrics),
	"data_summary": _sanitize_for_json(data_summary),
	"feature_metadata": _sanitize_for_json(feature_metadata),
	"numeric_features": _sanitize_for_json(numeric_features),
	"categorical_features": _sanitize_for_json(categorical_features),
	"text_features": _sanitize_for_json(text_features),
	"nlp_text_column": nlp_text_column,
	"results_per_mode": _sanitize_for_json(results_per_mode),
	"leaderboard": _sanitize_for_json(leaderboard),
	# NEW: Actual model hyperparameters from trained model
	"actual_model_params": _sanitize_for_json(actual_model_params),
	"actual_model_class": actual_model_class,
	"has_production_engineer": production_engineer is not None,
	}

	is_classification = 'classif' in task_type.lower()

	logger.info(f"[ZIP Gen] best_mode={best_mode}, modes_trained={modes_trained}, best_model={best_model_name}")

	with zipfile.ZipFile(buf, 'w', zipfile.ZIP_DEFLATED) as zf:
	# 1. Add standalone model.pkl (stripped of backend class dependencies)
	# This removes ProductionFeatureEngineer and other backend-specific
	# objects that would cause "ModuleNotFoundError: No module named 'ml'"
	# when users run the scripts outside of DataVision.
	if model_path and model_path.exists():
	if model_state is not None:
	# Create clean pkl with ONLY standard sklearn/numpy objects
	standalone_state = {}
	safe_keys = [
	'model', 'model_name', 'task_type', 'task_type_simple',
	'n_classes', 'target_column', 'feature_columns',
	'numeric_cols', 'categorical_cols', 'text_cols',
	'label_encoders', 'target_encoder', 'scaler',
	'numeric_fill_values', 'feature_metadata',
	'metrics', 'confusion_matrix',
	'y_test', 'y_pred', 'y_proba',
	'production_mode', 'mode',
	'is_nlp_task', 'primary_text_col',
	]
	for key in safe_keys:
	if key in model_state:
	standalone_state[key] = model_state[key]

	clean_buf = io.BytesIO()
	pickle.dump(standalone_state, clean_buf)
	clean_bytes = clean_buf.getvalue()
	zf.writestr('model.pkl', clean_bytes)
	logger.info(f"Added standalone model.pkl ({len(clean_bytes) / 1024:.1f} KB, stripped backend deps)")
	else:
	# Fallback: copy raw model.pkl if we couldn't load it to clean
	zf.write(model_path, 'model.pkl')
	logger.info(f"Added model.pkl ({model_path.stat().st_size / 1024:.1f} KB) [WARNING: may contain backend deps]")

	# 1b. Extract and save production_engineer state as standalone pkl
	# This contains all the fitted transformers (sklearn objects)
	# needed to replicate the EXACT preprocessing pipeline
	if production_engineer is not None:
	try:
	eng_state = {
	'transformers': production_engineer.transformers,
	'encoders': production_engineer.encoders,
	'feature_names': production_engineer.feature_names,
	'original_columns': production_engineer.original_columns,
	'selected_feature_indices': getattr(production_engineer, 'selected_feature_indices', None),
	'is_fitted': True,
	'mode': getattr(production_engineer, 'mode', 'fast'),
	}
	eng_buf = io.BytesIO()
	pickle.dump(eng_state, eng_buf)
	eng_bytes = eng_buf.getvalue()
	zf.writestr('feature_engineer.pkl', eng_bytes)
	logger.info(f"Added feature_engineer.pkl ({len(eng_bytes) / 1024:.1f} KB) — exact preprocessing pipeline")
	except Exception as e:
	logger.warning(f"Could not serialize feature_engineer: {e}")

	# 1c. Add deep_learning_model.pkl if DL was trained
	if dl_model_path and dl_model_path.exists():
	try:
	with open(dl_model_path, 'rb') as f:
	dl_state = pickle.load(f)
	# Create clean DL pkl with only standard sklearn/numpy objects
	dl_safe_keys = [
	'model', 'scaler', 'label_encoder', 'feature_columns',
	'numeric_cols', 'categorical_cols', 'feature_metadata',
	'target_column', 'algorithm', 'task_type', 'classes',
	'metrics', 'training_history', 'model_type',
	'y_test', 'y_pred', 'y_proba',
	]
	dl_clean = {}
	for key in dl_safe_keys:
	if key in dl_state:
	dl_clean[key] = dl_state[key]
	dl_buf = io.BytesIO()
	pickle.dump(dl_clean, dl_buf)
	dl_bytes = dl_buf.getvalue()
	zf.writestr('deep_learning_model.pkl', dl_bytes)
	logger.info(f"Added deep_learning_model.pkl ({len(dl_bytes) / 1024:.1f} KB)")
	except Exception as e:
	logger.warning(f"Could not bundle deep_learning_model.pkl: {e}")
	elif 'deep_learning' in modes_trained:
	# Try to find DL model automatically in multiple directories
	auto_dl_path = None
	if model_path and model_path.exists():
	# Search in model dir, plus common storage locations
	search_dirs = [model_path.parent]
	# Add storage/users, storage/automl, storage/models directories
	storage_base = model_path.parent.parent # Go up from user_id dir to storage base
	for subdir in ['users', 'automl', 'models', 'files']:
	candidate_dir = storage_base / subdir
	if not candidate_dir.exists():
	# Try from parent of storage_base (in case we're at storage/automl/user_id)
	candidate_dir = storage_base.parent / subdir
	if candidate_dir.exists():
	# Find user_id from the original path
	user_id_dir = model_path.parent.name
	candidate = candidate_dir / user_id_dir / "deep_learning_model.pkl"
	if candidate.exists():
	auto_dl_path = candidate
	break

	if auto_dl_path is None:
	for sd in search_dirs:
	candidate = sd / "deep_learning_model.pkl"
	if candidate.exists():
	auto_dl_path = candidate
	break

	if auto_dl_path and auto_dl_path.exists():
	try:
	with open(auto_dl_path, 'rb') as f:
	dl_state = pickle.load(f)
	dl_safe_keys = [
	'model', 'scaler', 'label_encoder', 'feature_columns',
	'numeric_cols', 'categorical_cols', 'feature_metadata',
	'target_column', 'algorithm', 'task_type', 'classes',
	'metrics', 'training_history', 'model_type',
	'y_test', 'y_pred', 'y_proba',
	]
	dl_clean = {}
	for key in dl_safe_keys:
	if key in dl_state:
	dl_clean[key] = dl_state[key]
	dl_buf = io.BytesIO()
	pickle.dump(dl_clean, dl_buf)
	dl_bytes = dl_buf.getvalue()
	zf.writestr('deep_learning_model.pkl', dl_bytes)
	logger.info(f"Added deep_learning_model.pkl ({len(dl_bytes) / 1024:.1f} KB) [auto-detected]")
	except Exception as e:
	logger.warning(f"Could not bundle auto-detected deep_learning_model.pkl: {e}")

	# 1d. Bundle nlp_model.pkl if NLP was trained (separate from model.pkl)
	if 'nlp' in modes_trained and model_path and model_path.exists():
	nlp_model_path = None
	# Search multiple directories for nlp_model.pkl
	user_id_dir = model_path.parent.name
	storage_base = model_path.parent.parent
	nlp_search = [
	model_path.parent / "nlp_model.pkl",
	]
	for subdir in ['users', 'automl', 'models', 'files']:
	for base in [storage_base, storage_base.parent]:
	candidate = base / subdir / user_id_dir / "nlp_model.pkl"
	if candidate not in nlp_search:
	nlp_search.append(candidate)

	for np_path in nlp_search:
	if np_path.exists():
	nlp_model_path = np_path
	break

	if nlp_model_path:
	try:
	zf.write(nlp_model_path, 'nlp_model.pkl')
	logger.info(f"Added nlp_model.pkl from {nlp_model_path}")
	except Exception as e:
	logger.warning(f"Could not bundle nlp_model.pkl: {e}")

	# 2. Add cleaned data
	if cleaned_data_path and cleaned_data_path.exists():
	zf.write(cleaned_data_path, 'data/cleaned_data.csv')
	logger.info(f"Added data/cleaned_data.csv")

	# 3. config.json with full real metadata
	zf.writestr('config.json', json.dumps(config, indent=2, default=str))

	# 4. ALWAYS: predict.py (uses the actual saved model)
	zf.writestr('predict.py', _generate_predict_script(config))

	# 5. ALWAYS: visualize.py (generates charts matching what user saw)
	zf.writestr('visualize.py', _generate_visualize_script(config))

	# 6. ALWAYS: evaluate.py
	zf.writestr('evaluate.py', _generate_evaluate_script(config))

	# 7. ONLY include training scripts for modes the user actually trained
	if 'traditional' in modes_trained:
	trad_result = results_per_mode.get('traditional') or {}
	zf.writestr('train_traditional.py', _generate_traditional_train_script(config, trad_result))
	logger.info("Added train_traditional.py")

	if 'nlp' in modes_trained and nlp_text_column:
	nlp_result = results_per_mode.get('nlp') or {}
	zf.writestr('train_nlp.py', _generate_nlp_train_script(config, nlp_result))
	logger.info("Added train_nlp.py")

	if 'deep_learning' in modes_trained:
	dl_result = results_per_mode.get('deep_learning') or {}
	zf.writestr('train_deep_learning.py', _generate_deep_learning_script(config, dl_result))
	logger.info("Added train_deep_learning.py")

	# 8. api_server.py for deployment
	zf.writestr('api_server.py', _generate_api_server_script(config))
	zf.writestr('templates/index.html', API_SERVER_HTML_TEMPLATE)

	# 9. Utilities
	zf.writestr('utils/__init__.py', '')
	zf.writestr('utils/preprocessing.py', _generate_preprocessing_utils(config))

	# 10. requirements.txt (only deps needed for actual modes)
	zf.writestr('requirements.txt', _generate_requirements(config))

	# 11. README.md (documents what was actually trained)
	zf.writestr('README.md', _generate_readme(config))

	# 12. Dockerfile
	zf.writestr('Dockerfile', _generate_dockerfile(config))

	# 13. Bundle actual training charts as PNG images
	# These are the EXACT charts the user saw in DataVision AutoML
	if charts_data:
	charts_added = 0
	for chart_name, chart_value in charts_data.items():
	try:
	# Charts are stored as "data:image/png;base64,..." data URIs
	if isinstance(chart_value, str) and 'base64,' in chart_value:
	b64_data = chart_value.split('base64,', 1)[1]
	img_bytes = base64.b64decode(b64_data)
	# Clean chart name for filename
	safe_name = chart_name.replace(' ', '_').replace('/', '_').lower()
	if not safe_name.endswith('.png'):
	safe_name += '.png'
	zf.writestr(f'charts/{safe_name}', img_bytes)
	charts_added += 1
	except Exception as e:
	logger.warning(f"Could not bundle chart '{chart_name}': {e}")
	if charts_added > 0:
	logger.info(f"Added {charts_added} training charts to charts/ directory")

	buf.seek(0)
	logger.info(f"Generated ZIP: {buf.getbuffer().nbytes / 1024:.1f} KB")
	return buf


	def _sanitize_for_json(obj):
	"""Make object JSON-serializable"""
	if isinstance(obj, dict):
	return {k: _sanitize_for_json(v) for k, v in obj.items()}
	elif isinstance(obj, (list, tuple)):
	return [_sanitize_for_json(v) for v in obj]
	elif isinstance(obj, float):
	if obj != obj: # NaN
	return None
	return obj
	else:
	try:
	json.dumps(obj)
	return obj
	except (TypeError, ValueError):
	return str(obj)


	def _params_to_python_literal(params: Dict) -> str:
	"""Convert model parameters to valid Python code literal.

	Handles numpy types, None, strings, etc. to produce code like:
	{'n_estimators': 100, 'max_depth': 10, 'random_state': 42}
	"""
	import numpy as np

	def _convert_value(v):
	"""Convert a single value to Python literal representation"""
	if v is None:
	return 'None'
	elif isinstance(v, bool):
	return 'True' if v else 'False'
	elif isinstance(v, (int, np.integer)):
	return str(int(v))
	elif isinstance(v, (float, np.floating)):
	if np.isnan(v):
	return 'None'
	elif np.isinf(v):
	return 'float("inf")' if v > 0 else 'float("-inf")'
	return repr(float(v))
	elif isinstance(v, str):
	return repr(v)
	elif isinstance(v, (list, tuple, np.ndarray)):
	items = [_convert_value(item) for item in v]
	if isinstance(v, tuple):
	return f"({', '.join(items)})"
	return f"[{', '.join(items)}]"
	elif isinstance(v, dict):
	items = [f"{repr(k)}: {_convert_value(val)}" for k, val in v.items()]
	return '{' + ', '.join(items) + '}'
	elif callable(v):
	# Skip callable parameters (like loss functions)
	return None
	else:
	# Try to convert to string
	try:
	return repr(str(v))
	except:
	return None

	# Filter and convert parameters
	result_items = []
	skip_params = {
	'oob_score', 'warm_start', 'verbose', 'callbacks', 'early_stopping_rounds',
	# XGBoost internal/deprecated params
	'use_label_encoder', 'gpu_id', 'predictor', 'enable_categorical',
	'feature_types', 'max_cat_to_onehot', 'max_cat_threshold',
	'multi_strategy', 'updater', 'refresh_leaf', 'process_type',
	'grow_policy', 'max_leaves', 'max_bin', 'num_parallel_tree',
	'monotone_constraints', 'interaction_constraints',
	'validate_parameters', 'nthread', 'device',
	# LightGBM internal params
	'silent', 'importance_type', 'n_jobs',
	# CatBoost internal params
	'cat_features', 'text_features', 'embedding_features',
	'task_type', 'devices', 'bootstrap_type',
	}

	for k, v in params.items():
	if k in skip_params:
	continue
	if k.startswith('_'): # Skip private params
	continue
	if callable(v):
	continue
	converted = _convert_value(v)
	if converted is not None:
	result_items.append(f"'{k}': {converted}")

	return '{' + ', '.join(result_items) + '}'


	# =============================================================================
	# TRADITIONAL ML TRAINING SCRIPT - DATA-AWARE
	# =============================================================================

	def _generate_traditional_train_script(config: Dict, mode_result: Dict) -> str:
	"""Generate training script that mirrors the REAL DataVision AutoML pipeline.

	Uses the EXACT hyperparameters extracted from the trained model to ensure
	retraining produces the SAME results as AutoML.
	"""
	target = config['target_column']
	task_type = config['task_type']
	features = config.get('feature_columns', [])
	numeric_features = config.get('numeric_features', [])
	categorical_features = config.get('categorical_features', [])
	leaderboard = config.get('leaderboard', [])
	best_model_name = mode_result.get('best_model', mode_result.get('algorithm', config.get('best_model_name', 'Unknown')))
	best_metrics = mode_result.get('metrics', config.get('metrics', {}))

	# CRITICAL: Get ACTUAL hyperparameters from the trained model
	actual_model_params = config.get('actual_model_params', {})
	actual_model_class = config.get('actual_model_class', best_model_name)

	is_clf = 'classif' in task_type.lower()

	# Build the list of ACTUAL algorithms from leaderboard
	actual_algorithms = []
	for entry in leaderboard[:15]:
	algo_name = entry.get('name', entry.get('model_name', ''))
	if algo_name:
	actual_algorithms.append(algo_name)
	if not actual_algorithms:
	actual_algorithms = [best_model_name]

	# Feature info comment
	feature_info_lines = []
	for fm in numeric_features[:10]:
	name = fm.get('name', '')
	mn = fm.get('min', '?')
	mx = fm.get('max', '?')
	mean = fm.get('mean', '?')
	feature_info_lines.append(f"# {name}: numeric (min={mn}, max={mx}, mean={mean})")
	for fm in categorical_features[:10]:
	name = fm.get('name', '')
	options = fm.get('options', [])
	feature_info_lines.append(f"# {name}: categorical ({', '.join(str(o) for o in options[:5])}{'...' if len(options) > 5 else ''})")
	feature_info_block = "\n".join(feature_info_lines) if feature_info_lines else "# (no feature metadata available)"

	# Metrics comment
	metrics_lines = []
	for k, v in best_metrics.items():
	if isinstance(v, float):
	metrics_lines.append(f"# {k}: {v:.4f}")
	else:
	metrics_lines.append(f"# {k}: {v}")
	metrics_block = "\n".join(metrics_lines) if metrics_lines else "# (no metrics available)"

	# Leaderboard comment
	leaderboard_lines = []
	for i, entry in enumerate(leaderboard[:10]):
	name = entry.get('name', entry.get('model_name', '?'))
	score = entry.get('score', entry.get('accuracy', entry.get('r2', '?')))
	if isinstance(score, float):
	leaderboard_lines.append(f"# {i+1}. {name}: {score:.4f}")
	else:
	leaderboard_lines.append(f"# {i+1}. {name}: {score}")
	leaderboard_block = "\n".join(leaderboard_lines) if leaderboard_lines else "# (no leaderboard data)"

	features_list = ", ".join(f"'{f}'" for f in features[:30])

	# Detect which optional packages are needed
	algo_lower_list = [a.lower() for a in actual_algorithms]
	algo_joined = " ".join(algo_lower_list)
	needs_xgb = any('xgb' in a or 'xgboost' in a for a in algo_lower_list)
	needs_lgb = any('lgb' in a or 'lightgbm' in a or 'lgbm' in a for a in algo_lower_list)
	needs_catboost = any('catboost' in a or 'cat_boost' in a or 'cat boost' in a for a in algo_lower_list)

	# Determine mode (fast vs ultra) from number of algorithms trained
	is_ultra = len(actual_algorithms) > 10

	metric_name = 'F1-Macro' if is_clf else 'R²'

	script = f'''"""
	🚀 Traditional ML Training Script — YOUR DataVision AutoML Results
	=====================================================================
	Auto-generated by DataVision AI

	╔═══════════════════════════════════════════════════════════════════════╗
	║ 📦 model.pkl contains your ALREADY TRAINED model with the metrics ║
	║ you saw in the AutoML UI. ║
	║ ║
	║ ▶ python evaluate.py — See your EXACT AutoML results ║
	║ ▶ python predict.py — Make predictions with your model ║
	║ ║
	║ This script is for RETRAINING only. Results may differ slightly ║
	║ due to train/test random split, even with same hyperparameters. ║
	╚═══════════════════════════════════════════════════════════════════════╝

	YOUR AUTOML TRAINING RESULTS:
	━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
	🏆 WINNING MODEL : {best_model_name}
	🎯 Target : {target}
	📊 Task : {task_type}
	📁 Features : {len(features)} columns

	YOUR METRICS:
	{metrics_block}

	YOUR LEADERBOARD (what AutoML trained):
	{leaderboard_block}

	FEATURE INFO:
	{feature_info_block}

	Usage:
	# VERIFY YOUR AUTOML RESULTS:
	python evaluate.py # Shows exact AutoML metrics

	# MAKE PREDICTIONS:
	python predict.py --json '{{"col1": value}}' # Your trained model

	# RETRAIN (if needed):
	python train_traditional.py # Train ONLY {best_model_name}
	python train_traditional.py --all # Train ALL algorithms
	"""

	import os
	import pickle
	import argparse
	import time
	import warnings
	import numpy as np
	import pandas as pd
	from scipy import stats as scipy_stats
	from sklearn.model_selection import train_test_split, StratifiedKFold, KFold, cross_val_score
	from sklearn.preprocessing import LabelEncoder, RobustScaler, StandardScaler
	from sklearn.feature_selection import VarianceThreshold
	from sklearn.impute import KNNImputer
	from sklearn.metrics import (
	accuracy_score, precision_score, recall_score, f1_score,
	r2_score, mean_absolute_error, mean_squared_error,
	classification_report, confusion_matrix
	)
	from sklearn.ensemble import (
	RandomForestClassifier, RandomForestRegressor,
	GradientBoostingClassifier, GradientBoostingRegressor,
	ExtraTreesClassifier, ExtraTreesRegressor,
	HistGradientBoostingClassifier, HistGradientBoostingRegressor,
	AdaBoostClassifier, AdaBoostRegressor,
	BaggingClassifier, BaggingRegressor,
	)
	from sklearn.linear_model import (
	LogisticRegression, Ridge, Lasso, ElasticNet,
	SGDClassifier, SGDRegressor, RidgeClassifier,
	PassiveAggressiveClassifier, PassiveAggressiveRegressor,
	BayesianRidge
	)
	from sklearn.svm import SVC, SVR
	from sklearn.neighbors import KNeighborsClassifier, KNeighborsRegressor
	from sklearn.naive_bayes import GaussianNB, BernoulliNB
	from sklearn.neural_network import MLPClassifier, MLPRegressor
	from sklearn.tree import DecisionTreeClassifier, DecisionTreeRegressor

	warnings.filterwarnings('ignore')

	HAS_XGB = False
	HAS_LGB = False
	HAS_CB = False
	HAS_SMOTE = False

	try:
	import xgboost as xgb
	HAS_XGB = True
	except ImportError:
	pass
	try:
	import lightgbm as lgb
	HAS_LGB = True
	except ImportError:
	pass
	try:
	from catboost import CatBoostClassifier, CatBoostRegressor
	HAS_CB = True
	except ImportError:
	pass
	try:
	from imblearn.over_sampling import SMOTE
	HAS_SMOTE = True
	except ImportError:
	pass

	TARGET_COLUMN = '{target}'
	TASK_TYPE = '{task_type}'

	# YOUR WINNING MODEL FROM AUTOML
	BEST_MODEL_NAME = '{best_model_name}'


	# ============================================================================
	# STEP 1: DATA CLEANING (mirrors ProductionDataCleaner.clean())
	# ============================================================================

	def clean_data(df):
	"""
	Mirrors DataVision's ProductionDataCleaner pipeline:
	1. Remove useless columns (IDs, URLs, high-cardinality strings)
	2. Remove exact duplicates
	3. Strip whitespace, replace 'nan'/'None'/'' with NaN
	4. Smart type conversion (parse $, M/K units, commas)
	5. KNN imputation for numerics (median fallback)
	6. Mode fill for categoricals
	7. Replace inf/-inf with median
	8. Remove constant columns (nunique <= 1)
	9. IQR-based outlier capping (Q1-3IQR to Q3+3IQR, clipped to [1st, 99th] percentile)
	10. Final NaN fill to 0
	"""
	print("\\n🧹 Step 1: Data Cleaning (ProductionDataCleaner)")
	print("-" * 50)
	original_shape = df.shape

	# Remove useless columns
	id_patterns = ['unnamed', 'index', '_id', 'guid', 'uuid', 'url', 'link', 'href', 'path']
	cols_to_drop = []
	for col in df.columns:
	col_lower = col.lower().strip()
	if any(pat in col_lower for pat in id_patterns):
	if col != TARGET_COLUMN:
	cols_to_drop.append(col)
	elif df[col].dtype == 'object' and col != TARGET_COLUMN:
	unique_ratio = df[col].nunique() / max(len(df), 1)
	if unique_ratio > 0.9:
	cols_to_drop.append(col)
	if cols_to_drop:
	df = df.drop(columns=cols_to_drop, errors='ignore')
	print(f" Dropped {{len(cols_to_drop)}} useless columns: {{cols_to_drop[:5]}}")

	# Remove exact duplicates
	before = len(df)
	df = df.drop_duplicates()
	if len(df) < before:
	print(f" Removed {{before - len(df)}} duplicate rows")

	# Strip whitespace + replace string NaN variants
	for col in df.select_dtypes(include=['object']).columns:
	df[col] = df[col].str.strip()
	df[col] = df[col].replace(
	['nan', 'NaN', 'None', 'null', 'NULL', 'none', 'NA', 'N/A', ''],
	np.nan
	)

	# Smart type conversion (parse $, M/K, commas)
	for col in df.select_dtypes(include=['object']).columns:
	if col == TARGET_COLUMN:
	continue
	sample = df[col].dropna().head(20)
	if len(sample) == 0:
	continue
	converted = 0
	for val in sample:
	try:
	v = str(val).replace('$', '').replace(',', '').strip()
	if v.upper().endswith('M'):
	float(v[:-1])
	elif v.upper().endswith('K'):
	float(v[:-1])
	elif v.endswith('+'):
	float(v[:-1])
	else:
	float(v)
	converted += 1
	except:
	pass
	if converted / max(len(sample), 1) > 0.6:
	def parse_value(val):
	if pd.isna(val):
	return np.nan
	v = str(val).replace('$', '').replace(',', '').strip()
	try:
	if v.upper().endswith('M'):
	return float(v[:-1]) * 1_000_000
	elif v.upper().endswith('K'):
	return float(v[:-1]) * 1_000
	elif v.endswith('+'):
	return float(v[:-1])
	return float(v)
	except:
	return np.nan
	df[col] = df[col].apply(parse_value)
	print(f" Converted {{col}} to numeric (parsed special formats)")

	# KNN imputation for numeric columns
	numeric_cols = df.select_dtypes(include=[np.number]).columns.tolist()
	if TARGET_COLUMN in numeric_cols:
	numeric_cols.remove(TARGET_COLUMN)
	if numeric_cols and df[numeric_cols].isna().any().any():
	try:
	imputer = KNNImputer(n_neighbors=5, weights='distance')
	df[numeric_cols] = imputer.fit_transform(df[numeric_cols])
	print(f" KNN imputed {{len(numeric_cols)}} numeric columns")
	except:
	for col in numeric_cols:
	df[col] = df[col].fillna(df[col].median())
	print(f" Median-filled {{len(numeric_cols)}} numeric columns (KNN fallback)")

	# Mode fill for categoricals
	for col in df.select_dtypes(include=['object', 'category']).columns:
	if col == TARGET_COLUMN:
	continue
	mode = df[col].mode()
	df[col] = df[col].fillna(mode.iloc[0] if len(mode) > 0 else '_MISSING_')

	# Replace inf/-inf with median
	for col in numeric_cols:
	if np.isinf(df[col]).any():
	med = df[col].replace([np.inf, -np.inf], np.nan).median()
	df[col] = df[col].replace([np.inf, -np.inf], med)

	# Remove constant columns
	const_cols = [col for col in df.columns if col != TARGET_COLUMN and df[col].nunique() <= 1]
	if const_cols:
	df = df.drop(columns=const_cols)
	print(f" Removed {{len(const_cols)}} constant columns")

	# IQR-based outlier capping (same as DataVision)
	capped_count = 0
	for col in df.select_dtypes(include=[np.number]).columns:
	if col == TARGET_COLUMN:
	continue
	Q1 = df[col].quantile(0.25)
	Q3 = df[col].quantile(0.75)
	IQR = Q3 - Q1
	p1 = df[col].quantile(0.01)
	p99 = df[col].quantile(0.99)
	lower = max(p1, Q1 - 3 * IQR)
	upper = min(p99, Q3 + 3 * IQR)
	before_clip = ((df[col] < lower) \| (df[col] > upper)).sum()
	if before_clip > 0:
	df[col] = df[col].clip(lower=lower, upper=upper)
	capped_count += before_clip
	if capped_count:
	print(f" Capped {{capped_count}} outlier values (IQR method)")

	# Final NaN fill
	for col in df.select_dtypes(include=[np.number]).columns:
	df[col] = df[col].fillna(0)

	print(f" Clean: {{original_shape}} → {{df.shape}}")
	return df


	# ============================================================================
	# STEP 2: FEATURE ENGINEERING (mirrors ProductionFeatureEngineer)
	# ============================================================================

	def engineer_features(X, mode='fast'):
	"""
	Mirrors DataVision's ProductionFeatureEngineer.fit_transform():
	1. Detect column types (numeric, categorical, text, datetime)
	2. Log-transform skewed numerics (skewness > 2)
	3. RobustScaler for numeric features
	4. Polynomial features (squared + interactions for top columns)
	5. Categorical: binary→LabelEncoder, low-card→one-hot, high-card→label+frequency
	6. VarianceThreshold (remove near-constant features, threshold=0.01)
	"""
	print("\\n🔧 Step 2: Feature Engineering (ProductionFeatureEngineer)")
	print("-" * 50)

	numeric_cols = X.select_dtypes(include=[np.number]).columns.tolist()
	categorical_cols = X.select_dtypes(include=['object', 'category']).columns.tolist()

	label_encoders = {{}}

	# --- Numeric feature engineering ---
	# Log transform skewed features (skewness > 2, non-negative only)
	log_transformed = []
	for col in numeric_cols:
	if (X[col] >= 0).all():
	skew = X[col].skew()
	if abs(skew) > 2:
	X[f'{{col}}_log'] = np.log1p(X[col])
	log_transformed.append(col)
	if log_transformed:
	print(f" Log-transformed {{len(log_transformed)}} skewed features")

	# Scale with RobustScaler (same as DataVision)
	numeric_cols_updated = X.select_dtypes(include=[np.number]).columns.tolist()
	scaler = RobustScaler()
	if numeric_cols_updated:
	X[numeric_cols_updated] = scaler.fit_transform(X[numeric_cols_updated])

	# Polynomial features (top 4 numeric columns — squared + interactions)
	if 2 <= len(numeric_cols) <= 10:
	top_n = min(4, len(numeric_cols))
	top_cols = numeric_cols[:top_n]
	for col in top_cols:
	X[f'{{col}}_sq'] = X[col] ** 2
	for i in range(len(top_cols)):
	for j in range(i + 1, len(top_cols)):
	X[f'{{top_cols[i]}}_x_{{top_cols[j]}}'] = X[top_cols[i]] * X[top_cols[j]]
	print(f" Added polynomial features for top {{top_n}} numeric columns")

	# --- Categorical feature engineering ---
	for col in categorical_cols:
	nuniq = X[col].nunique()
	if nuniq <= 2:
	# Binary: LabelEncoder
	le = LabelEncoder()
	X[col] = le.fit_transform(X[col].astype(str))
	label_encoders[col] = le
	elif nuniq <= 10:
	# Low cardinality: one-hot encoding (no drop_first — same as DataVision)
	dummies = pd.get_dummies(X[col], prefix=col, drop_first=False)
	X = pd.concat([X.drop(columns=[col]), dummies], axis=1)
	else:
	# High cardinality: LabelEncoder + frequency encoding
	le = LabelEncoder()
	X[f'{{col}}_label'] = le.fit_transform(X[col].astype(str))
	freq = X[col].value_counts(normalize=True)
	X[f'{{col}}_freq'] = X[col].map(freq)
	label_encoders[col] = le
	X = X.drop(columns=[col])

	# Force all to float
	for col in X.columns:
	try:
	X[col] = X[col].astype(float)
	except:
	le = LabelEncoder()
	X[col] = le.fit_transform(X[col].astype(str)).astype(float)

	# Handle NaN/Inf
	X = X.replace([np.inf, -np.inf], 0)
	X = X.fillna(0)

	# VarianceThreshold (remove near-constant features)
	try:
	vt = VarianceThreshold(threshold=0.01)
	cols_before = X.columns.tolist()
	X_filtered = pd.DataFrame(vt.fit_transform(X), columns=X.columns[vt.get_support()], index=X.index)
	removed = len(cols_before) - X_filtered.shape[1]
	if removed > 0:
	print(f" VarianceThreshold removed {{removed}} near-constant features")
	X = X_filtered
	except:
	pass

	print(f" Final features: {{X.shape[1]}}")
	return X, label_encoders, scaler


	# ============================================================================
	# STEP 3: YOUR EXACT TRAINED MODEL
	# ============================================================================

	# These are the EXACT hyperparameters from YOUR trained model
	ACTUAL_MODEL_CLASS = '{actual_model_class}'
	ACTUAL_MODEL_PARAMS = {_params_to_python_literal(actual_model_params) if actual_model_params else '{}'}

	def get_exact_trained_model():
	"""Returns YOUR exact model with the SAME hyperparameters as AutoML used.

	This ensures retraining produces the SAME results as your AutoML session.
	"""
	params = ACTUAL_MODEL_PARAMS.copy()

	# Remove non-constructor / problematic params that cause crashes
	for bad_key in ['oob_score', 'warm_start', 'verbose', 'callbacks',
	'early_stopping_rounds', 'use_label_encoder', 'gpu_id',
	'predictor', 'enable_categorical', 'feature_types',
	'validate_parameters', 'nthread', 'device',
	'n_jobs', 'silent', 'importance_type',
	'cat_features', 'text_features', 'task_type',
	'multi_strategy', 'max_cat_to_onehot', 'max_cat_threshold',
	'num_parallel_tree', 'monotone_constraints',
	'interaction_constraints', 'grow_policy', 'max_leaves',
	'process_type', 'updater', 'refresh_leaf', 'max_bin',
	'bootstrap_type', 'devices', 'embedding_features']:
	params.pop(bad_key, None)
	# Remove private params
	params = {{k: v for k, v in params.items() if not k.startswith('_')}}

	'''

	# Build the model instantiation based on actual class
	if actual_model_class == 'RandomForestClassifier':
	script += f'''
	return {{'RandomForest': RandomForestClassifier(**params)}}
	'''
	elif actual_model_class == 'RandomForestRegressor':
	script += f'''
	return {{'RandomForest': RandomForestRegressor(**params)}}
	'''
	elif actual_model_class == 'GradientBoostingClassifier':
	script += f'''
	return {{'GradientBoosting': GradientBoostingClassifier(**params)}}
	'''
	elif actual_model_class == 'GradientBoostingRegressor':
	script += f'''
	return {{'GradientBoosting': GradientBoostingRegressor(**params)}}
	'''
	elif actual_model_class == 'XGBClassifier':
	script += f'''
	if HAS_XGB:
	# Clean XGBoost params for version compatibility
	xgb_safe = {{k: v for k, v in params.items() if k in {{
	'n_estimators', 'max_depth', 'learning_rate', 'subsample',
	'colsample_bytree', 'colsample_bylevel', 'colsample_bynode',
	'reg_alpha', 'reg_lambda', 'gamma', 'min_child_weight',
	'scale_pos_weight', 'base_score', 'random_state', 'seed',
	'objective', 'booster', 'tree_method', 'eval_metric',
	}}}}
	xgb_safe['verbosity'] = 0
	xgb_safe['n_jobs'] = -1
	return {{'XGBoost': xgb.XGBClassifier(**xgb_safe)}}
	else:
	print("⚠️ XGBoost not installed, using RandomForest")
	return get_models(1000, 'fast')
	'''
	elif actual_model_class == 'XGBRegressor':
	script += f'''
	if HAS_XGB:
	xgb_safe = {{k: v for k, v in params.items() if k in {{
	'n_estimators', 'max_depth', 'learning_rate', 'subsample',
	'colsample_bytree', 'colsample_bylevel', 'colsample_bynode',
	'reg_alpha', 'reg_lambda', 'gamma', 'min_child_weight',
	'scale_pos_weight', 'base_score', 'random_state', 'seed',
	'objective', 'booster', 'tree_method', 'eval_metric',
	}}}}
	xgb_safe['verbosity'] = 0
	xgb_safe['n_jobs'] = -1
	return {{'XGBoost': xgb.XGBRegressor(**xgb_safe)}}
	else:
	print("⚠️ XGBoost not installed, using RandomForest")
	return get_models(1000, 'fast')
	'''
	elif actual_model_class == 'LGBMClassifier':
	script += f'''
	if HAS_LGB:
	return {{'LightGBM': lgb.LGBMClassifier(**params)}}
	else:
	print("⚠️ LightGBM not installed, using RandomForest")
	return get_models(1000, 'fast')
	'''
	elif actual_model_class == 'LGBMRegressor':
	script += f'''
	if HAS_LGB:
	return {{'LightGBM': lgb.LGBMRegressor(**params)}}
	else:
	print("⚠️ LightGBM not installed, using RandomForest")
	return get_models(1000, 'fast')
	'''
	elif actual_model_class == 'CatBoostClassifier':
	script += f'''
	if HAS_CB:
	params['verbose'] = False
	params['allow_writing_files'] = False
	return {{'CatBoost': CatBoostClassifier(**params)}}
	else:
	print("⚠️ CatBoost not installed, using RandomForest")
	return get_models(1000, 'fast')
	'''
	elif actual_model_class == 'CatBoostRegressor':
	script += f'''
	if HAS_CB:
	params['verbose'] = False
	params['allow_writing_files'] = False
	return {{'CatBoost': CatBoostRegressor(**params)}}
	else:
	print("⚠️ CatBoost not installed, using RandomForest")
	return get_models(1000, 'fast')
	'''
	elif actual_model_class == 'LogisticRegression':
	script += f'''
	return {{'LogisticRegression': LogisticRegression(**params)}}
	'''
	elif actual_model_class in ('HistGradientBoostingClassifier', 'HistGradientBoostingRegressor'):
	if is_clf:
	script += f'''
	return {{'HistGradientBoosting': HistGradientBoostingClassifier(**params)}}
	'''
	else:
	script += f'''
	return {{'HistGradientBoosting': HistGradientBoostingRegressor(**params)}}
	'''
	else:
	# Fallback - try to use it with generic RF params
	script += f'''
	# Model class '{actual_model_class}' - using extracted params with RandomForest as fallback
	try:
	from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor
	if TASK_TYPE == 'classification':
	return {{'RandomForest': RandomForestClassifier(n_estimators=150, max_depth=12, random_state=42, class_weight='balanced', n_jobs=-1)}}
	else:
	return {{'RandomForest': RandomForestRegressor(n_estimators=150, max_depth=12, random_state=42, n_jobs=-1)}}
	except:
	return get_models(1000, 'fast')
	'''

	script += f'''

	def get_single_model(model_name, n_samples):
	"""Get ONLY your winning model.

	Uses get_exact_trained_model() to get YOUR exact hyperparameters.
	"""
	# First try to use exact trained model params
	if model_name == BEST_MODEL_NAME or model_name == '{best_model_name}':
	return get_exact_trained_model()

	# Otherwise fall back to generic models
	is_large = n_samples > 50000
	all_models = get_models(n_samples, mode='fast')

	# Try exact match first
	if model_name in all_models:
	return {{model_name: all_models[model_name]}}

	# Try case-insensitive match
	model_name_lower = model_name.lower().replace('_', '').replace(' ', '')
	for k, v in all_models.items():
	if k.lower().replace('_', '').replace(' ', '') == model_name_lower:
	return {{k: v}}

	# Check in ultra models too
	all_models = get_models(n_samples, mode='ultra')
	if model_name in all_models:
	return {{model_name: all_models[model_name]}}
	for k, v in all_models.items():
	if k.lower().replace('_', '').replace(' ', '') == model_name_lower:
	return {{k: v}}

	# Fallback to exact trained model
	print(f" ⚠️ Model '{{model_name}}' not found, using your trained model")
	return get_exact_trained_model()


	def get_models(n_samples, mode='{"ultra" if is_ultra else "fast"}'):
	"""Returns the EXACT algorithms & hyperparameters used by DataVision AutoML.

	Mirrors ProductionModelTrainer.get_models() with the same conditional
	logic for dataset size and mode.
	"""
	is_large = n_samples > 50000
	is_ultra = mode == 'ultra'

	'''

	if is_clf:
	script += f'''
	models = {{
	'LogisticRegression': LogisticRegression(
	max_iter=2000, C=0.1, random_state=42, n_jobs=1,
	class_weight='balanced'
	),
	'RandomForest': RandomForestClassifier(
	n_estimators=100 if is_large else (150 if not is_ultra else 250),
	max_depth=8 if is_large else (12 if not is_ultra else 18),
	min_samples_split=4, random_state=42, n_jobs=-1,
	class_weight='balanced'
	),
	'GradientBoosting': GradientBoostingClassifier(
	n_estimators=80 if is_large else 120,
	max_depth=5 if is_large else 6,
	learning_rate=0.1, random_state=42, subsample=0.8
	),
	'HistGradientBoosting': HistGradientBoostingClassifier(
	max_iter=100 if is_large else (150 if not is_ultra else 250),
	max_depth=6 if is_large else (8 if not is_ultra else 12),
	learning_rate=0.1, random_state=42, class_weight='balanced'
	),
	'GaussianNB': GaussianNB(),
	'ExtraTrees': ExtraTreesClassifier(
	n_estimators=80 if is_large else (100 if not is_ultra else 200),
	max_depth=8 if is_large else (10 if not is_ultra else 15),
	random_state=42, n_jobs=-1, class_weight='balanced'
	),
	}}

	if not is_large:
	models['KNN'] = KNeighborsClassifier(n_neighbors=7, weights='distance', n_jobs=-1)

	if HAS_XGB:
	models['XGBoost'] = xgb.XGBClassifier(
	n_estimators=100 if is_large else (150 if not is_ultra else 300),
	max_depth=6 if is_large else (7 if not is_ultra else 10),
	learning_rate=0.1 if not is_ultra else 0.05,
	subsample=0.8, colsample_bytree=0.8,
	random_state=42, n_jobs=-1, verbosity=0,
	eval_metric='mlogloss',
	tree_method='hist' if is_large else 'auto',
	scale_pos_weight=10
	)
	if HAS_LGB:
	models['LightGBM'] = lgb.LGBMClassifier(
	n_estimators=100 if is_large else (150 if not is_ultra else 300),
	max_depth=6 if is_large else (7 if not is_ultra else 10),
	learning_rate=0.1 if not is_ultra else 0.05,
	num_leaves=31 if not is_ultra else 50,
	subsample=0.8, colsample_bytree=0.8,
	random_state=42, n_jobs=-1, verbose=-1,
	class_weight='balanced'
	)
	if HAS_CB:
	if is_ultra:
	models['CatBoost'] = CatBoostClassifier(
	iterations=150 if is_large else 300, depth=6 if is_large else 10,
	learning_rate=0.05 if is_large else 0.03,
	l2_leaf_reg=3, random_strength=1, bagging_temperature=0.5,
	random_state=42, verbose=False,
	auto_class_weights='Balanced', allow_writing_files=False
	)
	else:
	models['CatBoost'] = CatBoostClassifier(
	iterations=100, depth=6, learning_rate=0.1,
	random_state=42, verbose=False,
	auto_class_weights='Balanced', allow_writing_files=False
	)

	# Ultra mode adds more algorithms
	if is_ultra:
	if not is_large:
	models['SVM_RBF'] = SVC(kernel='rbf', C=1.0, gamma='scale', probability=True, random_state=42, class_weight='balanced')
	models['SVM_Linear'] = SVC(kernel='linear', C=0.5, probability=True, random_state=42, class_weight='balanced')
	models['RidgeClassifier'] = RidgeClassifier(alpha=1.0, random_state=42, class_weight='balanced')
	models['MLP'] = MLPClassifier(
	hidden_layer_sizes=(128, 64) if is_large else (256, 128, 64, 32),
	max_iter=500 if is_large else 1500,
	learning_rate='adaptive', early_stopping=True,
	batch_size=256 if is_large else 'auto',
	validation_fraction=0.1, random_state=42
	)
	models['AdaBoost'] = AdaBoostClassifier(n_estimators=100 if is_large else 150, learning_rate=0.5, random_state=42)
	models['Bagging'] = BaggingClassifier(
	n_estimators=50 if is_large else 100,
	max_samples=0.5 if is_large else 0.8,
	max_features=0.8, random_state=42, n_jobs=-1
	)
	models['SGD'] = SGDClassifier(loss='log_loss', max_iter=1500, random_state=42, class_weight='balanced', early_stopping=True)
	models['PassiveAggressive'] = PassiveAggressiveClassifier(max_iter=1500, random_state=42, class_weight='balanced')
	models['DecisionTree'] = DecisionTreeClassifier(max_depth=12 if is_large else 18, min_samples_split=3, random_state=42, class_weight='balanced')
	models['BernoulliNB'] = BernoulliNB(alpha=0.5)
	'''
	else: # regression
	script += f'''
	models = {{
	'Ridge': Ridge(alpha=1.0, random_state=42),
	'ElasticNet': ElasticNet(alpha=0.1, l1_ratio=0.5, random_state=42, max_iter=2000),
	'RandomForest': RandomForestRegressor(
	n_estimators=100 if is_large else (150 if not is_ultra else 250),
	max_depth=8 if is_large else (12 if not is_ultra else 18),
	min_samples_split=4, random_state=42, n_jobs=-1
	),
	'GradientBoosting': GradientBoostingRegressor(
	n_estimators=80 if is_large else 120,
	max_depth=5 if is_large else 6,
	learning_rate=0.1, subsample=0.8, random_state=42
	),
	'HistGradientBoosting': HistGradientBoostingRegressor(
	max_iter=100 if is_large else (150 if not is_ultra else 250),
	max_depth=6 if is_large else (8 if not is_ultra else 12),
	learning_rate=0.1, random_state=42
	),
	'ExtraTrees': ExtraTreesRegressor(
	n_estimators=80 if is_large else (100 if not is_ultra else 200),
	max_depth=8 if is_large else (10 if not is_ultra else 15),
	random_state=42, n_jobs=-1
	),
	}}

	if not is_large:
	models['KNN'] = KNeighborsRegressor(n_neighbors=7, weights='distance', n_jobs=-1)

	if HAS_XGB:
	models['XGBoost'] = xgb.XGBRegressor(
	n_estimators=100 if is_large else (150 if not is_ultra else 300),
	max_depth=6 if is_large else (7 if not is_ultra else 10),
	learning_rate=0.1 if not is_ultra else 0.05,
	subsample=0.8, colsample_bytree=0.8,
	tree_method='hist' if is_large else 'auto',
	random_state=42, n_jobs=-1, verbosity=0
	)
	if HAS_LGB:
	models['LightGBM'] = lgb.LGBMRegressor(
	n_estimators=100 if is_large else (150 if not is_ultra else 300),
	max_depth=6 if is_large else (7 if not is_ultra else 10),
	learning_rate=0.1 if not is_ultra else 0.05,
	num_leaves=31 if not is_ultra else 50,
	subsample=0.8, colsample_bytree=0.8,
	random_state=42, n_jobs=-1, verbose=-1
	)
	if HAS_CB:
	if is_ultra:
	models['CatBoost'] = CatBoostRegressor(
	iterations=300, depth=10, learning_rate=0.03,
	l2_leaf_reg=3, random_strength=1, bagging_temperature=0.5,
	random_state=42, verbose=False, allow_writing_files=False
	)
	else:
	models['CatBoost'] = CatBoostRegressor(
	iterations=100, depth=6, learning_rate=0.1,
	random_state=42, verbose=False, allow_writing_files=False
	)

	# Ultra mode adds more algorithms
	if is_ultra:
	models['Lasso'] = Lasso(alpha=0.01, random_state=42, max_iter=2000)
	models['BayesianRidge'] = BayesianRidge()
	models['MLP'] = MLPRegressor(
	hidden_layer_sizes=(128, 64) if is_large else (256, 128, 64, 32),
	max_iter=500 if is_large else 1500,
	batch_size=256 if is_large else 'auto',
	learning_rate='adaptive', early_stopping=True,
	validation_fraction=0.1, random_state=42
	)
	models['AdaBoost'] = AdaBoostRegressor(n_estimators=100 if is_large else 150, learning_rate=0.5, random_state=42)
	models['Bagging'] = BaggingRegressor(
	n_estimators=50 if is_large else 100,
	max_samples=0.5 if is_large else 0.8,
	max_features=0.8, random_state=42, n_jobs=-1
	)
	models['SGD'] = SGDRegressor(max_iter=1500, early_stopping=True, random_state=42)
	models['DecisionTree'] = DecisionTreeRegressor(max_depth=12 if is_large else 18, min_samples_split=3, random_state=42)
	models['PassiveAggressive'] = PassiveAggressiveRegressor(max_iter=1500, random_state=42)
	if not is_large:
	models['SVR_RBF'] = SVR(kernel='rbf', C=1.0, gamma='scale')
	'''

	script += f'''
	return models


	# ============================================================================
	# STEP 4: TRAINING (mirrors ProductionModelTrainer.train_all())
	# ============================================================================

	def apply_smote(X_train, y_train):
	"""Apply SMOTE for imbalanced classification (same as DataVision).

	Only applied when:
	- imbalance_ratio > 1.5
	- min class count >= 6
	Falls back to class_weight='balanced' if SMOTE fails.
	"""
	if not HAS_SMOTE or TASK_TYPE != 'classification':
	return X_train, y_train, False

	class_counts = pd.Series(y_train).value_counts()
	max_count = class_counts.max()
	min_count = class_counts.min()
	imbalance_ratio = max_count / max(min_count, 1)

	if imbalance_ratio <= 1.5 or min_count < 6:
	return X_train, y_train, False

	try:
	k_neighbors = min(5, min_count - 1)
	n_classes = len(class_counts)

	if n_classes == 2:
	if imbalance_ratio > 100:
	sampling_strategy = 0.1
	elif imbalance_ratio > 20:
	sampling_strategy = 0.25
	else:
	sampling_strategy = 1.0
	else:
	sampling_strategy = 'not majority' if imbalance_ratio > 20 else 'auto'

	smote = SMOTE(k_neighbors=k_neighbors, sampling_strategy=sampling_strategy, random_state=42)
	X_resampled, y_resampled = smote.fit_resample(X_train, y_train)
	print(f" ⚖️ SMOTE applied: {{len(X_train)}} → {{len(X_resampled)}} samples (ratio: {{imbalance_ratio:.1f}}x)")
	return X_resampled, y_resampled, True
	except Exception as e:
	print(f" ⚠️ SMOTE failed ({{e}}), using class_weight='balanced'")
	return X_train, y_train, False


	def train_and_evaluate(X_train, X_test, y_train, y_test, mode='{"ultra" if is_ultra else "fast"}', selected_models=None):
	"""Train models with real DataVision scoring logic.

	Args:
	selected_models: Optional dict of {{name: model}} to train only specific models.
	If None, trains all models for the specified mode.

	Scoring:
	- Classification (balanced): F1-macro
	- Classification (imbalanced >5x): minority_recall0.6 + f1_macro0.4
	- Regression: R² score

	Cross-validation:
	- Fast: 3-fold StratifiedKFold/KFold
	- Ultra: 5-fold
	"""
	is_clf = TASK_TYPE == 'classification'
	is_ultra = mode == 'ultra'

	# Apply SMOTE (classification only)
	X_train_sm, y_train_sm, smote_applied = apply_smote(X_train, y_train)

	# Use selected_models if provided, otherwise get all models for the mode
	if selected_models is not None:
	models = selected_models
	else:
	models = get_models(len(X_train), mode)
	results = []
	best_model = None
	best_score = -999
	best_name = ""

	# Check imbalance for scoring
	if is_clf:
	class_counts = pd.Series(y_train).value_counts()
	imbalance_ratio = class_counts.max() / max(class_counts.min(), 1)
	is_imbalanced = imbalance_ratio > 5
	minority_class = class_counts.idxmin()

	# Cross-validation setup (same as DataVision)
	cv_folds = 3 if mode == 'fast' else 5
	if is_clf:
	cv = StratifiedKFold(n_splits=cv_folds, shuffle=True, random_state=42)
	else:
	cv = KFold(n_splits=cv_folds, shuffle=True, random_state=42)

	print(f"\\n🏋️ Training {{len(models)}} model(s) ({{'ALL' if selected_models is None else 'SELECTED'}} mode, {{cv_folds}}-fold CV)...")
	print("=" * 70)

	for name, model in models.items():
	try:
	t0 = time.time()
	model.fit(X_train_sm, y_train_sm)
	y_pred = model.predict(X_test)
	elapsed = time.time() - t0

	if is_clf:
	f1_macro = f1_score(y_test, y_pred, average='macro', zero_division=0)

	if is_imbalanced and len(pd.Series(y_train).unique()) == 2:
	# DataVision's real scoring for imbalanced binary:
	# minority_recall * 0.6 + f1_macro * 0.4
	from sklearn.metrics import recall_score as rs
	minority_recall = rs(y_test, y_pred, pos_label=minority_class, zero_division=0)
	score = minority_recall * 0.6 + f1_macro * 0.4
	else:
	score = f1_macro

	metrics = {{
	'accuracy': accuracy_score(y_test, y_pred),
	'precision': precision_score(y_test, y_pred, average='weighted', zero_division=0),
	'recall': recall_score(y_test, y_pred, average='weighted', zero_division=0),
	'f1_weighted': f1_score(y_test, y_pred, average='weighted', zero_division=0),
	'f1_macro': f1_macro,
	'score': score,
	}}
	else:
	score = r2_score(y_test, y_pred)
	metrics = {{
	'r2': score,
	'mae': mean_absolute_error(y_test, y_pred),
	'rmse': np.sqrt(mean_squared_error(y_test, y_pred)),
	'score': score,
	}}

	# Cross-validation
	try:
	cv_scoring = 'f1_weighted' if is_clf else 'r2'
	cv_scores = cross_val_score(model, X_train_sm, y_train_sm, cv=cv, scoring=cv_scoring)
	metrics['cv_mean'] = cv_scores.mean()
	metrics['cv_std'] = cv_scores.std()
	except:
	metrics['cv_mean'] = score
	metrics['cv_std'] = 0

	results.append({{
	'name': name, 'score': score, 'metrics': metrics, 'model': model,
	'y_pred': y_pred, 'time': elapsed, 'cv_mean': metrics.get('cv_mean', score)
	}})

	marker = "🏆" if score > best_score else " "
	cv_str = f"CV: {{metrics.get('cv_mean', 0):.4f}}±{{metrics.get('cv_std', 0):.3f}}"
	print(f" {{marker}} {{name:.<30}} Score: {{score:.4f}} \| {{cv_str}} ({{elapsed:.1f}}s)")

	if score > best_score:
	best_score = score
	best_model = model
	best_name = name

	except Exception as e:
	print(f" ❌ {{name:.<30}} Failed: {{str(e)[:60]}}")

	# Sort results by score
	results.sort(key=lambda x: x['score'], reverse=True)

	print("=" * 70)

	if not results:
	print("\\n❌ All models failed to train!")
	print(" Try: python train_traditional.py --all (to try all algorithms)")
	return None, '', {{}}, []

	print(f"\\n🏆 Best Model: {{best_name}} (Score: {{best_score:.4f}})")

	best_result = results[0]
	if is_clf:
	print(f"\\n📊 Classification Report:")
	print(classification_report(y_test, best_result['y_pred']))
	else:
	print(f"\\n📊 Regression Metrics:")
	for k, v in best_result['metrics'].items():
	if k not in ('score', 'cv_mean', 'cv_std'):
	print(f" {{k.upper()}}: {{v:.4f}}")

	# Overfitting check (same as DataVision's Production Intelligence)
	y_pred_train = best_model.predict(X_train)
	if is_clf:
	train_score = accuracy_score(y_train, y_pred_train)
	test_score = accuracy_score(y_test, best_result['y_pred'])
	else:
	train_score = r2_score(y_train, y_pred_train)
	test_score = r2_score(y_test, best_result['y_pred'])
	gap = train_score - test_score
	if gap > 0.1:
	print(f"\\n⚠️ Overfitting detected (train-test gap: {{gap:.4f}})")
	elif gap < 0.02:
	print(f"\\n✅ Good generalization (train-test gap: {{gap:.4f}})")

	return best_model, best_name, best_result['metrics'], results


	def main():
	parser = argparse.ArgumentParser(description="Train ML model (reproduces your DataVision AutoML)")
	parser.add_argument('--data', type=str, default='data/cleaned_data.csv', help='Path to CSV data file')
	parser.add_argument('--output', type=str, default='model.pkl', help='Output model file')
	parser.add_argument('--test-size', type=float, default=0.2, help='Test split ratio (DataVision uses 0.2)')
	parser.add_argument('--all', action='store_true', help='Train ALL algorithms (like full AutoML)')
	parser.add_argument('--mode', type=str, default='{"ultra" if is_ultra else "fast"}', choices=['fast', 'ultra'],
	help='Training mode when using --all (fast: ~10 models, ultra: ~25 models)')
	args = parser.parse_args()

	print("\\n" + "=" * 70)
	if args.all:
	print("🚀 DataVision AutoML — Training ALL Algorithms")
	else:
	print(f"🚀 DataVision AutoML — Training YOUR Winning Model: {best_model_name}")
	print(" (Use --all to train all algorithms like full AutoML)")
	print("=" * 70)

	if not os.path.exists(args.data):
	print(f"❌ Data file not found: {{args.data}}")
	return

	start_time = time.time()

	# Load data
	df = pd.read_csv(args.data)

	# Step 1: Clean data (ProductionDataCleaner)
	df = clean_data(df)
	df = df.dropna(subset=[TARGET_COLUMN])

	# Separate target BEFORE feature engineering (anti-leakage — same as DataVision)
	X = df.drop(columns=[TARGET_COLUMN])
	y = df[TARGET_COLUMN]

	# Encode target
	target_encoder = None
	if TASK_TYPE == 'classification':
	target_encoder = LabelEncoder()
	y = pd.Series(target_encoder.fit_transform(y), name=TARGET_COLUMN)
	print(f"\\n🎯 Target: {target} ({{len(target_encoder.classes_)}} classes: {{list(target_encoder.classes_)[:10]}})")
	else:
	print(f"\\n🎯 Target: {target} (regression)")

	# Train/test split BEFORE feature engineering (anti-leakage)
	X_train, X_test, y_train, y_test = train_test_split(
	X, y, test_size=args.test_size, random_state=42,
	stratify=y if TASK_TYPE == 'classification' else None
	)
	print(f"📦 Split: {{len(X_train)}} train / {{len(X_test)}} test")
	print(f" ⚠️ Feature engineering applied on train ONLY (anti-leakage)")

	# Step 2: Feature engineering (fit on train, transform test — anti-leakage)
	X_train_eng, label_encoders, scaler = engineer_features(X_train.copy(), mode=args.mode)

	# Transform test set using same encoding (simplified — uses same column structure)
	X_test_eng = X_test.copy()
	# Apply same encoding to test
	for col in X_test_eng.select_dtypes(include=['object', 'category']).columns:
	if col in label_encoders:
	le = label_encoders[col]
	X_test_eng[col] = X_test_eng[col].astype(str).map(
	lambda x: le.transform([x])[0] if x in le.classes_ else 0
	)
	else:
	le = LabelEncoder()
	X_test_eng[col] = le.fit_transform(X_test_eng[col].astype(str))
	for col in X_test_eng.select_dtypes(include=['object', 'category']).columns:
	X_test_eng[col] = pd.to_numeric(X_test_eng[col], errors='coerce').fillna(0)
	X_test_eng = X_test_eng.fillna(0)
	X_test_eng = X_test_eng.replace([np.inf, -np.inf], 0)

	# Ensure same columns as train
	for col in X_train_eng.columns:
	if col not in X_test_eng.columns:
	X_test_eng[col] = 0
	X_test_eng = X_test_eng[X_train_eng.columns]
	for col in X_test_eng.columns:
	X_test_eng[col] = pd.to_numeric(X_test_eng[col], errors='coerce').fillna(0)

	# Step 3 & 4: Train models
	# DEFAULT: Train ONLY your winning model (to reproduce exact result)
	# --all: Train all algorithms like full AutoML
	if args.all:
	print(f"\\n📊 Training ALL models (mode={{args.mode}})...")
	best_model, best_name, metrics, all_results = train_and_evaluate(
	X_train_eng, X_test_eng, y_train, y_test, mode=args.mode
	)
	else:
	print(f"\\n📊 Training ONLY your winning model: {best_model_name}")
	models = get_single_model(BEST_MODEL_NAME, len(X_train_eng))
	best_model, best_name, metrics, all_results = train_and_evaluate(
	X_train_eng, X_test_eng, y_train, y_test, mode=args.mode, selected_models=models
	)

	if best_model is None:
	print("\\n⚠️ Selected model failed. Retrying with ALL algorithms...")
	best_model, best_name, metrics, all_results = train_and_evaluate(
	X_train_eng, X_test_eng, y_train, y_test, mode='fast'
	)

	if best_model is None:
	print("\\n❌ All models failed. Check your data and dependencies.")
	return

	total_time = time.time() - start_time

	# Save model state (same structure as DataVision's engine_state)
	engine_state = {{
	'model': best_model,
	'model_name': best_name,
	'task_type': TASK_TYPE,
	'task_type_simple': TASK_TYPE,
	'target_column': TARGET_COLUMN,
	'feature_columns': X_train_eng.columns.tolist(),
	'label_encoders': label_encoders,
	'target_encoder': target_encoder,
	'scaler': scaler,
	'metrics': metrics,
	'production_mode': True,
	'mode': args.mode,
	'training_time_seconds': total_time,
	'leaderboard': [
	{{'name': r['name'], 'score': r['score'], 'metrics': r['metrics'], 'time': r.get('time', 0)}}
	for r in all_results
	],
	}}

	with open(args.output, 'wb') as f:
	pickle.dump(engine_state, f)

	file_size = os.path.getsize(args.output) / 1024
	print(f"\\n💾 Model saved to: {{args.output}} ({{file_size:.1f}} KB)")
	print(f"⏱️ Total time: {{total_time:.1f}}s")
	print("✅ Training complete!")


	if __name__ == '__main__':
	main()
	'''
	return script


	# =============================================================================
	# PREDICT.PY GENERATOR — DATA-AWARE
	# =============================================================================

	def _generate_predict_script(config: Dict) -> str:
	target = config['target_column']
	task_type = config['task_type']
	features = config.get('feature_columns', [])
	feature_metadata = config.get('feature_metadata', [])
	numeric_features = config.get('numeric_features', [])
	categorical_features = config.get('categorical_features', [])
	best_model = config.get('best_model_name', 'Unknown')

	is_classification = 'classif' in task_type.lower()

	# Build real feature input examples with actual values from metadata
	feature_examples = []
	for fm in feature_metadata[:20]:
	name = fm.get('name', '')
	ftype = fm.get('type', 'numeric')
	if ftype == 'numeric':
	mean_val = fm.get('mean', 0)
	mn = fm.get('min', '?')
	mx = fm.get('max', '?')
	feature_examples.append(f"# '{name}': {mean_val:.2f}, # numeric, range [{mn} .. {mx}]")
	elif ftype == 'categorical':
	options = fm.get('options', ['A'])
	first_opt = options[0] if options else 'A'
	opts_str = ', '.join(str(o) for o in options[:5])
	feature_examples.append(f"# '{name}': '{first_opt}', # categorical: [{opts_str}]")
	elif ftype == 'text':
	feature_examples.append(f"# '{name}': 'sample text here', # text field")

	if not feature_examples and features:
	for f in features[:20]:
	feature_examples.append(f"# '{f}': 0,")

	example_dict = "\n".join(feature_examples)

	# Build interactive prompts with actual types and ranges
	interactive_prompts = []
	for fm in feature_metadata[:30]:
	name = fm.get('name', '')
	ftype = fm.get('type', 'numeric')
	if ftype == 'numeric':
	mn = fm.get('min', '?')
	mx = fm.get('max', '?')
	mean_val = fm.get('mean', 0)
	interactive_prompts.append(f' "{name}": {{"type": "numeric", "min": {mn}, "max": {mx}, "default": {mean_val:.2f}}},')
	elif ftype == 'categorical':
	options = fm.get('options', [])
	opts_str = str(options[:10])
	interactive_prompts.append(f' "{name}": {{"type": "categorical", "options": {opts_str}}},')
	elif ftype == 'text':
	interactive_prompts.append(f' "{name}": {{"type": "text"}},')

	if not interactive_prompts and features:
	for f in features[:30]:
	interactive_prompts.append(f' "{f}": {{"type": "numeric", "default": 0}},')

	prompts_block = "\n".join(interactive_prompts)

	features_list = ", ".join(f"'{f}'" for f in features[:30])

	script = f'''"""
	🔮 Prediction Script — Uses Your ACTUAL Trained Model
	========================================================
	Auto-generated by DataVision AI

	╔════════════════════════════════════════════════════════════════════╗
	║ This script loads YOUR model.pkl that was trained by AutoML. ║
	║ The predictions match EXACTLY what you saw in DataVision. ║
	╚════════════════════════════════════════════════════════════════════╝

	YOUR MODEL:
	🏆 Model : {best_model}
	🎯 Target : {target}
	📊 Task : {task_type}
	📁 Features : {len(features)} columns

	Usage:
	python predict.py # Interactive mode
	python predict.py --json '{{"col1": 1, "col2": "A"}}' # JSON input
	python predict.py --csv input.csv # Batch predictions
	"""

	import os
	import pickle
	import argparse
	import json
	import re
	import warnings
	import numpy as np
	import pandas as pd
	from sklearn.preprocessing import LabelEncoder, RobustScaler

	warnings.filterwarnings('ignore')

	# Your actual feature columns from training
	FEATURE_COLUMNS = [{features_list}]
	TARGET_COLUMN = '{target}'
	BEST_MODE = '{config.get("best_mode", "traditional")}' # Which mode won during AutoML

	# Feature metadata with actual types and ranges from your dataset
	FEATURE_INFO = {{
	{prompts_block}
	}}


	# =============================================================================
	# Standalone Feature Engineer — Replicates DataVision's EXACT preprocessing
	# =============================================================================
	class StandaloneFeatureEngineer:
	"""Replicate DataVision's ProductionFeatureEngineer.transform() pipeline.

	Loaded from feature_engineer.pkl which contains all fitted sklearn
	transformers, encoders, and feature names from your AutoML training.
	"""

	def __init__(self, state):
	self.transformers = state.get('transformers', {{}})
	self.encoders = state.get('encoders', {{}})
	self.feature_names = state.get('feature_names', [])
	self.original_columns = state.get('original_columns', [])
	self.selected_feature_indices = state.get('selected_feature_indices', None)
	self.is_fitted = state.get('is_fitted', True)
	# Stopwords / sentiment words for text features
	self.stopwords = set("i me my myself we our ours ourselves you your yours yourself yourselves he him his himself she her hers herself it its itself they them their theirs themselves what which who whom this that these those am is are was were be been being have has had having do does did doing a an the and but if or because as until while of at by for with about against between through during before after above below to from up down in out on off over under again further then once here there when where why how all both each few more most other some such no nor not only own same so than too very s t can will just don should now d ll m o re ve y ain aren couldn didn doesn hadn hasn haven isn ma mightn mustn needn shan shouldn wasn weren won wouldn".split())
	self.positive_words = set("good great excellent amazing wonderful fantastic brilliant outstanding superb love best perfect beautiful awesome incredible impressive magnificent exceptional fabulous terrific marvelous stellar remarkable extraordinary delightful gorgeous splendid spectacular phenomenal exquisite lovely charming pleasant nice fine happy joy enjoy satisfied pleased glad cheerful excited thrilled".split())
	self.negative_words = set("bad terrible horrible awful worst poor disappointing disgusting pathetic useless dreadful atrocious appalling abysmal mediocre inferior unpleasant nasty vile wretched miserable lousy horrendous hideous ghastly hideous deplorable lamentable pitiful woeful tragic sad angry frustrated annoyed irritated disappointed upset unhappy distressed sorrowful gloomy depressed".split())

	def _clean_text(self, text):
	text = str(text).lower()
	text = re.sub(r'<[^>]+>', ' ', text)
	text = re.sub(r'http\\S+', ' ', text)
	text = re.sub(r'[^a-z0-9\\s]', ' ', text)
	text = ' '.join(w for w in text.split() if w not in self.stopwords and len(w) > 1)
	return text if text.strip() else 'empty'

	def _text_statistics(self, text):
	text = str(text)
	words = text.split()
	word_lengths = [len(w) for w in words] if words else [0]
	unique_ratio = len(set(words)) / max(len(words), 1)
	features = [
	len(text), len(words), np.mean(word_lengths), np.std(word_lengths),
	np.max(word_lengths) if word_lengths else 0, unique_ratio,
	text.count('!'), text.count('?'),
	sum(1 for c in text if c.isupper()) / max(len(text), 1),
	len([w for w in words if len(w) > 6]) / max(len(words), 1),
	]
	return features

	def _sentiment_score(self, text):
	words = set(str(text).lower().split())
	pos = len(words & self.positive_words)
	neg = len(words & self.negative_words)
	total = max(pos + neg, 1)
	return [pos / total, neg / total, (pos - neg) / total, total]

	def transform(self, df):
	if not self.is_fitted:
	raise ValueError("Feature engineer not fitted!")
	df = df.copy()

	# Handle duplicate column names
	cols = df.columns.tolist()
	seen = {{}}
	new_cols = []
	for col in cols:
	if col in seen:
	seen[col] += 1
	new_cols.append(f"{{col}}_{{seen[col]}}")
	else:
	seen[col] = 0
	new_cols.append(col)
	df.columns = new_cols

	feature_parts = []

	# ===== NUMERIC FEATURES =====
	if 'numeric_cols' in self.transformers:
	numeric_cols = self.transformers['numeric_cols']
	for col in numeric_cols:
	if col not in df.columns:
	df[col] = 0
	try:
	numeric_data = df[numeric_cols].apply(pd.to_numeric, errors='coerce').fillna(0).values
	scaler = self.transformers['numeric_scaler']
	numeric_scaled = scaler.transform(numeric_data)
	numeric_scaled = np.nan_to_num(numeric_scaled, nan=0.0, posinf=0.0, neginf=0.0)
	feature_parts.append(numeric_scaled)

	# Polynomial features
	if 2 <= len(numeric_cols) <= 10:
	interactions = []
	for i in range(min(len(numeric_cols), 4)):
	interactions.append((numeric_scaled[:, i] ** 2).reshape(-1, 1))
	for i in range(min(len(numeric_cols), 4)):
	for j in range(i + 1, min(len(numeric_cols), 4)):
	interactions.append((numeric_scaled[:, i] * numeric_scaled[:, j]).reshape(-1, 1))
	if interactions:
	feature_parts.append(np.hstack(interactions))
	except Exception as e:
	print(f" Warning: Numeric transform error: {{e}}")
	feature_parts.append(np.zeros((len(df), len(numeric_cols))))

	# ===== CATEGORICAL: One-hot =====
	for key, cols in self.transformers.items():
	if key.endswith('_onehot'):
	col_name = key.replace('_onehot', '')
	try:
	if col_name in df.columns:
	series = df[col_name].fillna('_MISSING_').astype(str).str.strip()
	dummies = pd.get_dummies(series, prefix=col_name)
	dummies = dummies.reindex(columns=cols, fill_value=0)
	feature_parts.append(dummies.values.astype(float))
	else:
	feature_parts.append(np.zeros((len(df), len(cols))))
	except:
	feature_parts.append(np.zeros((len(df), len(cols))))

	# ===== CATEGORICAL: Label + Frequency =====
	for col, le in self.encoders.items():
	try:
	if col in df.columns:
	series = df[col].fillna('_MISSING_').astype(str).str.strip()
	encoded = series.map(lambda s: le.transform([s])[0] if s in le.classes_ else -1)
	feature_parts.append(encoded.values.reshape(-1, 1).astype(float))
	freq_key = f'{{col}}_freq_map'
	if freq_key in self.transformers:
	freq_map = self.transformers[freq_key]
	freq_encoded = series.map(lambda s: freq_map.get(s, 0.0)).values.astype(float)
	feature_parts.append(freq_encoded.reshape(-1, 1))
	else:
	feature_parts.append(np.zeros((len(df), 1)))
	if f'{{col}}_freq_map' in self.transformers:
	feature_parts.append(np.zeros((len(df), 1)))
	except:
	feature_parts.append(np.zeros((len(df), 1)))

	# ===== CATEGORICAL INTERACTIONS =====
	if 'cat_interactions' in self.transformers:
	interaction_encoders = self.transformers['cat_interactions']
	for key, value in interaction_encoders.items():
	if key.endswith('_freq_map'):
	continue
	try:
	col1, col2, le = value
	if col1 in df.columns and col2 in df.columns:
	combined = df[col1].fillna('_NA_').astype(str) + "_X_" + df[col2].fillna('_NA_').astype(str)
	encoded = combined.map(lambda s: le.transform([s])[0] if s in le.classes_ else -1)
	feature_parts.append(encoded.values.reshape(-1, 1).astype(float))
	freq_key = f"{{key}}_freq_map"
	if freq_key in interaction_encoders:
	freq_map = interaction_encoders[freq_key]
	freq_encoded = combined.map(lambda s: freq_map.get(s, 0.0)).values.astype(float)
	feature_parts.append(freq_encoded.reshape(-1, 1))
	else:
	feature_parts.append(np.zeros((len(df), 1)))
	if f"{{key}}_freq_map" in interaction_encoders:
	feature_parts.append(np.zeros((len(df), 1)))
	except:
	pass

	# ===== NUMERIC-CATEGORICAL AGGREGATES =====
	if 'num_cat_aggs' in self.transformers:
	agg_maps = self.transformers['num_cat_aggs']
	for key, mean_map in agg_maps.items():
	try:
	parts = key.rsplit('_by_', 1)
	if len(parts) == 2:
	num_col = parts[0]
	cat_col = parts[1].rsplit('_mean', 1)[0]
	if num_col in df.columns and cat_col in df.columns:
	cat_series = df[cat_col].fillna('_NA_').astype(str)
	default_val = np.mean(list(mean_map.values())) if mean_map else 0
	agg_values = cat_series.map(lambda s: mean_map.get(s, default_val)).values.astype(float)
	feature_parts.append(agg_values.reshape(-1, 1))
	deviation = df[num_col].fillna(0).values - agg_values
	feature_parts.append(deviation.reshape(-1, 1))
	else:
	feature_parts.append(np.zeros((len(df), 1)))
	feature_parts.append(np.zeros((len(df), 1)))
	except:
	pass

	# ===== TEXT / NLP FEATURES =====
	for key, tfidf in self.transformers.items():
	if key.endswith('_tfidf'):
	col = key.replace('_tfidf', '')
	try:
	if col in df.columns:
	series = df[col].fillna('').astype(str)
	cleaned = series.apply(self._clean_text)
	tfidf_matrix = tfidf.transform(cleaned)
	svd_key = f'{{col}}_svd'
	if svd_key in self.transformers:
	text_features = self.transformers[svd_key].transform(tfidf_matrix)
	else:
	text_features = tfidf_matrix.toarray()
	feature_parts.append(text_features)
	text_stats = np.array([self._text_statistics(t) for t in series])
	feature_parts.append(text_stats)
	sentiment = np.array([self._sentiment_score(t) for t in series])
	feature_parts.append(sentiment)
	except:
	pass

	# ===== COMBINE =====
	if not feature_parts:
	return np.zeros((len(df), len(self.feature_names)))

	X = np.hstack(feature_parts)
	X = np.nan_to_num(X, nan=0.0, posinf=0.0, neginf=0.0)

	# Variance selector
	if 'variance_selector' in self.transformers:
	selector = self.transformers['variance_selector']
	try:
	if X.shape[1] == selector.n_features_in_:
	X = selector.transform(X)
	elif hasattr(self, 'selected_feature_indices') and self.selected_feature_indices is not None:
	valid_idx = [i for i in self.selected_feature_indices if i < X.shape[1]]
	if valid_idx:
	X = X[:, valid_idx]
	except:
	pass

	# PCA (if used during training)
	if 'pca' in self.transformers:
	try:
	pca = self.transformers['pca']
	if X.shape[1] == pca.n_features_in_:
	X = pca.transform(X)
	except:
	pass

	# Pad/truncate to match training dimensions
	if X.shape[1] != len(self.feature_names):
	if X.shape[1] < len(self.feature_names):
	padding = np.zeros((len(df), len(self.feature_names) - X.shape[1]))
	X = np.hstack([X, padding])
	else:
	X = X[:, :len(self.feature_names)]

	return X

	def transform_single(self, data):
	for col in self.original_columns:
	if col not in data:
	data[col] = 0
	df = pd.DataFrame([data])
	X = self.transform(df)
	expected = len(self.feature_names)
	if X.shape[1] != expected:
	if X.shape[1] < expected:
	X = np.hstack([X, np.zeros((1, expected - X.shape[1]))])
	else:
	X = X[:, :expected]
	return X


	# =============================================================================
	# Load model and feature engineer
	# =============================================================================
	FEATURE_ENGINEER = None

	def _load_feature_engineer():
	global FEATURE_ENGINEER
	if FEATURE_ENGINEER is not None:
	return FEATURE_ENGINEER
	eng_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'feature_engineer.pkl')
	if os.path.exists(eng_path):
	try:
	with open(eng_path, 'rb') as f:
	eng_state = pickle.load(f)
	FEATURE_ENGINEER = StandaloneFeatureEngineer(eng_state)
	print(f" Pipeline: feature_engineer.pkl ({{len(FEATURE_ENGINEER.feature_names)}} engineered features)")
	return FEATURE_ENGINEER
	except Exception as e:
	print(f" Warning: Could not load feature_engineer.pkl: {{e}}")
	return None


	def load_model(model_path='model.pkl'):
	"""Load the trained model and its components."""
	if not os.path.exists(model_path):
	print(f"❌ Model not found: {{model_path}}")
	print(" Run the training script first, or ensure model.pkl is in this directory.")
	return None

	with open(model_path, 'rb') as f:
	state = pickle.load(f)

	print(f"✅ Loaded model: {{state.get('model_name', 'Unknown')}}")
	print(f" Task: {{state.get('task_type', 'Unknown')}}")
	print(f" Target: {{state.get('target_column', 'Unknown')}}")

	# Load feature engineer (exact preprocessing pipeline)
	_load_feature_engineer()
	return state


	def predict_single(state, input_data: dict):
	"""Make a prediction for a single data point.

	Uses the EXACT same preprocessing pipeline as DataVision AutoML:
	1. StandaloneFeatureEngineer from feature_engineer.pkl (preferred)
	2. Fallback to basic label-encode + scale (less accurate)
	"""
	model = state['model']
	feature_cols = state.get('feature_columns', FEATURE_COLUMNS)
	target_encoder = state.get('target_encoder', None)
	task_type = state.get('task_type', '{task_type}')

	# Use feature engineer for EXACT AutoML-matching preprocessing
	engineer = _load_feature_engineer()
	if engineer is not None:
	try:
	X = engineer.transform_single(input_data)
	except Exception as e:
	print(f"Warning: Feature engineer failed ({{e}}), using fallback")
	X = _fallback_preprocess(input_data, state)
	else:
	X = _fallback_preprocess(input_data, state)

	prediction = model.predict(X)[0]
	result = {{'raw_prediction': prediction}}

	if 'classif' in task_type.lower() and target_encoder is not None:
	try:
	result['prediction'] = target_encoder.inverse_transform([int(prediction)])[0]
	except:
	result['prediction'] = prediction
	else:
	result['prediction'] = float(prediction)

	if 'classif' in task_type.lower() and hasattr(model, 'predict_proba'):
	try:
	proba = model.predict_proba(X)[0]
	result['confidence'] = float(max(proba))
	if target_encoder is not None:
	result['probabilities'] = {{
	str(cls): float(p) for cls, p in zip(target_encoder.classes_, proba)
	}}
	else:
	result['probabilities'] = {{str(i): float(p) for i, p in enumerate(proba)}}
	except:
	pass

	return result


	def _fallback_preprocess(input_data, state):
	"""Fallback: basic label-encode + scale (used only if feature_engineer.pkl missing)."""
	feature_cols = state.get('feature_columns', FEATURE_COLUMNS)
	label_encoders = state.get('label_encoders', {{}})
	scaler = state.get('scaler', None)

	row = {{}}
	for col in feature_cols:
	val = input_data.get(col, 0)
	if col in label_encoders:
	le = label_encoders[col]
	val_str = str(val)
	val = le.transform([val_str])[0] if val_str in le.classes_ else 0
	else:
	try:
	val = float(val)
	except (ValueError, TypeError):
	val = 0
	row[col] = val

	X_proc = pd.DataFrame([row], columns=feature_cols)
	if scaler is not None:
	try:
	X_proc = pd.DataFrame(scaler.transform(X_proc), columns=feature_cols)
	except:
	pass
	return X_proc


	def predict_batch(state, csv_path: str, output_path: str = None):
	"""Make predictions for a CSV file."""
	df = pd.read_csv(csv_path)
	print(f"\\n📊 Predicting for {{len(df)}} rows...")

	predictions = []
	for _, row in df.iterrows():
	result = predict_single(state, row.to_dict())
	predictions.append(result['prediction'])

	df['prediction'] = predictions
	if output_path is None:
	output_path = csv_path.replace('.csv', '_predictions.csv')
	df.to_csv(output_path, index=False)
	print(f"💾 Predictions saved to: {{output_path}}")
	return df


	# =============================================================================
	# Deep Learning Model Support (sklearn MLPClassifier/MLPRegressor)
	# =============================================================================
	DL_STATE = None

	def load_dl_model(dl_path='deep_learning_model.pkl'):
	"""Load the Deep Learning model (sklearn MLP) with its preprocessing state."""
	global DL_STATE
	if not os.path.exists(dl_path):
	return None
	try:
	with open(dl_path, 'rb') as f:
	DL_STATE = pickle.load(f)
	print(f"✅ Loaded DL model: {{DL_STATE.get('algorithm', 'MLP')}}")
	print(f" Task: {{DL_STATE.get('task_type', 'Unknown')}}")
	print(f" Features: {{len(DL_STATE.get('feature_metadata', []))}}")
	return DL_STATE
	except Exception as e:
	print(f"⚠️ Failed to load DL model: {{e}}")
	return None


	def predict_single_dl(dl_state, input_data: dict):
	"""Make a prediction using the Deep Learning model.

	Uses feature_metadata for correct preprocessing:
	- Numeric features: float conversion, fillna with mean
	- Categorical features: one-hot encoding matching training
	- StandardScaler applied to full feature vector
	"""
	model = dl_state['model']
	scaler = dl_state.get('scaler')
	label_encoder = dl_state.get('label_encoder')
	feature_metadata = dl_state.get('feature_metadata', [])
	task_type = dl_state.get('task_type', '{task_type}')
	classes = dl_state.get('classes', [])

	# Build feature vector using feature_metadata (same as DL engine)
	feature_values = []

	# Process numeric features first
	for meta in feature_metadata:
	if meta.get('type') == 'numeric':
	col = meta['name']
	if col in input_data:
	try:
	feature_values.append(float(input_data[col]))
	except (ValueError, TypeError):
	feature_values.append(meta.get('mean', 0))
	else:
	feature_values.append(meta.get('mean', 0))

	# Process categorical features (one-hot encoded during training)
	for meta in feature_metadata:
	if meta.get('type') == 'categorical':
	col = meta['name']
	value = str(input_data.get(col, ''))
	options = meta.get('options', [])

	# One-hot encode: one value per option
	for opt in options:
	feature_values.append(1.0 if value == opt else 0.0)
	# NaN dummy (dummy_na=True during training)
	feature_values.append(1.0 if not value or value == 'nan' else 0.0)

	# Fallback to feature_columns if no metadata
	if not feature_values and dl_state.get('feature_columns'):
	for col in dl_state['feature_columns']:
	if col in input_data:
	try:
	feature_values.append(float(input_data[col]) if not isinstance(input_data[col], str) else 0)
	except:
	feature_values.append(0)
	else:
	feature_values.append(0)

	X = np.array([feature_values])

	# Handle dimension mismatch with scaler
	if scaler and hasattr(scaler, 'n_features_in_'):
	expected = scaler.n_features_in_
	actual = X.shape[1]
	if actual < expected:
	padding = np.zeros((1, expected - actual))
	X = np.hstack([X, padding])
	elif actual > expected:
	X = X[:, :expected]

	# Scale
	if scaler:
	X_scaled = scaler.transform(X)
	else:
	X_scaled = X

	# Predict
	pred = model.predict(X_scaled)[0]
	result = {{'raw_prediction': pred}}

	if 'classif' in task_type.lower() and label_encoder is not None:
	try:
	result['prediction'] = str(label_encoder.inverse_transform([int(pred)])[0])
	except:
	result['prediction'] = str(pred)
	else:
	result['prediction'] = float(pred)

	if 'classif' in task_type.lower() and hasattr(model, 'predict_proba'):
	try:
	proba = model.predict_proba(X_scaled)[0]
	result['confidence'] = float(max(proba))
	if classes:
	result['probabilities'] = {{
	str(cls): round(float(p), 4) for cls, p in zip(classes, proba)
	}}
	except:
	pass

	return result


	def interactive_mode(state, use_dl=False):
	"""Interactive prediction mode with actual feature names and types."""
	print("\\n" + "=" * 50)
	print("🔮 Interactive Prediction Mode")
	if use_dl:
	print(" (Using Deep Learning model)")
	print("=" * 50)
	print("Enter values for each feature:\\n")

	# For DL mode, use feature_metadata from DL state
	if use_dl and DL_STATE:
	input_data = {{}}
	for meta in DL_STATE.get('feature_metadata', []):
	col = meta['name']
	ftype = meta.get('type', 'numeric')
	if ftype == 'numeric':
	mn = meta.get('min', '?')
	mx = meta.get('max', '?')
	default = meta.get('mean', 0)
	val = input(f" {{col}} [{{mn}} - {{mx}}] (default: {{default:.2f}}): ").strip()
	input_data[col] = float(val) if val else default
	elif ftype == 'categorical':
	options = meta.get('options', [])
	print(f" {{col}} options: {{options}}")
	val = input(f" {{col}} (pick one): ").strip()
	input_data[col] = val if val else (options[0] if options else '')
	result = predict_single_dl(DL_STATE, input_data)
	else:
	input_data = {{}}
	for col, info in FEATURE_INFO.items():
	ftype = info.get('type', 'numeric')

	if ftype == 'numeric':
	mn = info.get('min', '?')
	mx = info.get('max', '?')
	default = info.get('default', 0)
	val = input(f" {{col}} [{{mn}} - {{mx}}] (default: {{default}}): ").strip()
	input_data[col] = float(val) if val else default

	elif ftype == 'categorical':
	options = info.get('options', [])
	print(f" {{col}} options: {{options}}")
	val = input(f" {{col}} (pick one): ").strip()
	input_data[col] = val if val else (options[0] if options else '')

	elif ftype == 'text':
	val = input(f" {{col}} (enter text): ").strip()
	input_data[col] = val if val else ''
	result = predict_single(state, input_data)

	print(f"\\n🎯 Prediction: {{result['prediction']}}")
	if 'confidence' in result:
	print(f"📊 Confidence: {{result['confidence']:.1%}}")
	if 'probabilities' in result:
	print(f"📊 Probabilities:")
	for cls, prob in sorted(result['probabilities'].items(), key=lambda x: x[1], reverse=True):
	bar = "█" * int(prob * 30)
	print(f" {{cls:>15}}: {{prob:.1%}} {{bar}}")

	return result


	def main():
	parser = argparse.ArgumentParser(description="Make predictions with your trained model")
	parser.add_argument('--model', type=str, default='model.pkl', help='Path to model file')
	parser.add_argument('--dl-model', type=str, default='deep_learning_model.pkl', help='Path to DL model file')
	parser.add_argument('--mode', type=str, default='auto', choices=['auto', 'traditional', 'dl'], help='Prediction mode')
	parser.add_argument('--json', type=str, default=None, help='JSON string with input data')
	parser.add_argument('--csv', type=str, default=None, help='CSV file for batch predictions')
	parser.add_argument('--output', type=str, default=None, help='Output CSV for batch predictions')
	parser.add_argument('--interactive', action='store_true', help='Interactive mode')
	args = parser.parse_args()

	# Auto-detect which model to use based on BEST_MODE from training
	use_dl = False
	dl_state = None
	state = None

	# Use DL if: explicitly requested, or BEST_MODE is deep_learning, or DL model exists
	if args.mode == 'dl' or (args.mode == 'auto' and BEST_MODE == 'deep_learning') or (args.mode == 'auto' and os.path.exists(args.dl_model) and not os.path.exists(args.model)):
	dl_state = load_dl_model(args.dl_model)
	if dl_state:
	use_dl = True

	if not use_dl:
	state = load_model(args.model)
	if state is None:
	# Last resort: try DL model
	dl_state = load_dl_model(args.dl_model)
	if dl_state:
	use_dl = True
	else:
	return

	if args.json:
	input_data = json.loads(args.json)
	if use_dl:
	result = predict_single_dl(dl_state, input_data)
	else:
	result = predict_single(state, input_data)
	print(f"\\n🎯 Prediction: {{result['prediction']}}")
	if 'confidence' in result:
	print(f"📊 Confidence: {{result['confidence']:.1%}}")
	print(f"\\n📋 Full result: {{json.dumps(result, indent=2, default=str)}}")
	elif args.csv:
	if use_dl:
	# Batch DL predictions
	df = pd.read_csv(args.csv)
	print(f"\\n📊 Predicting for {{len(df)}} rows (DL mode)...")
	predictions = []
	for _, row in df.iterrows():
	result = predict_single_dl(dl_state, row.to_dict())
	predictions.append(result['prediction'])
	df['prediction'] = predictions
	output_path = args.output or args.csv.replace('.csv', '_predictions.csv')
	df.to_csv(output_path, index=False)
	print(f"💾 Predictions saved to: {{output_path}}")
	else:
	predict_batch(state, args.csv, args.output)
	else:
	interactive_mode(state, use_dl=use_dl)


	# =========================================================================
	# Example (using YOUR actual feature names):
	#
	# from predict import load_model, predict_single
	# state = load_model('model.pkl')
	# result = predict_single(state, {{
	{example_dict}
	# }})
	# print(result['prediction'])
	# =========================================================================


	if __name__ == '__main__':
	main()
	'''
	return script


	# =============================================================================
	# VISUALIZE.PY GENERATOR — DATA-AWARE
	# =============================================================================

	def _generate_visualize_script(config: Dict) -> str:
	target = config['target_column']
	task_type = config['task_type']
	is_classification = 'classif' in task_type.lower()
	features = config.get('feature_columns', [])
	numeric_features = config.get('numeric_features', [])
	categorical_features = config.get('categorical_features', [])
	best_model = config.get('best_model_name', 'Unknown')
	leaderboard = config.get('leaderboard', [])

	lb_names = [e.get('name', '?') for e in leaderboard[:10]]
	lb_scores = [e.get('score', e.get('accuracy', e.get('r2', 0))) for e in leaderboard[:10]]
	numeric_col_names = [f.get('name', '') for f in numeric_features[:9]]

	script = f'''"""
	📊 Visualization Script — ALL Charts Matching DataVision AutoML
	================================================================
	Auto-generated by DataVision AI

	Model : {best_model}
	Target Column : {target}
	Task Type : {task_type}
	Features : {len(features)} columns

	Supports Traditional ML AND Deep Learning models.
	Automatically loads the correct model based on BEST_MODE.

	Generates the SAME charts you see in DataVision AutoML:
	Classification: confusion matrix, class distribution, ROC curve,
	precision-recall, calibration, confidence histogram,
	metrics heatmap, feature importance, model comparison,
	correlation heatmap, feature distributions, boxplots
	Regression: actual vs predicted, residuals analysis, error distribution,
	prediction overview, feature importance, model comparison,
	correlation heatmap, feature distributions, boxplots
	Deep Learning: training loss curve, validation score curve,
	model architecture summary, MLP weight importance
	Unsupervised: KMeans clustering scatter plot (PCA 2D),
	cluster distribution chart

	Usage:
	python visualize.py # Auto-detect best model
	python visualize.py --mode dl # Force Deep Learning model
	python visualize.py --mode traditional # Force traditional ML model
	python visualize.py --data my_data.csv
	python visualize.py --output charts/
	"""

	import os
	import pickle
	import argparse
	import warnings
	import numpy as np
	import pandas as pd
	import matplotlib
	matplotlib.use('Agg')
	import matplotlib.pyplot as plt
	import seaborn as sns
	from sklearn.model_selection import train_test_split
	from sklearn.preprocessing import LabelEncoder, label_binarize
	from sklearn.metrics import (
	accuracy_score, confusion_matrix, classification_report,
	r2_score, mean_absolute_error, mean_squared_error,
	roc_curve, auc, precision_recall_curve, average_precision_score,
	precision_score, recall_score, f1_score
	)

	try:
	from sklearn.calibration import calibration_curve
	HAS_CALIBRATION = True
	except ImportError:
	HAS_CALIBRATION = False

	warnings.filterwarnings('ignore')
	sns.set_style("whitegrid")
	plt.rcParams.update({{
	'figure.figsize': (10, 6),
	'font.size': 12,
	'axes.titlesize': 14,
	'axes.labelsize': 12,
	}})

	TARGET_COLUMN = '{target}'
	TASK_TYPE = '{task_type}'
	BEST_MODE = '{config.get("best_mode", "traditional")}' # Which mode won during AutoML
	NUMERIC_FEATURES = {numeric_col_names}
	LEADERBOARD_NAMES = {lb_names}
	LEADERBOARD_SCORES = {lb_scores}


	def load_model_and_data(model_path='model.pkl', data_path='data/cleaned_data.csv'):
	if not os.path.exists(model_path):
	print(f"❌ Model not found: {{model_path}}")
	return None, None
	with open(model_path, 'rb') as f:
	state = pickle.load(f)
	df = pd.read_csv(data_path) if os.path.exists(data_path) else None
	return state, df


	def load_dl_model(dl_path='deep_learning_model.pkl'):
	"""Load the Deep Learning model (sklearn MLP)."""
	if not os.path.exists(dl_path):
	return None
	try:
	with open(dl_path, 'rb') as f:
	state = pickle.load(f)
	print(f"✅ Loaded DL model: {{state.get('algorithm', 'MLP')}}")
	print(f" Task: {{state.get('task_type', 'Unknown')}}")
	print(f" Features: {{len(state.get('feature_columns', state.get('feature_metadata', [])))}}")
	return state
	except Exception as e:
	print(f"⚠️ Failed to load DL model: {{e}}")
	return None


	def generate_charts(state, df, output_dir='charts'):
	os.makedirs(output_dir, exist_ok=True)

	model = state['model']
	task_type = state.get('task_type', '{task_type}')
	target_col = state.get('target_column', TARGET_COLUMN)
	feature_cols = state.get('feature_columns', [])
	is_clf = 'classif' in task_type.lower()
	is_dl = state.get('model_type') == 'deep_learning'
	charts = []

	# Load SAVED predictions from AutoML (exact results, no feature mismatch)
	y_test = np.array(state['y_test']) if state.get('y_test') is not None else None
	y_pred = np.array(state['y_pred']) if state.get('y_pred') is not None else None
	y_proba = np.array(state['y_proba']) if state.get('y_proba') is not None else None
	# Handle both ML (target_encoder) and DL (label_encoder) naming
	target_encoder = state.get('target_encoder', None)
	label_encoder = state.get('label_encoder', None)
	encoder = target_encoder or label_encoder
	class_names = list(encoder.classes_) if encoder is not None else None
	mode_display = 'Deep Learning (MLP)' if is_dl else 'Traditional ML'
	model_name = state.get('model_name', state.get('algorithm', 'Unknown'))

	print(f"\\n📊 Generating charts for {{model_name}} ({{mode_display}})...")
	print(f" Saved predictions: y_test={{y_test is not None}}, y_pred={{y_pred is not None}}, y_proba={{y_proba is not None}}")

	# =====================================================================
	# 1. MODEL COMPARISON / LEADERBOARD (same as AutoML)
	# =====================================================================
	if LEADERBOARD_NAMES and LEADERBOARD_SCORES:
	try:
	fig, ax = plt.subplots(figsize=(10, max(4, len(LEADERBOARD_NAMES) * 0.5)))
	colors = sns.color_palette('viridis', len(LEADERBOARD_NAMES))
	y_pos = range(len(LEADERBOARD_NAMES))
	ax.barh(y_pos, LEADERBOARD_SCORES, color=colors)
	ax.set_yticks(y_pos)
	ax.set_yticklabels(LEADERBOARD_NAMES)
	metric_label = 'F1-Score' if is_clf else 'R²'
	ax.set_xlabel(metric_label)
	ax.set_title(f'Model Leaderboard — {{metric_label}}')
	for i, v in enumerate(LEADERBOARD_SCORES):
	ax.text(v + 0.001, i, f'{{v:.4f}}', va='center', fontsize=9)
	plt.tight_layout()
	path = os.path.join(output_dir, 'model_comparison.png')
	plt.savefig(path, dpi=150, bbox_inches='tight')
	plt.close()
	charts.append(path)
	print(f" ✅ Model Comparison → {{path}}")
	except Exception as e:
	print(f" ⚠️ Model Comparison failed: {{e}}")

	# =====================================================================
	# 2. FEATURE IMPORTANCE (from model)
	# =====================================================================
	try:
	importances = None
	if hasattr(model, 'feature_importances_'):
	# Tree-based models (RF, XGB, etc.)
	importances = model.feature_importances_
	elif hasattr(model, 'coefs_') and len(model.coefs_) > 0:
	# MLP/Neural Network — use mean absolute weight of first layer
	importances = np.mean(np.abs(model.coefs_[0]), axis=1)
	elif hasattr(model, 'coef_'):
	# Linear models (LogReg, SVM, etc.)
	coef = model.coef_
	importances = np.mean(np.abs(coef), axis=0) if coef.ndim > 1 else np.abs(coef)

	if importances is not None:
	names = feature_cols[:len(importances)] if feature_cols else [f'Feature_{{i}}' for i in range(len(importances))]
	fi = pd.Series(importances, index=names).nlargest(min(20, len(names)))

	fig, ax = plt.subplots(figsize=(10, max(4, len(fi) * 0.35)))
	bars = fi.sort_values().plot(kind='barh', ax=ax,
	color=plt.cm.viridis(np.linspace(0.2, 0.8, len(fi))))
	title_suffix = ' (MLP Weight Magnitude)' if is_dl else ''
	ax.set_title(f'Feature Importance — Top Features{{title_suffix}}')
	ax.set_xlabel('Importance')
	for i, (val, name) in enumerate(zip(fi.sort_values(), fi.sort_values().index)):
	ax.text(val + 0.001, i, f'{{val:.4f}}', va='center', fontsize=8)
	plt.tight_layout()
	path = os.path.join(output_dir, 'feature_importance.png')
	plt.savefig(path, dpi=150, bbox_inches='tight')
	plt.close()
	charts.append(path)
	print(f" ✅ Feature Importance → {{path}}")
	except Exception as e:
	print(f" ⚠️ Feature Importance failed: {{e}}")

	# =====================================================================
	# DL-SPECIFIC: TRAINING HISTORY (loss curve + validation score)
	# =====================================================================
	if is_dl:
	training_history = state.get('training_history', {{}})
	loss_curve = training_history.get('loss', [])
	val_scores = training_history.get('val_score', [])

	if loss_curve:
	try:
	n_plots = 1 + (1 if val_scores else 0)
	fig, axes = plt.subplots(1, n_plots, figsize=(6 * n_plots, 5))
	if n_plots == 1:
	axes = [axes]

	# Loss curve
	axes[0].plot(range(1, len(loss_curve) + 1), loss_curve, color='#EF4444', lw=2)
	axes[0].set_xlabel('Epoch')
	axes[0].set_ylabel('Loss')
	axes[0].set_title('Training Loss Curve')
	axes[0].grid(True, alpha=0.3)

	# Validation score curve
	if val_scores and n_plots > 1:
	axes[1].plot(range(1, len(val_scores) + 1), val_scores, color='#22C55E', lw=2)
	axes[1].set_xlabel('Epoch')
	axes[1].set_ylabel('Validation Score')
	axes[1].set_title('Validation Score Curve')
	axes[1].grid(True, alpha=0.3)

	plt.suptitle(f'Deep Learning Training History — {{state.get("algorithm", "MLP")}}', fontsize=14)
	plt.tight_layout()
	path = os.path.join(output_dir, 'training_history.png')
	plt.savefig(path, dpi=150, bbox_inches='tight')
	plt.close()
	charts.append(path)
	print(f" ✅ Training History → {{path}}")
	except Exception as e:
	print(f" ⚠️ Training History failed: {{e}}")

	# DL Model Architecture Summary
	try:
	fig, ax = plt.subplots(figsize=(8, 4))
	ax.axis('off')
	info_lines = [
	f"Model: {{state.get('algorithm', 'MLP')}}",
	f"Task: {{state.get('task_type', 'Unknown')}}",
	f"Features: {{len(feature_cols)}}",
	]
	if hasattr(model, 'hidden_layer_sizes'):
	info_lines.append(f"Hidden Layers: {{model.hidden_layer_sizes}}")
	if hasattr(model, 'activation'):
	info_lines.append(f"Activation: {{model.activation}}")
	if hasattr(model, 'n_iter_'):
	info_lines.append(f"Epochs Trained: {{model.n_iter_}}")
	metrics = state.get('metrics', {{}})
	if metrics:
	info_lines.append("")
	info_lines.append("Metrics:")
	for k, v in metrics.items():
	if isinstance(v, float):
	info_lines.append(f" {{k}}: {{v:.4f}}")
	text = "\\n".join(info_lines)
	ax.text(0.1, 0.9, text, transform=ax.transAxes, fontsize=12,
	verticalalignment='top', fontfamily='monospace',
	bbox=dict(boxstyle='round', facecolor='#F0F9FF', alpha=0.8))
	ax.set_title('Deep Learning Model Summary', fontsize=14, fontweight='bold')
	plt.tight_layout()
	path = os.path.join(output_dir, 'dl_model_summary.png')
	plt.savefig(path, dpi=150, bbox_inches='tight')
	plt.close()
	charts.append(path)
	print(f" ✅ DL Model Summary → {{path}}")
	except Exception as e:
	print(f" ⚠️ DL Model Summary failed: {{e}}")

	# =====================================================================
	# CLASSIFICATION-SPECIFIC CHARTS
	# =====================================================================
	if is_clf and y_test is not None and y_pred is not None:
	n_classes = len(np.unique(y_test))
	is_binary = n_classes == 2

	# --- 3. CONFUSION MATRIX (normalized + counts) ---
	try:
	cm = confusion_matrix(y_test, y_pred)
	cm_norm = cm.astype('float') / cm.sum(axis=1, keepdims=True)
	labels = class_names if class_names else sorted(np.unique(y_test).tolist())

	fig, axes = plt.subplots(1, 2, figsize=(14, 5))
	# Counts
	sns.heatmap(cm, annot=True, fmt='d', cmap='Blues', ax=axes[0],
	xticklabels=labels, yticklabels=labels)
	axes[0].set_title('Confusion Matrix (Counts)')
	axes[0].set_xlabel('Predicted')
	axes[0].set_ylabel('Actual')
	# Normalized
	sns.heatmap(cm_norm, annot=True, fmt='.2%', cmap='Blues', ax=axes[1],
	xticklabels=labels, yticklabels=labels)
	axes[1].set_title('Confusion Matrix (Normalized)')
	axes[1].set_xlabel('Predicted')
	axes[1].set_ylabel('Actual')
	plt.tight_layout()
	path = os.path.join(output_dir, 'confusion_matrix.png')
	plt.savefig(path, dpi=150, bbox_inches='tight')
	plt.close()
	charts.append(path)
	print(f" ✅ Confusion Matrix → {{path}}")
	except Exception as e:
	print(f" ⚠️ Confusion Matrix failed: {{e}}")

	# --- 4. CLASS DISTRIBUTION (actual vs predicted) ---
	try:
	fig, axes = plt.subplots(1, 2, figsize=(12, 5))
	labels = class_names if class_names else sorted(np.unique(y_test).tolist())
	# Actual
	actual_counts = pd.Series(y_test).value_counts().sort_index()
	pred_counts = pd.Series(y_pred).value_counts().sort_index()
	axes[0].bar(range(len(actual_counts)), actual_counts.values, color=sns.color_palette('Set2'))
	axes[0].set_xticks(range(len(actual_counts)))
	axes[0].set_xticklabels(labels, rotation=45, ha='right')
	axes[0].set_title('Actual Distribution')
	axes[0].set_ylabel('Count')
	axes[1].bar(range(len(pred_counts)), pred_counts.reindex(actual_counts.index, fill_value=0).values,
	color=sns.color_palette('Set3'))
	axes[1].set_xticks(range(len(actual_counts)))
	axes[1].set_xticklabels(labels, rotation=45, ha='right')
	axes[1].set_title('Predicted Distribution')
	axes[1].set_ylabel('Count')
	plt.suptitle('Class Distribution: Actual vs Predicted', fontsize=14)
	plt.tight_layout()
	path = os.path.join(output_dir, 'class_distribution.png')
	plt.savefig(path, dpi=150, bbox_inches='tight')
	plt.close()
	charts.append(path)
	print(f" ✅ Class Distribution → {{path}}")
	except Exception as e:
	print(f" ⚠️ Class Distribution failed: {{e}}")

	# --- 5. ROC CURVE (binary + multi-class) ---
	if y_proba is not None:
	try:
	fig, ax = plt.subplots(figsize=(8, 8))
	if is_binary:
	proba_pos = y_proba[:, 1] if y_proba.ndim == 2 else y_proba
	fpr, tpr, _ = roc_curve(y_test, proba_pos)
	roc_auc = auc(fpr, tpr)
	ax.plot(fpr, tpr, color='#4F46E5', lw=2, label=f'ROC (AUC = {{roc_auc:.4f}})')
	else:
	y_test_bin = label_binarize(y_test, classes=sorted(np.unique(y_test)))
	colors = plt.cm.Set1(np.linspace(0, 1, n_classes))
	for i in range(min(n_classes, y_proba.shape[1])):
	fpr, tpr, _ = roc_curve(y_test_bin[:, i], y_proba[:, i])
	roc_auc = auc(fpr, tpr)
	label = class_names[i] if class_names and i < len(class_names) else f'Class {{i}}'
	ax.plot(fpr, tpr, color=colors[i], lw=2, label=f'{{label}} (AUC={{roc_auc:.3f}})')
	ax.plot([0, 1], [0, 1], 'k--', lw=1, alpha=0.5)
	ax.set_xlabel('False Positive Rate')
	ax.set_ylabel('True Positive Rate')
	ax.set_title('ROC Curve')
	ax.legend(loc='lower right')
	ax.set_xlim([0, 1])
	ax.set_ylim([0, 1.02])
	plt.tight_layout()
	path = os.path.join(output_dir, 'roc_curve.png')
	plt.savefig(path, dpi=150, bbox_inches='tight')
	plt.close()
	charts.append(path)
	print(f" ✅ ROC Curve → {{path}}")
	except Exception as e:
	print(f" ⚠️ ROC Curve failed: {{e}}")

	# --- 6. PRECISION-RECALL CURVE (binary) ---
	if y_proba is not None and is_binary:
	try:
	proba_pos = y_proba[:, 1] if y_proba.ndim == 2 else y_proba
	precision_vals, recall_vals, _ = precision_recall_curve(y_test, proba_pos)
	ap = average_precision_score(y_test, proba_pos)
	fig, ax = plt.subplots(figsize=(8, 6))
	ax.plot(recall_vals, precision_vals, color='#059669', lw=2,
	label=f'AP = {{ap:.4f}}')
	ax.set_xlabel('Recall')
	ax.set_ylabel('Precision')
	ax.set_title('Precision-Recall Curve')
	ax.legend(loc='lower left')
	ax.set_xlim([0, 1])
	ax.set_ylim([0, 1.02])
	plt.tight_layout()
	path = os.path.join(output_dir, 'precision_recall.png')
	plt.savefig(path, dpi=150, bbox_inches='tight')
	plt.close()
	charts.append(path)
	print(f" ✅ Precision-Recall → {{path}}")
	except Exception as e:
	print(f" ⚠️ Precision-Recall failed: {{e}}")

	# --- 7. CALIBRATION CURVE (binary) ---
	if y_proba is not None and is_binary and HAS_CALIBRATION and len(y_test) >= 20:
	try:
	proba_pos = y_proba[:, 1] if y_proba.ndim == 2 else y_proba
	fraction_of_positives, mean_predicted_value = calibration_curve(
	y_test, proba_pos, n_bins=10
	)
	fig, ax = plt.subplots(figsize=(8, 6))
	ax.plot(mean_predicted_value, fraction_of_positives, 's-', color='#4F46E5',
	label='Model', lw=2)
	ax.plot([0, 1], [0, 1], 'k--', label='Perfectly calibrated')
	ax.set_xlabel('Mean Predicted Probability')
	ax.set_ylabel('Fraction of Positives')
	ax.set_title('Calibration Curve')
	ax.legend()
	plt.tight_layout()
	path = os.path.join(output_dir, 'calibration.png')
	plt.savefig(path, dpi=150, bbox_inches='tight')
	plt.close()
	charts.append(path)
	print(f" ✅ Calibration Curve → {{path}}")
	except Exception as e:
	print(f" ⚠️ Calibration Curve failed: {{e}}")

	# --- 8. CONFIDENCE HISTOGRAM ---
	if y_proba is not None:
	try:
	max_proba = np.max(y_proba, axis=1) if y_proba.ndim == 2 else y_proba
	correct = (y_test == y_pred)
	fig, ax = plt.subplots(figsize=(8, 5))
	ax.hist(max_proba[correct], bins=20, alpha=0.7, color='#22C55E', label='Correct', edgecolor='white')
	ax.hist(max_proba[~correct], bins=20, alpha=0.7, color='#EF4444', label='Incorrect', edgecolor='white')
	ax.set_xlabel('Prediction Confidence')
	ax.set_ylabel('Count')
	ax.set_title('Confidence Distribution: Correct vs Incorrect')
	ax.legend()
	plt.tight_layout()
	path = os.path.join(output_dir, 'confidence_histogram.png')
	plt.savefig(path, dpi=150, bbox_inches='tight')
	plt.close()
	charts.append(path)
	print(f" ✅ Confidence Histogram → {{path}}")
	except Exception as e:
	print(f" ⚠️ Confidence Histogram failed: {{e}}")

	# --- 9. METRICS HEATMAP (per-class precision/recall/f1) ---
	if n_classes <= 10:
	try:
	labels = class_names if class_names else sorted(np.unique(y_test).tolist())
	prec_per_class = precision_score(y_test, y_pred, average=None, zero_division=0)
	rec_per_class = recall_score(y_test, y_pred, average=None, zero_division=0)
	f1_per_class = f1_score(y_test, y_pred, average=None, zero_division=0)

	metrics_df = pd.DataFrame({{
	'Precision': prec_per_class[:len(labels)],
	'Recall': rec_per_class[:len(labels)],
	'F1-Score': f1_per_class[:len(labels)],
	}}, index=labels[:len(prec_per_class)])

	fig, ax = plt.subplots(figsize=(8, max(3, len(labels) * 0.5)))
	sns.heatmap(metrics_df, annot=True, fmt='.3f', cmap='YlGn', ax=ax,
	vmin=0, vmax=1, linewidths=0.5)
	ax.set_title('Per-Class Metrics Heatmap')
	plt.tight_layout()
	path = os.path.join(output_dir, 'metrics_heatmap.png')
	plt.savefig(path, dpi=150, bbox_inches='tight')
	plt.close()
	charts.append(path)
	print(f" ✅ Metrics Heatmap → {{path}}")
	except Exception as e:
	print(f" ⚠️ Metrics Heatmap failed: {{e}}")

	# --- 10. CLASS BALANCE PIE CHART (actual vs predicted) ---
	try:
	fig, axes = plt.subplots(1, 2, figsize=(12, 5))
	labels = class_names if class_names else sorted(np.unique(y_test).tolist())
	actual_counts = pd.Series(y_test).value_counts().sort_index()
	pred_counts = pd.Series(y_pred).value_counts().sort_index()
	colors_pie = sns.color_palette('Set2', len(labels))
	axes[0].pie(actual_counts.values, labels=[str(l)[:15] for l in labels],
	autopct='%1.1f%%', colors=colors_pie,
	wedgeprops={{'edgecolor': 'white'}})
	axes[0].set_title('Actual Class Balance')
	axes[1].pie(pred_counts.reindex(actual_counts.index, fill_value=0).values,
	labels=[str(l)[:15] for l in labels],
	autopct='%1.1f%%', colors=colors_pie,
	wedgeprops={{'edgecolor': 'white'}})
	axes[1].set_title('Predicted Class Balance')
	plt.suptitle('Class Balance: Actual vs Predicted', fontsize=14)
	plt.tight_layout()
	path = os.path.join(output_dir, 'class_balance_pie.png')
	plt.savefig(path, dpi=150, bbox_inches='tight')
	plt.close()
	charts.append(path)
	print(f" ✅ Class Balance Pie → {{path}}")
	except Exception as e:
	print(f" ⚠️ Class Balance Pie failed: {{e}}")

	# --- 11. CONFIDENCE BOX PLOT BY CLASS ---
	if y_proba is not None:
	try:
	fig, ax = plt.subplots(figsize=(10, 6))
	max_proba = np.max(y_proba, axis=1) if y_proba.ndim == 2 else y_proba
	classes = sorted(np.unique(y_test).tolist())
	data_boxes = [max_proba[y_test == c] for c in classes]
	bp = ax.boxplot(data_boxes, patch_artist=True)
	box_colors = sns.color_palette('Set2', len(classes))
	for i, patch in enumerate(bp['boxes']):
	patch.set_facecolor(box_colors[i % len(box_colors)])
	patch.set_alpha(0.7)
	labels = [str(class_names[i])[:15] if class_names and i < len(class_names) else str(c) for i, c in enumerate(classes)]
	ax.set_xticklabels(labels, rotation=45, ha='right')
	ax.set_xlabel('Class')
	ax.set_ylabel('Prediction Confidence')
	ax.set_title('Confidence Distribution by Class')
	plt.tight_layout()
	path = os.path.join(output_dir, 'confidence_boxplot.png')
	plt.savefig(path, dpi=150, bbox_inches='tight')
	plt.close()
	charts.append(path)
	print(f" ✅ Confidence Boxplot → {{path}}")
	except Exception as e:
	print(f" ⚠️ Confidence Boxplot failed: {{e}}")

	# --- 12. SCORE DISTRIBUTION (KDE per class) ---
	if y_proba is not None and y_proba.ndim == 2:
	try:
	fig, ax = plt.subplots(figsize=(10, 6))
	for i in range(min(y_proba.shape[1], 5)):
	label = str(class_names[i])[:15] if class_names and i < len(class_names) else f'Class {{i}}'
	try:
	sns.kdeplot(y_proba[:, i], ax=ax, label=label, lw=2)
	except:
	ax.hist(y_proba[:, i], bins=30, alpha=0.5, label=label, density=True)
	ax.set_xlabel('Prediction Score')
	ax.set_ylabel('Density')
	ax.set_title('Score Distribution by Class')
	ax.legend()
	plt.tight_layout()
	path = os.path.join(output_dir, 'score_distribution.png')
	plt.savefig(path, dpi=150, bbox_inches='tight')
	plt.close()
	charts.append(path)
	print(f" ✅ Score Distribution → {{path}}")
	except Exception as e:
	print(f" ⚠️ Score Distribution failed: {{e}}")

	# =====================================================================
	# REGRESSION-SPECIFIC CHARTS
	# =====================================================================
	if not is_clf and y_test is not None and y_pred is not None:

	# --- 3. ACTUAL VS PREDICTED ---
	try:
	r2 = r2_score(y_test, y_pred)
	mae = mean_absolute_error(y_test, y_pred)
	rmse = np.sqrt(mean_squared_error(y_test, y_pred))

	fig, ax = plt.subplots(figsize=(8, 8))
	ax.scatter(y_test, y_pred, alpha=0.5, color='#3B82F6', edgecolors='white', s=50)
	lims = [min(y_test.min(), y_pred.min()), max(y_test.max(), y_pred.max())]
	ax.plot(lims, lims, '--', color='#EF4444', linewidth=2, label='Perfect')
	ax.set_title(f'Actual vs Predicted — R²={{r2:.4f}}, MAE={{mae:.4f}}, RMSE={{rmse:.4f}}')
	ax.set_xlabel('Actual')
	ax.set_ylabel('Predicted')
	ax.legend()
	plt.tight_layout()
	path = os.path.join(output_dir, 'actual_vs_predicted.png')
	plt.savefig(path, dpi=150, bbox_inches='tight')
	plt.close()
	charts.append(path)
	print(f" ✅ Actual vs Predicted → {{path}}")
	except Exception as e:
	print(f" ⚠️ Actual vs Predicted failed: {{e}}")

	# --- 4. RESIDUALS ANALYSIS (2x2 grid) ---
	try:
	residuals = y_test - y_pred
	fig, axes = plt.subplots(2, 2, figsize=(12, 10))
	# Residuals vs Predicted
	axes[0, 0].scatter(y_pred, residuals, alpha=0.5, color='#8B5CF6', s=30)
	axes[0, 0].axhline(y=0, color='#EF4444', linestyle='--', linewidth=2)
	axes[0, 0].set_title('Residuals vs Predicted')
	axes[0, 0].set_xlabel('Predicted'); axes[0, 0].set_ylabel('Residuals')
	# Histogram
	axes[0, 1].hist(residuals, bins=30, color='#06B6D4', edgecolor='white', alpha=0.8)
	axes[0, 1].set_title('Residual Distribution')
	axes[0, 1].set_xlabel('Residual'); axes[0, 1].set_ylabel('Frequency')
	# Q-Q plot
	sorted_res = np.sort(residuals)
	n = len(sorted_res)
	theoretical = np.random.normal(0, 1, n)
	theoretical.sort()
	axes[1, 0].scatter(theoretical, sorted_res, alpha=0.5, color='#F59E0B', s=20)
	lims = [min(theoretical.min(), sorted_res.min()), max(theoretical.max(), sorted_res.max())]
	axes[1, 0].plot(lims, lims, 'r--')
	axes[1, 0].set_title('Q-Q Plot')
	axes[1, 0].set_xlabel('Theoretical Quantiles'); axes[1, 0].set_ylabel('Sample Quantiles')
	# Residuals vs Index
	axes[1, 1].plot(range(len(residuals)), residuals, 'o', alpha=0.5, color='#10B981', markersize=4)
	axes[1, 1].axhline(y=0, color='#EF4444', linestyle='--')
	axes[1, 1].set_title('Residuals vs Index')
	axes[1, 1].set_xlabel('Sample Index'); axes[1, 1].set_ylabel('Residual')
	plt.suptitle('Residuals Analysis', fontsize=14)
	plt.tight_layout()
	path = os.path.join(output_dir, 'residuals_analysis.png')
	plt.savefig(path, dpi=150, bbox_inches='tight')
	plt.close()
	charts.append(path)
	print(f" ✅ Residuals Analysis → {{path}}")
	except Exception as e:
	print(f" ⚠️ Residuals Analysis failed: {{e}}")

	# --- 5. ERROR DISTRIBUTION ---
	try:
	abs_errors = np.abs(y_test - y_pred)
	fig, ax = plt.subplots(figsize=(8, 5))
	ax.hist(abs_errors, bins=30, color='#F97316', edgecolor='white', alpha=0.8, density=True)
	p50 = np.percentile(abs_errors, 50)
	p90 = np.percentile(abs_errors, 90)
	p95 = np.percentile(abs_errors, 95)
	ax.axvline(p50, color='green', linestyle='--', label=f'Median: {{p50:.3f}}')
	ax.axvline(p90, color='orange', linestyle='--', label=f'P90: {{p90:.3f}}')
	ax.axvline(p95, color='red', linestyle='--', label=f'P95: {{p95:.3f}}')
	ax.set_title('Absolute Error Distribution')
	ax.set_xlabel('Absolute Error')
	ax.set_ylabel('Density')
	ax.legend()
	plt.tight_layout()
	path = os.path.join(output_dir, 'error_distribution.png')
	plt.savefig(path, dpi=150, bbox_inches='tight')
	plt.close()
	charts.append(path)
	print(f" ✅ Error Distribution → {{path}}")
	except Exception as e:
	print(f" ⚠️ Error Distribution failed: {{e}}")

	# --- 6. PREDICTION OVERVIEW (sorted with ±2sigma) ---
	try:
	idx_sorted = np.argsort(y_test)
	fig, ax = plt.subplots(figsize=(10, 5))
	ax.plot(range(len(y_test)), y_test[idx_sorted], 'o', color='#3B82F6', markersize=3, label='Actual', alpha=0.7)
	ax.plot(range(len(y_pred)), y_pred[idx_sorted], 'x', color='#EF4444', markersize=3, label='Predicted', alpha=0.7)
	residuals_sorted = y_test[idx_sorted] - y_pred[idx_sorted]
	sigma = np.std(residuals_sorted)
	ax.fill_between(range(len(y_test)), y_pred[idx_sorted] - 2sigma, y_pred[idx_sorted] + 2sigma,
	alpha=0.15, color='red', label=f'±2σ ({{sigma:.3f}})')
	ax.set_title('Prediction Overview (Sorted)')
	ax.set_xlabel('Sorted Sample Index')
	ax.set_ylabel('Value')
	ax.legend()
	plt.tight_layout()
	path = os.path.join(output_dir, 'prediction_overview.png')
	plt.savefig(path, dpi=150, bbox_inches='tight')
	plt.close()
	charts.append(path)
	print(f" ✅ Prediction Overview → {{path}}")
	except Exception as e:
	print(f" ⚠️ Prediction Overview failed: {{e}}")

	# --- 7. ERROR BOX PLOT BY QUANTILES ---
	try:
	abs_errors = np.abs(y_test - y_pred)
	quantiles = np.percentile(y_test, [0, 25, 50, 75, 100])
	groups = []
	q_labels = []
	for qi in range(len(quantiles) - 1):
	mask = (y_test >= quantiles[qi]) & (y_test < quantiles[qi+1])
	if mask.sum() > 0:
	groups.append(abs_errors[mask])
	q_labels.append(f'Q{{qi+1}}')
	if groups:
	fig, ax = plt.subplots(figsize=(10, 6))
	bp = ax.boxplot(groups, patch_artist=True)
	box_colors = ['#3B82F6', '#22C55E', '#F59E0B', '#EF4444']
	for i, patch in enumerate(bp['boxes']):
	patch.set_facecolor(box_colors[i % len(box_colors)])
	patch.set_alpha(0.7)
	ax.set_xticklabels(q_labels)
	ax.set_xlabel('Target Quantile')
	ax.set_ylabel('Absolute Error')
	ax.set_title('Error Distribution by Target Range')
	plt.tight_layout()
	path = os.path.join(output_dir, 'error_boxplot.png')
	plt.savefig(path, dpi=150, bbox_inches='tight')
	plt.close()
	charts.append(path)
	print(f" ✅ Error Boxplot → {{path}}")
	except Exception as e:
	print(f" ⚠️ Error Boxplot failed: {{e}}")

	# --- 8. SCALE-LOCATION PLOT ---
	try:
	residuals_sl = y_test - y_pred
	sqrt_abs_resid = np.sqrt(np.abs(residuals_sl))
	fig, ax = plt.subplots(figsize=(8, 6))
	ax.scatter(y_pred, sqrt_abs_resid, alpha=0.5, color='#8B5CF6', s=30)
	z = np.polyfit(y_pred, sqrt_abs_resid, 1)
	p = np.poly1d(z)
	ax.plot(np.sort(y_pred), p(np.sort(y_pred)), 'r--', lw=2)
	ax.set_xlabel('Fitted Values')
	ax.set_ylabel('√\|Residuals\|')
	ax.set_title('Scale-Location Plot')
	plt.tight_layout()
	path = os.path.join(output_dir, 'scale_location.png')
	plt.savefig(path, dpi=150, bbox_inches='tight')
	plt.close()
	charts.append(path)
	print(f" ✅ Scale-Location → {{path}}")
	except Exception as e:
	print(f" ⚠️ Scale-Location failed: {{e}}")

	# =====================================================================
	# DATA EXPLORATION CHARTS (both classification & regression)
	# =====================================================================
	if df is not None:
	y = df[target_col] if target_col in df.columns else None

	# --- 10. CORRELATION HEATMAP ---
	try:
	numeric_df = df.select_dtypes(include=[np.number])
	if len(numeric_df.columns) >= 2:
	corr = numeric_df.corr()
	fig, ax = plt.subplots(figsize=(min(14, len(corr) + 2), min(12, len(corr) + 1)))
	mask = np.triu(np.ones_like(corr, dtype=bool))
	sns.heatmap(corr, mask=mask, annot=len(corr) <= 15, fmt='.2f',
	cmap='RdBu_r', center=0, ax=ax, square=True, linewidths=0.5)
	ax.set_title('Feature Correlation Heatmap')
	plt.tight_layout()
	path = os.path.join(output_dir, 'correlation_heatmap.png')
	plt.savefig(path, dpi=150, bbox_inches='tight')
	plt.close()
	charts.append(path)
	print(f" ✅ Correlation Heatmap → {{path}}")
	except Exception as e:
	print(f" ⚠️ Correlation Heatmap failed: {{e}}")

	# --- 11. FEATURE DISTRIBUTIONS ---
	try:
	plot_cols = NUMERIC_FEATURES if NUMERIC_FEATURES else df.select_dtypes(include=[np.number]).columns[:9].tolist()
	cols_to_plot = [c for c in plot_cols if c in df.columns][:9]
	if cols_to_plot:
	ncols = min(3, len(cols_to_plot))
	nrows = (len(cols_to_plot) + ncols - 1) // ncols
	fig, axes = plt.subplots(nrows, ncols, figsize=(5 * ncols, 4 * nrows))
	if nrows * ncols == 1:
	axes = np.array([axes])
	axes = axes.flatten()
	for i, col in enumerate(cols_to_plot):
	axes[i].hist(df[col].dropna(), bins=30, color='#4F46E5', edgecolor='white', alpha=0.8)
	axes[i].set_title(col, fontsize=10)
	axes[i].tick_params(labelsize=8)
	for i in range(len(cols_to_plot), len(axes)):
	axes[i].set_visible(False)
	plt.suptitle('Feature Distributions', fontsize=14)
	plt.tight_layout()
	path = os.path.join(output_dir, 'distribution_grid.png')
	plt.savefig(path, dpi=150, bbox_inches='tight')
	plt.close()
	charts.append(path)
	print(f" ✅ Distribution Grid → {{path}}")
	except Exception as e:
	print(f" ⚠️ Distribution Grid failed: {{e}}")

	# --- 12. BOXPLOT GRID ---
	try:
	plot_cols = NUMERIC_FEATURES if NUMERIC_FEATURES else df.select_dtypes(include=[np.number]).columns[:9].tolist()
	cols_to_plot = [c for c in plot_cols if c in df.columns][:9]
	if cols_to_plot:
	ncols = min(3, len(cols_to_plot))
	nrows = (len(cols_to_plot) + ncols - 1) // ncols
	fig, axes = plt.subplots(nrows, ncols, figsize=(5 * ncols, 4 * nrows))
	if nrows * ncols == 1:
	axes = np.array([axes])
	axes = axes.flatten()
	colors = sns.color_palette('Set2', len(cols_to_plot))
	for i, col in enumerate(cols_to_plot):
	axes[i].boxplot(df[col].dropna(), patch_artist=True,
	boxprops=dict(facecolor=colors[i], alpha=0.7))
	axes[i].set_title(col, fontsize=10)
	axes[i].tick_params(labelsize=8)
	for i in range(len(cols_to_plot), len(axes)):
	axes[i].set_visible(False)
	plt.suptitle('Feature Boxplots (Outlier Detection)', fontsize=14)
	plt.tight_layout()
	path = os.path.join(output_dir, 'boxplot_grid.png')
	plt.savefig(path, dpi=150, bbox_inches='tight')
	plt.close()
	charts.append(path)
	print(f" ✅ Boxplot Grid → {{path}}")
	except Exception as e:
	print(f" ⚠️ Boxplot Grid failed: {{e}}")

	# --- 13. TARGET DISTRIBUTION (from raw data) ---
	if y is not None:
	try:
	fig, ax = plt.subplots(figsize=(8, 5))
	if is_clf:
	y.value_counts().plot(kind='bar', ax=ax, color=sns.color_palette('Set2'))
	ax.set_ylabel('Count')
	else:
	ax.hist(y.dropna(), bins=30, color='#06B6D4', edgecolor='white', alpha=0.8)
	ax.set_ylabel('Frequency')
	ax.set_title(f'Target Distribution: {target}')
	ax.set_xlabel('{target}')
	plt.xticks(rotation=45, ha='right')
	plt.tight_layout()
	path = os.path.join(output_dir, 'target_distribution.png')
	plt.savefig(path, dpi=150, bbox_inches='tight')
	plt.close()
	charts.append(path)
	print(f" ✅ Target Distribution → {{path}}")
	except Exception as e:
	print(f" ⚠️ Target Distribution failed: {{e}}")

	# =====================================================================
	# UNSUPERVISED LEARNING — Auto Clustering (KMeans)
	# =====================================================================
	try:
	from sklearn.cluster import KMeans
	from sklearn.preprocessing import StandardScaler as ClusterScaler
	from sklearn.decomposition import PCA
	from sklearn.metrics import silhouette_score as sil_score

	numeric_df = df.select_dtypes(include=[np.number]).dropna()
	if len(numeric_df) >= 10 and len(numeric_df.columns) >= 2:
	# Scale for clustering
	c_scaler = ClusterScaler()
	X_clust = c_scaler.fit_transform(numeric_df)

	# Find optimal k using silhouette score
	best_k = 3
	best_sil = -1
	for k in range(2, min(8, len(numeric_df))):
	try:
	km = KMeans(n_clusters=k, random_state=42, n_init=10)
	labels_k = km.fit_predict(X_clust)
	sil_k = sil_score(X_clust, labels_k)
	if sil_k > best_sil:
	best_sil = sil_k
	best_k = k
	except:
	pass

	# Run clustering with best k
	km_final = KMeans(n_clusters=best_k, random_state=42, n_init=10)
	labels = km_final.fit_predict(X_clust)

	# PCA for 2D visualization
	pca = PCA(n_components=2)
	X_2d = pca.fit_transform(X_clust)

	# --- Cluster Scatter Plot ---
	fig, ax = plt.subplots(figsize=(10, 8))
	colors_clust = ['#2563eb', '#16a34a', '#dc2626', '#ea580c', '#9333ea',
	'#0891b2', '#db2777', '#d97706', '#0d9488', '#4f46e5']
	for i in range(best_k):
	mask = labels == i
	ax.scatter(X_2d[mask, 0], X_2d[mask, 1],
	c=colors_clust[i % len(colors_clust)],
	label=f'Cluster {{i}}', alpha=0.7, s=50,
	edgecolors='white', linewidth=0.5)
	ax.set_xlabel('PCA Component 1', fontweight='bold')
	ax.set_ylabel('PCA Component 2', fontweight='bold')
	ax.set_title(f'KMeans Clustering (k={{best_k}}, Silhouette={{best_sil:.3f}})',
	fontweight='bold', pad=15, fontsize=14)
	ax.legend(loc='best')
	ax.grid(alpha=0.3)
	plt.tight_layout()
	path = os.path.join(output_dir, 'cluster_scatter.png')
	plt.savefig(path, dpi=150, bbox_inches='tight')
	plt.close()
	charts.append(path)
	print(f" ✅ Cluster Scatter → {{path}}")

	# --- Cluster Distribution Bar Chart ---
	fig, ax = plt.subplots(figsize=(8, 5))
	unique, counts = np.unique(labels, return_counts=True)
	ax.bar([f'Cluster {{u}}' for u in unique], counts,
	color=[colors_clust[u % len(colors_clust)] for u in unique],
	edgecolor='white')
	ax.set_xlabel('Cluster')
	ax.set_ylabel('Count')
	ax.set_title('Cluster Size Distribution')
	for i, (u, c) in enumerate(zip(unique, counts)):
	ax.text(i, c + 0.5, str(c), ha='center', fontsize=10)
	plt.tight_layout()
	path = os.path.join(output_dir, 'cluster_distribution.png')
	plt.savefig(path, dpi=150, bbox_inches='tight')
	plt.close()
	charts.append(path)
	print(f" ✅ Cluster Distribution → {{path}}")
	except Exception as e:
	print(f" ⚠️ Clustering charts failed: {{e}}")

	print(f"\\n📊 Generated {{len(charts)}} charts in '{{output_dir}}/'")
	return charts


	def main():
	parser = argparse.ArgumentParser(description="Generate charts from your trained model")
	parser.add_argument('--model', type=str, default='model.pkl', help='Path to traditional ML model')
	parser.add_argument('--dl-model', type=str, default='deep_learning_model.pkl', help='Path to DL model')
	parser.add_argument('--mode', type=str, default='auto', choices=['auto', 'traditional', 'dl'],
	help='Which model to visualize (auto uses BEST_MODE)')
	parser.add_argument('--data', type=str, default='data/cleaned_data.csv')
	parser.add_argument('--output', type=str, default='charts')
	args = parser.parse_args()

	print("\\n" + "=" * 60)
	print("📊 DataVision Chart Generator")
	print("=" * 60)

	# Decide which model to load based on BEST_MODE
	use_dl = False
	state = None
	df = None

	if args.mode == 'dl' or (args.mode == 'auto' and BEST_MODE == 'deep_learning'):
	state = load_dl_model(args.dl_model)
	if state:
	use_dl = True
	else:
	print("⚠️ DL model not found, falling back to traditional ML model...")

	if not use_dl and args.mode != 'dl':
	state, df = load_model_and_data(args.model, args.data)
	if state is None and os.path.exists(args.dl_model):
	state = load_dl_model(args.dl_model)
	if state:
	use_dl = True

	if state is None:
	print("❌ No model found. Provide model.pkl or deep_learning_model.pkl")
	return

	# Load data if not yet loaded
	if df is None:
	data_path = args.data
	df = pd.read_csv(data_path) if os.path.exists(data_path) else None

	mode_display = 'deep_learning' if use_dl else state.get('mode', 'traditional')
	model_name = state.get('model_name', state.get('algorithm', 'Unknown'))

	print(f"\\n🏆 Model: {{model_name}}")
	print(f"📊 Mode: {{mode_display}}")
	print(f"🎯 Task: {{state.get('task_type', TASK_TYPE)}}")
	if use_dl:
	print(f"🧠 Architecture: {{state.get('algorithm', 'MLP')}}")
	if df is not None:
	print(f"📋 Data: {{len(df)}} rows, {{len(df.columns)}} columns")

	generate_charts(state, df, args.output)
	print(f"\\n✅ Done! Open '{{args.output}}/' to view charts.")


	if __name__ == '__main__':
	main()
	'''
	return script


	# =============================================================================
	# REQUIREMENTS.TXT — ONLY DEPS FOR MODES ACTUALLY TRAINED
	# =============================================================================

	def _generate_requirements(config: Dict) -> str:
	"""Generate requirements.txt matching DataVision's real dependencies."""
	reqs = [
	"numpy>=1.21.0",
	"pandas>=1.3.0",
	"scikit-learn>=1.0.0",
	"matplotlib>=3.5.0",
	"seaborn>=0.12.0",
	"joblib>=1.1.0",
	"scipy>=1.7.0",
	"flask>=2.2.0",
	"imbalanced-learn>=0.10.0", # SMOTE for class balancing
	]

	modes = config.get('modes_trained', ['traditional'])
	leaderboard = config.get('leaderboard', [])

	# Check what algorithms were actually used
	algo_names_lower = [e.get('name', '').lower() for e in leaderboard]
	algo_joined = " ".join(algo_names_lower)

	if 'xgb' in algo_joined or 'xgboost' in algo_joined:
	reqs.append("xgboost>=1.7.0")
	if 'lgb' in algo_joined or 'lightgbm' in algo_joined or 'lgbm' in algo_joined:
	reqs.append("lightgbm>=3.3.0")
	if 'catboost' in algo_joined or 'cat_boost' in algo_joined:
	reqs.append("catboost>=1.1.0")

	# NLP dependencies only if NLP was trained
	if 'nlp' in modes:
	nlp_result = config.get('results_per_mode', {}).get('nlp', {})
	algo_key = nlp_result.get('algorithm_key', nlp_result.get('algorithm', ''))

	reqs.append("nltk>=3.7")

	if 'transformer' in algo_key or 'sentence' in algo_key or 'bert' in algo_key:
	reqs.extend([
	"transformers>=4.20.0",
	"torch>=2.0.0",
	"sentence-transformers>=2.2.0",
	])

	# Deep Learning uses sklearn MLPClassifier/MLPRegressor (already included)
	# No additional dependencies needed for DL mode

	return "\n".join(sorted(set(reqs))) + "\n"


	# =============================================================================
	# README.MD — DOCUMENTS ACTUAL TRAINING RESULTS
	# =============================================================================

	def _generate_readme(config: Dict) -> str:
	target = config['target_column']
	task_type = config['task_type']
	best_model = config['best_model_name']
	best_mode = config['best_mode']
	metrics = config.get('metrics', {})
	data_summary = config.get('data_summary', {})
	features = config.get('feature_columns', [])
	modes_trained = config.get('modes_trained', ['traditional'])
	leaderboard = config.get('leaderboard', [])
	numeric_features = config.get('numeric_features', [])
	categorical_features = config.get('categorical_features', [])

	# Format metrics
	metrics_lines = []
	for k, v in metrics.items():
	if isinstance(v, float):
	if 0 < v < 1:
	metrics_lines.append(f"- {k.upper()}: {v:.4f} ({v*100:.1f}%)")
	else:
	metrics_lines.append(f"- {k.upper()}: {v:.4f}")
	else:
	metrics_lines.append(f"- {k.upper()}: {v}")
	metrics_block = "\n".join(metrics_lines) if metrics_lines else "- No metrics available"

	# Build leaderboard table
	lb_lines = []
	for i, entry in enumerate(leaderboard[:10]):
	name = entry.get('name', entry.get('model_name', '?'))
	score = entry.get('score', entry.get('accuracy', entry.get('r2', '?')))
	if isinstance(score, float):
	lb_lines.append(f"\| {i+1} \| {name} \| {score:.4f} \|")
	else:
	lb_lines.append(f"\| {i+1} \| {name} \| {score} \|")
	lb_block = "\n".join(lb_lines) if lb_lines else "\| - \| No leaderboard data \| - \|"

	# Project structure — only include scripts for modes trained
	structure_lines = [
	"├── model.pkl # Your trained model",
	"├── data/",
	"│ └── cleaned_data.csv # Your cleaned dataset",
	"├── predict.py # Make predictions (CLI / import)",
	"├── visualize.py # Generate performance charts",
	"├── evaluate.py # Comprehensive model evaluation",
	]
	if 'traditional' in modes_trained:
	structure_lines.append("├── train_traditional.py # Traditional ML training")
	if 'nlp' in modes_trained:
	structure_lines.append("├── train_nlp.py # NLP text classification")
	if 'deep_learning' in modes_trained:
	structure_lines.append("├── train_deep_learning.py # Deep Learning (sklearn MLP)")
	structure_lines.append("├── deep_learning_model.pkl # Trained DL model")
	structure_lines.extend([
	"├── charts/ # Pre-generated training charts (PNG)",
	"├── api_server.py # Flask REST API for deployment",
	"├── utils/",
	"│ └── preprocessing.py # Data preprocessing utilities",
	"├── config.json # Full training configuration",
	"├── requirements.txt # Python dependencies",
	"├── Dockerfile # Docker deployment",
	"└── README.md # This file",
	])
	structure_block = "\n".join(structure_lines)

	# Feature info table
	feature_lines = []
	for fm in numeric_features[:10]:
	name = fm.get('name', '')
	mn = fm.get('min', '?')
	mx = fm.get('max', '?')
	feature_lines.append(f"\| `{name}` \| numeric \| {mn} - {mx} \|")
	for fm in categorical_features[:10]:
	name = fm.get('name', '')
	options = fm.get('options', [])
	opts_str = ', '.join(str(o) for o in options[:4])
	feature_lines.append(f"\| `{name}` \| categorical \| {opts_str} \|")
	feature_block = "\n".join(feature_lines) if feature_lines else "\| (see config.json for details) \| \| \|"

	# Mode-specific usage instructions
	usage_modes = []
	if 'traditional' in modes_trained:
	usage_modes.append(f"""#### Traditional ML
	```bash
	python train_traditional.py # Reproduces your training
	python train_traditional.py --data your.csv # Custom data
	```""")
	if 'nlp' in modes_trained:
	usage_modes.append(f"""#### NLP Text Classification
	```bash
	python train_nlp.py # NLP training
	```""")
	if 'deep_learning' in modes_trained:
	usage_modes.append(f"""#### Deep Learning
	```bash
	python train_deep_learning.py # sklearn MLP neural network
	python train_deep_learning.py --epochs 100 # Custom epochs
	python predict.py --mode dl # Predict using DL model
	```""")
	usage_block = "\n\n".join(usage_modes) if usage_modes else "```bash\npython predict.py\n```"

	readme = f"""# 🚀 DataVision ML Project — Complete Guide

	Auto-generated by DataVision AI Business Analyst
	This ZIP contains YOUR actual trained model, data, scripts, and everything needed to use it.

	---

	## 📦 What's Inside This ZIP?

	```
	{structure_block}
	```

	### What Each File Does:

	\| File \| What It Does \| When To Use \|
	\|------\|-------------\|-------------\|
	\| `model.pkl` \| Your already trained ML model (ready to use) \| Loaded automatically by other scripts \|
	\| `data/cleaned_data.csv` \| Your cleaned dataset (used for charts & evaluation) \| Used by visualize.py and evaluate.py \|
	\| `predict.py` \| Make predictions on new data \| When you want to predict something \|
	\| `visualize.py` \| Generate all charts (confusion matrix, ROC, etc.) \| When you want to see model performance charts \|
	\| `evaluate.py` \| Show detailed model metrics and scores \| When you want to see accuracy, F1, R2 scores \|
	\| `api_server.py` \| Start a web API server with a form \| When you want a web interface or REST API \|
	\| `requirements.txt` \| List of Python packages needed \| Install once before running anything \|
	\| `config.json` \| Full training configuration and metadata \| Reference only \|
	\| `README.md` \| This guide \| You are reading it now \|

	---

	## Step-by-Step: Getting Started (For Beginners)

	### STEP 1: Install Python (if not installed)

	Download and install Python 3.8 or newer from https://www.python.org/downloads/

	> Important: During installation, check the box "Add Python to PATH"

	### STEP 2: Open Terminal / Command Prompt

	- Windows: Press `Win + R`, type `cmd`, press Enter
	- Mac/Linux: Open Terminal app

	Navigate to the project folder:
	```bash
	cd path/to/this/folder
	```

	### STEP 3: Install Dependencies (Run Once)

	```bash
	pip install -r requirements.txt
	```

	This installs all the Python packages needed. You only need to do this once.

	> If you get an error, try: `pip3 install -r requirements.txt`

	---

	## STEP 4: See Your Model Results (evaluate.py)

	This shows your exact same scores from DataVision.
	Automatically evaluates the correct model (Traditional ML or Deep Learning) based on your best training mode.

	```bash
	python evaluate.py # Auto-detect best model
	python evaluate.py --mode dl # Force Deep Learning model
	python evaluate.py --mode traditional # Force traditional ML model
	```

	What you will see:
	- Your model accuracy or R2 score
	- Classification report (precision, recall, F1 per class)
	- Confusion matrix
	- All stored metrics from training

	Additional options:
	```bash
	python evaluate.py --cv 5 # Run 5-fold cross-validation
	python evaluate.py --reeval # Re-evaluate model on dataset
	```

	Expected output example:
	```
	Model: RandomForest
	Task: classification
	Accuracy: 0.9450
	F1-Score: 0.9380
	```

	---

	## STEP 5: Make Predictions (predict.py)

	### Option A: Interactive Mode (Easiest)
	```bash
	python predict.py
	```
	It will ask you to enter a value for each feature one by one, then show the prediction.

	### Option B: JSON Input (Single Prediction)
	```bash
	python predict.py --json "{{\\"feature1\\": 25, \\"feature2\\": \\"value\\"}}"
	```
	Replace `feature1`, `feature2` with your actual column names.

	### Option C: CSV Batch Predictions (Many Rows)
	```bash
	python predict.py --csv your_data.csv
	```
	This reads your CSV, predicts for every row, saves results to `your_data_predictions.csv`.

	```bash
	python predict.py --csv input.csv --output results.csv
	```

	Expected output example:
	```
	Prediction: Normal
	Confidence: 94.5%
	Probabilities:
	Normal: 94.5%
	Abnormal: 5.5%
	```

	---

	## STEP 6: Generate Charts (visualize.py)

	Creates all the same charts you saw in DataVision AutoML.
	Automatically loads the correct model (Traditional ML or Deep Learning) based on your best training mode.

	```bash
	python visualize.py # Auto-detect best model
	python visualize.py --mode dl # Force Deep Learning model
	python visualize.py --mode traditional # Force traditional ML model
	```

	Charts are saved to a `charts/` folder. Open them with any image viewer.

	Classification charts generated:

	\| Chart File \| What It Shows \|
	\|------------\|---------------\|
	\| `model_comparison.png` \| All models ranked by score \|
	\| `feature_importance.png` \| Which features matter most \|
	\| `confusion_matrix.png` \| Correct vs wrong predictions per class \|
	\| `class_distribution.png` \| Actual vs predicted class counts \|
	\| `roc_curve.png` \| ROC curve with AUC score \|
	\| `precision_recall.png` \| Precision-Recall curve \|
	\| `calibration.png` \| Probability calibration curve \|
	\| `confidence_histogram.png` \| Confidence for correct vs incorrect \|
	\| `metrics_heatmap.png` \| Per-class precision, recall, F1 \|
	\| `class_balance_pie.png` \| Class balance pie charts \|
	\| `confidence_boxplot.png` \| Confidence box plot per class \|
	\| `score_distribution.png` \| Score distribution per class \|

	Regression charts generated:

	\| Chart File \| What It Shows \|
	\|------------\|---------------\|
	\| `actual_vs_predicted.png` \| Scatter: actual vs predicted values \|
	\| `residuals_analysis.png` \| 4-panel residual analysis \|
	\| `error_distribution.png` \| Error distribution with percentiles \|
	\| `prediction_overview.png` \| Sorted predictions with error band \|
	\| `error_boxplot.png` \| Error by target range (Q1-Q4) \|
	\| `scale_location.png` \| Scale-Location diagnostic plot \|

	Common charts (both):

	\| Chart File \| What It Shows \|
	\|------------\|---------------\|
	\| `correlation_heatmap.png` \| Feature correlations \|
	\| `distribution_grid.png` \| Distribution of each feature \|
	\| `boxplot_grid.png` \| Outlier detection per feature \|
	\| `target_distribution.png` \| Target column distribution \|

	Deep Learning charts (when DL model is loaded):

	\| Chart File \| What It Shows \|
	\|------------\|---------------\|
	\| `training_history.png` \| Training loss curve and validation scores \|
	\| `dl_model_summary.png` \| Model architecture, hyperparameters, metrics \|
	\| `feature_importance.png` \| MLP weight-based feature importance \|

	Unsupervised Learning charts (auto-generated):

	\| Chart File \| What It Shows \|
	\|------------\|---------------\|
	\| `cluster_scatter.png` \| KMeans clustering with PCA 2D projection \|
	\| `cluster_distribution.png` \| Cluster size distribution \|

	> Note: Your ZIP already contains pre-generated charts in the `charts/` folder
	> from your actual training. These are the EXACT same charts you saw in DataVision.
	> Running `visualize.py` will regenerate them from your model data.

	Custom options:
	```bash
	python visualize.py --data other_data.csv # Use different data
	python visualize.py --output my_charts/ # Save to different folder
	```

	---

	## STEP 7: Deploy as Web API (api_server.py)

	Start a local web server with a prediction form:

	```bash
	python api_server.py
	```

	Then open your browser and go to: http://localhost:5000/predict

	You will see a web form with your actual column names where you can:
	1. Enter values for each feature
	2. Click "Predict"
	3. See the prediction result instantly

	API Endpoints:

	\| URL \| Method \| What It Does \|
	\|-----\|--------\|-------------\|
	\| `/predict` \| GET \| Web form with your feature names \|
	\| `/predict` \| POST \| JSON API for developers \|
	\| `/health` \| GET \| Health check (is server running?) \|
	\| `/model-info` \| GET \| Model metadata and metrics \|

	Change port:
	```bash
	python api_server.py --port 8080
	```

	---

	## STEP 8: Retrain the Model (Optional)

	> Your model is already trained in `model.pkl`. Only retrain if you want to
	> experiment with new data or verify the training.

	{usage_block}

	---

	## STEP 9: Docker Deployment (Advanced)

	```bash
	docker build -t my-ml-model .
	docker run -p 5000:5000 my-ml-model
	```

	Then access: `http://localhost:5000/predict`

	---

	## Your Model Summary

	\| Property \| Value \|
	\|----------\|-------\|
	\| Target Column \| `{target}` \|
	\| Task Type \| {task_type} \|
	\| Best Model \| {best_model} \|
	\| Training Mode \| {best_mode} \|
	\| Features \| {len(features)} columns \|
	\| Dataset Rows \| {data_summary.get('rows', 'N/A')} \|
	\| Modes Trained \| {', '.join(modes_trained)} \|

	### Training Metrics
	{metrics_block}

	### Model Leaderboard

	\| Rank \| Model \| Score \|
	\|------\|-------\|-------\|
	{lb_block}

	### Feature Details

	\| Feature \| Type \| Range/Values \|
	\|---------\|------\|-------------\|
	{feature_block}

	---

	## Troubleshooting

	\| Problem \| Solution \|
	\|---------\|----------\|
	\| `ModuleNotFoundError` \| Run `pip install -r requirements.txt` \|
	\| `model.pkl not found` \| Make sure you are in the project folder \|
	\| `No module named flask` \| Run `pip install flask` \|
	\| `Port already in use` \| Use `python api_server.py --port 8080` \|
	\| Charts not generating \| Make sure `data/cleaned_data.csv` exists \|
	\| Wrong model loaded \| Use `--mode dl` or `--mode traditional` to force specific model \|
	\| DL model not found \| Make sure `deep_learning_model.pkl` exists \|
	\| Permission denied \| On Mac/Linux: try `python3` instead of `python` \|

	---

	## Using as Python Module (For Developers)

	```python
	from predict import load_model, predict_single

	state = load_model('model.pkl')
	result = predict_single(state, {{
	# Use your actual column names here
	# Run 'python predict.py' to see what columns are needed
	}})
	print(f"Prediction: {{result['prediction']}}")
	if 'confidence' in result:
	print(f"Confidence: {{result['confidence']:.1%}}")
	```

	---
	Generated by DataVision AI Business Analyst
	"""
	return readme


	# =============================================================================
	# NLP TRAINING SCRIPT — DATA-AWARE
	# =============================================================================

	def _generate_nlp_train_script(config: Dict, nlp_result: Dict) -> str:
	"""Generate NLP training script based on ACTUAL NLP training results."""
	target = config['target_column']
	task_type = config['task_type']
	text_col = config.get('nlp_text_column', '')

	# Extract actual NLP results
	actual_algorithm = nlp_result.get('algorithm', nlp_result.get('algorithm_key', 'tfidf_lr'))
	actual_best_model = nlp_result.get('best_model', actual_algorithm)
	actual_metrics = nlp_result.get('metrics', {})
	actual_classes = nlp_result.get('classes', [])

	# Determine what approach was used
	is_tfidf = 'tfidf' in actual_algorithm.lower()
	is_transformer = 'transformer' in actual_algorithm.lower() or 'bert' in actual_algorithm.lower() or 'sentence' in actual_algorithm.lower()

	# Build metrics comment
	metrics_lines = []
	for k, v in actual_metrics.items():
	if isinstance(v, float):
	metrics_lines.append(f"# {k}: {v:.4f}")
	else:
	metrics_lines.append(f"# {k}: {v}")
	metrics_block = "\n".join(metrics_lines) if metrics_lines else "# (metrics unavailable)"

	classes_str = str(actual_classes[:20]) if actual_classes else "[]"

	return f'''"""
	🔤 NLP Training Script — Based on Your Actual NLP Training
	=============================================================
	Auto-generated by DataVision AI

	YOUR NLP TRAINING RESULTS:
	Text Column : {text_col}
	Target : {target}
	Algorithm : {actual_algorithm}
	Best Model : {actual_best_model}
	Classes : {classes_str}

	{metrics_block}

	Usage:
	python train_nlp.py
	python train_nlp.py --data my_data.csv
	"""

	import os
	import re
	import pickle
	import argparse
	import warnings
	import numpy as np
	import pandas as pd
	from sklearn.model_selection import train_test_split
	from sklearn.preprocessing import LabelEncoder
	from sklearn.metrics import accuracy_score, f1_score, classification_report
	from sklearn.feature_extraction.text import TfidfVectorizer
	from sklearn.linear_model import LogisticRegression
	from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
	from sklearn.svm import LinearSVC
	from sklearn.naive_bayes import MultinomialNB
	from sklearn.pipeline import Pipeline

	warnings.filterwarnings('ignore')

	# Your actual training configuration
	TEXT_COLUMN = '{text_col}'
	TARGET_COLUMN = '{target}'
	ALGORITHM = '{actual_algorithm}'
	CLASSES = {classes_str}


	def clean_text(text):
	"""Basic text cleaning."""
	if not isinstance(text, str):
	return ""
	text = text.lower().strip()
	text = re.sub(r'<[^>]+>', '', text)
	text = re.sub(r'http\\S+', '', text)
	text = re.sub(r'[^a-zA-Z0-9\\s]', ' ', text)
	text = re.sub(r'\\s+', ' ', text).strip()
	return text


	def train_tfidf(X_train, X_test, y_train, y_test):
	"""Train with TF-IDF — matching your actual training approach."""
	print("\\n📊 Mode: TF-IDF + ML (your trained approach)")
	print("=" * 50)

	models = {{
	'Logistic Regression': Pipeline([
	('tfidf', TfidfVectorizer(max_features=10000, ngram_range=(1, 2), sublinear_tf=True)),
	('clf', LogisticRegression(max_iter=1000, C=1.0, random_state=42))
	]),
	'Linear SVM': Pipeline([
	('tfidf', TfidfVectorizer(max_features=10000, ngram_range=(1, 2), sublinear_tf=True)),
	('clf', LinearSVC(max_iter=2000, C=1.0, random_state=42))
	]),
	'Naive Bayes': Pipeline([
	('tfidf', TfidfVectorizer(max_features=10000, ngram_range=(1, 2))),
	('clf', MultinomialNB(alpha=0.1))
	]),
	'Random Forest': Pipeline([
	('tfidf', TfidfVectorizer(max_features=5000, ngram_range=(1, 2))),
	('clf', RandomForestClassifier(n_estimators=100, random_state=42, n_jobs=-1))
	]),
	}}

	best_score = -1
	best_model = None
	best_name = ""

	for name, pipeline in models.items():
	try:
	pipeline.fit(X_train, y_train)
	y_pred = pipeline.predict(X_test)
	acc = accuracy_score(y_test, y_pred)
	f1 = f1_score(y_test, y_pred, average='weighted', zero_division=0)
	marker = "🏆" if acc > best_score else " "
	print(f" {{marker}} {{name:.<35}} Acc: {{acc:.4f}} \| F1: {{f1:.4f}}")
	if acc > best_score:
	best_score = acc
	best_model = pipeline
	best_name = name
	except Exception as e:
	print(f" ❌ {{name:.<35}} Failed: {{e}}")

	print(f"\\n🏆 Best: {{best_name}} (Accuracy: {{best_score:.4f}})")
	y_pred = best_model.predict(X_test)
	print(f"\\n{{classification_report(y_test, y_pred)}}")

	return best_model, best_name, {{'accuracy': best_score}}


	def main():
	parser = argparse.ArgumentParser(description="NLP Training — Reproducing your DataVision results")
	parser.add_argument('--data', type=str, default='data/cleaned_data.csv')
	parser.add_argument('--output', type=str, default='nlp_model.pkl')
	parser.add_argument('--test-size', type=float, default=0.2)
	args = parser.parse_args()

	print("\\n" + "=" * 60)
	print("🔤 DataVision NLP Training — Your Configuration")
	print("=" * 60)

	if not os.path.exists(args.data):
	print(f"❌ Data file not found: {{args.data}}")
	return

	df = pd.read_csv(args.data)
	print(f"📊 Dataset: {{len(df)}} rows")

	text_col = TEXT_COLUMN
	if text_col not in df.columns:
	text_candidates = [c for c in df.columns if df[c].dtype == 'object' and c != TARGET_COLUMN]
	if text_candidates:
	text_col = max(text_candidates, key=lambda c: df[c].str.len().mean())
	print(f" Auto-detected text column: {{text_col}}")
	else:
	print("❌ No text column found")
	return

	df = df.dropna(subset=[text_col, TARGET_COLUMN])
	df['__clean_text__'] = df[text_col].apply(clean_text)

	X = df['__clean_text__']
	y = df[TARGET_COLUMN]

	le = LabelEncoder()
	y_encoded = le.fit_transform(y)
	print(f"🎯 Target: {{TARGET_COLUMN}} ({{len(le.classes_)}} classes: {{list(le.classes_)[:10]}})")

	X_train, X_test, y_train, y_test = train_test_split(
	X, y_encoded, test_size=args.test_size, random_state=42, stratify=y_encoded
	)

	model, name, metrics = train_tfidf(X_train, X_test, y_train, y_test)

	if model is None:
	print("❌ Training failed")
	return

	state = {{
	'model': model,
	'model_name': f'NLP_{{name}}',
	'task_type': 'classification',
	'target_column': TARGET_COLUMN,
	'text_column': text_col,
	'target_encoder': le,
	'metrics': metrics,
	'mode': 'nlp',
	}}

	with open(args.output, 'wb') as f:
	pickle.dump(state, f)

	print(f"\\n💾 NLP model saved to: {{args.output}}")
	print("✅ NLP training complete!")


	if __name__ == '__main__':
	main()
	'''


	# =============================================================================
	# DEEP LEARNING SCRIPT — DATA-AWARE (sklearn MLPClassifier/MLPRegressor)
	# =============================================================================

	def _generate_deep_learning_script(config: Dict, dl_result: Dict) -> str:
	"""Generate DL training script based on ACTUAL deep learning results.

	Uses sklearn's MLPClassifier/MLPRegressor (same as DataVision's DL engine).
	"""
	target = config['target_column']
	task_type = config['task_type']
	features = config.get('feature_columns', [])
	is_clf = 'classif' in task_type.lower()

	# Extract actual DL results
	actual_algorithm = dl_result.get('algorithm', dl_result.get('algorithm_key', 'auto'))
	actual_epochs = dl_result.get('epochs_completed', dl_result.get('epochs', 100))
	actual_metrics = dl_result.get('metrics', {})
	actual_best_model = dl_result.get('best_model', f'DL_{actual_algorithm}')

	metrics_lines = []
	for k, v in actual_metrics.items():
	if isinstance(v, float):
	metrics_lines.append(f"# {k}: {v:.4f}")
	else:
	metrics_lines.append(f"# {k}: {v}")
	metrics_block = "\n".join(metrics_lines) if metrics_lines else "# (metrics unavailable)"

	features_list = ", ".join(f"'{f}'" for f in features[:30])

	return f'''"""
	🧠 Deep Learning Training Script — Based on Your Training
	============================================================
	Auto-generated by DataVision AI

	Uses sklearn MLPClassifier/MLPRegressor (same as DataVision's DL engine).
	CPU-friendly neural networks with anti-overfitting intelligence.

	YOUR DL TRAINING RESULTS:
	Algorithm : {actual_algorithm}
	Best Model : {actual_best_model}
	Epochs : {actual_epochs}
	Target : {target}
	Task : {task_type}
	Features : {len(features)} columns

	{metrics_block}

	Usage:
	python train_deep_learning.py
	python train_deep_learning.py --epochs {actual_epochs}
	python train_deep_learning.py --data my_data.csv
	"""

	import os
	import pickle
	import argparse
	import warnings
	import numpy as np
	import pandas as pd
	from sklearn.model_selection import train_test_split
	from sklearn.preprocessing import LabelEncoder, StandardScaler
	from sklearn.neural_network import MLPClassifier, MLPRegressor
	from sklearn.metrics import (
	accuracy_score, f1_score, precision_score, recall_score,
	classification_report,
	r2_score, mean_absolute_error, mean_squared_error
	)

	warnings.filterwarnings('ignore')

	# Your actual training configuration
	FEATURE_COLUMNS = [{features_list}]
	TARGET_COLUMN = '{target}'
	TASK_TYPE = '{task_type}'
	MAX_EPOCHS = {actual_epochs}


	def get_smart_config(n_samples, n_features):
	"""Smart architecture selection based on dataset size (same as DataVision)."""
	if n_samples < 200:
	hidden = (32, 16)
	alpha = 0.1
	max_iter = 50
	patience = 10
	elif n_samples < 500:
	hidden = (64, 32)
	alpha = 0.05
	max_iter = 75
	patience = 12
	elif n_samples < 1000:
	hidden = (64, 32)
	alpha = 0.01
	max_iter = 100
	patience = 15
	elif n_samples < 5000:
	hidden = (128, 64)
	alpha = 0.005
	max_iter = 100
	patience = 15
	elif n_samples < 20000:
	hidden = (128, 64, 32)
	alpha = 0.001
	max_iter = 75
	patience = 12
	else:
	hidden = (256, 128, 64)
	alpha = 0.0005
	max_iter = 50
	patience = 10

	# High-dimensional adjustment
	if n_features > 1000:
	hidden = tuple(min(s, 64) for s in hidden)
	alpha = max(alpha, 0.01)
	elif n_features > 500:
	hidden = tuple(min(s, 128) for s in hidden)
	alpha = max(alpha, 0.005)

	return {{
	'hidden_layer_sizes': hidden,
	'alpha': alpha,
	'max_iter': max_iter,
	'n_iter_no_change': patience,
	}}


	def preprocess_for_dl(df):
	"""Preprocess data for deep learning — matches DataVision DL engine exactly."""
	df = df.dropna(subset=[TARGET_COLUMN])

	feature_df = df.drop(columns=[TARGET_COLUMN])
	y = df[TARGET_COLUMN]

	is_clf = 'classif' in TASK_TYPE.lower()

	# Encode target
	label_encoder = None
	if is_clf:
	label_encoder = LabelEncoder()
	y = pd.Series(label_encoder.fit_transform(y))

	# Separate numeric and categorical columns
	numeric_cols = feature_df.select_dtypes(include=[np.number]).columns.tolist()
	categorical_cols = feature_df.select_dtypes(include=['object', 'category']).columns.tolist()

	processed_parts = []
	feature_names = []
	feature_metadata = []

	# Process numeric features
	if numeric_cols:
	X_numeric = feature_df[numeric_cols].fillna(feature_df[numeric_cols].median()).values
	processed_parts.append(X_numeric)
	feature_names.extend(numeric_cols)

	for col in numeric_cols:
	try:
	feature_metadata.append({{
	'name': col,
	'type': 'numeric',
	'min': float(feature_df[col].min()),
	'max': float(feature_df[col].max()),
	'mean': float(feature_df[col].mean())
	}})
	except:
	feature_metadata.append({{
	'name': col, 'type': 'numeric',
	'min': 0, 'max': 100, 'mean': 50
	}})

	# Process categorical features (one-hot encode with dummy_na)
	if categorical_cols:
	for col in categorical_cols:
	dummies = pd.get_dummies(feature_df[col], prefix=col, dummy_na=True)
	processed_parts.append(dummies.values)
	feature_names.extend(dummies.columns.tolist())

	try:
	options = feature_df[col].dropna().unique().tolist()[:50]
	feature_metadata.append({{
	'name': col,
	'type': 'categorical',
	'options': [str(x) for x in options]
	}})
	except:
	pass

	if not processed_parts:
	raise ValueError("No valid features found")

	X = np.hstack(processed_parts)

	# Scale features
	scaler = StandardScaler()
	X_scaled = scaler.fit_transform(X)

	return X_scaled, y, is_clf, label_encoder, scaler, feature_names, feature_metadata, numeric_cols, categorical_cols


	def train_dl_model(X_train, X_test, y_train, y_test, is_clf, config, classes=None):
	"""Train sklearn MLP model with smart configuration."""
	hidden = config['hidden_layer_sizes']
	alpha = config['alpha']
	max_iter = config['max_iter']
	patience = config['n_iter_no_change']

	print(f"\\n🏋️ Training MLP Neural Network...")
	print(f" Architecture: {{hidden}}")
	print(f" Regularization (alpha): {{alpha}}")
	print(f" Max epochs: {{max_iter}}")
	print(f" Early stopping patience: {{patience}}")

	if is_clf:
	model = MLPClassifier(
	hidden_layer_sizes=hidden,
	activation='relu',
	solver='adam',
	alpha=alpha,
	batch_size='auto',
	learning_rate='adaptive',
	learning_rate_init=0.001,
	max_iter=max_iter,
	early_stopping=True,
	validation_fraction=0.15,
	n_iter_no_change=patience,
	random_state=42,
	verbose=True
	)
	else:
	model = MLPRegressor(
	hidden_layer_sizes=hidden,
	activation='relu',
	solver='adam',
	alpha=alpha,
	batch_size='auto',
	learning_rate='adaptive',
	learning_rate_init=0.001,
	max_iter=max_iter,
	early_stopping=True,
	validation_fraction=0.15,
	n_iter_no_change=patience,
	random_state=42,
	verbose=True
	)

	model.fit(X_train, y_train)

	y_pred = model.predict(X_test)

	if is_clf:
	acc = accuracy_score(y_test, y_pred)
	prec = precision_score(y_test, y_pred, average='weighted', zero_division=0)
	rec = recall_score(y_test, y_pred, average='weighted', zero_division=0)
	f1 = f1_score(y_test, y_pred, average='weighted', zero_division=0)
	print(f"\\n🏆 Final Accuracy: {{acc:.4f}} \| F1: {{f1:.4f}}")
	print(f" Precision: {{prec:.4f}} \| Recall: {{rec:.4f}}")
	print(f"\\n{{classification_report(y_test, y_pred)}}")
	metrics = {{'accuracy': acc, 'precision': prec, 'recall': rec, 'f1': f1}}
	# ROC-AUC
	try:
	from sklearn.metrics import roc_auc_score
	n_cls = len(np.unique(y_test))
	if n_cls == 2 and hasattr(model, 'predict_proba'):
	y_proba = model.predict_proba(X_test)[:, 1]
	roc = roc_auc_score(y_test, y_proba)
	metrics['roc_auc'] = roc
	print(f" ROC-AUC: {{roc:.4f}}")
	elif n_cls > 2 and hasattr(model, 'predict_proba'):
	y_proba = model.predict_proba(X_test)
	roc = roc_auc_score(y_test, y_proba, multi_class='ovr', average='weighted')
	metrics['roc_auc'] = roc
	print(f" ROC-AUC: {{roc:.4f}}")
	except Exception:
	pass
	else:
	r2 = r2_score(y_test, y_pred)
	mae = mean_absolute_error(y_test, y_pred)
	mse = mean_squared_error(y_test, y_pred)
	rmse = np.sqrt(mse)
	print(f"\\n🏆 R²: {{r2:.4f}} \| MAE: {{mae:.4f}} \| RMSE: {{rmse:.4f}}")
	print(f" MSE: {{mse:.4f}}")
	metrics = {{'r2': r2, 'mse': mse, 'mae': mae, 'rmse': rmse}}

	return model, metrics


	def main():
	parser = argparse.ArgumentParser(description="Deep Learning Training — Your Configuration")
	parser.add_argument('--data', type=str, default='data/cleaned_data.csv')
	parser.add_argument('--output', type=str, default='deep_learning_model.pkl')
	parser.add_argument('--epochs', type=int, default=MAX_EPOCHS)
	parser.add_argument('--test-size', type=float, default=0.2)
	args = parser.parse_args()

	print("\\n" + "=" * 60)
	print("🧠 DataVision Deep Learning — sklearn MLP")
	print("=" * 60)

	if not os.path.exists(args.data):
	print(f"❌ Data file not found: {{args.data}}")
	return

	df = pd.read_csv(args.data)
	print(f"📊 Dataset: {{len(df)}} rows × {{len(df.columns)}} columns")

	X, y, is_clf, label_encoder, scaler, feature_names, feature_metadata, numeric_cols, categorical_cols = preprocess_for_dl(df)
	print(f"📐 Features after preprocessing: {{X.shape[1]}}")

	classes = list(label_encoder.classes_) if label_encoder else []

	X_train, X_test, y_train, y_test = train_test_split(
	X, y, test_size=args.test_size, random_state=42,
	stratify=y if is_clf else None
	)
	print(f"📦 Split: {{len(X_train)}} train / {{len(X_test)}} test")

	# Get smart configuration based on data size
	config = get_smart_config(len(X_train), X.shape[1])
	config['max_iter'] = args.epochs

	model, metrics = train_dl_model(X_train, X_test, y_train, y_test, is_clf, config, classes)

	# Generate predictions for saved evaluation
	y_pred = model.predict(X_test)

	# Save in same format as DataVision DL engine
	state = {{
	'model': model,
	'scaler': scaler,
	'label_encoder': label_encoder,
	'feature_columns': feature_names,
	'numeric_cols': numeric_cols,
	'categorical_cols': categorical_cols,
	'feature_metadata': feature_metadata,
	'target_column': TARGET_COLUMN,
	'algorithm': '{actual_algorithm}',
	'task_type': TASK_TYPE,
	'classes': classes,
	'metrics': metrics,
	'model_type': 'deep_learning',
	'y_test': y_test.tolist() if hasattr(y_test, 'tolist') else list(y_test),
	'y_pred': y_pred.tolist() if hasattr(y_pred, 'tolist') else list(y_pred),
	}}

	with open(args.output, 'wb') as f:
	pickle.dump(state, f)

	print(f"\\n💾 Model saved to: {{args.output}}")
	print("✅ Deep learning training complete!")
	print("\\nTo predict: python predict.py --mode dl")


	if __name__ == '__main__':
	main()
	'''


	# =============================================================================
	# EVALUATE SCRIPT — DATA-AWARE
	# =============================================================================

	def _generate_evaluate_script(config: Dict) -> str:
	target = config['target_column']
	task_type = config['task_type']
	best_model = config.get('best_model_name', 'Unknown')
	metrics = config.get('metrics', {})
	is_clf = 'classif' in task_type.lower()

	metrics_lines = []
	for k, v in metrics.items():
	if isinstance(v, float):
	metrics_lines.append(f"# {k}: {v:.4f}")
	else:
	metrics_lines.append(f"# {k}: {v}")
	metrics_block = "\n".join(metrics_lines) if metrics_lines else "# (no stored metrics)"

	return f'''"""
	📊 Model Evaluation Script — Shows YOUR AutoML Results
	========================================================
	Auto-generated by DataVision AI

	╔════════════════════════════════════════════════════════════════════╗
	║ This script shows the EXACT SAME metrics as your AutoML training. ║
	║ Supports Traditional ML, Deep Learning, and NLP models. ║
	╚════════════════════════════════════════════════════════════════════╝

	Model : {best_model}
	Target Column : {target}
	Task Type : {task_type}

	YOUR AUTOML METRICS:
	{metrics_block}

	Usage:
	python evaluate.py # Show YOUR AutoML results
	python evaluate.py --reeval # Re-evaluate with data (may differ)
	"""

	import os
	import pickle
	import argparse
	import warnings
	import numpy as np
	import pandas as pd
	from sklearn.model_selection import train_test_split, cross_val_score
	from sklearn.preprocessing import LabelEncoder, StandardScaler
	from sklearn.metrics import (
	accuracy_score, precision_score, recall_score, f1_score,
	r2_score, mean_absolute_error, mean_squared_error,
	confusion_matrix, classification_report
	)
	try:
	from sklearn.metrics import roc_auc_score
	except ImportError:
	roc_auc_score = None

	warnings.filterwarnings('ignore')

	TARGET_COLUMN = '{target}'
	TASK_TYPE = '{task_type}'
	BEST_MODE = '{config.get("best_mode", "traditional")}' # Which mode won during AutoML


	def print_classification_metrics(y_test, y_pred, model=None, X_test=None, target_encoder=None):
	"""Print all classification metrics including ROC-AUC."""
	acc = accuracy_score(y_test, y_pred)
	prec = precision_score(y_test, y_pred, average='weighted', zero_division=0)
	rec = recall_score(y_test, y_pred, average='weighted', zero_division=0)
	f1 = f1_score(y_test, y_pred, average='weighted', zero_division=0)
	f1_macro = f1_score(y_test, y_pred, average='macro', zero_division=0)

	print(f"\\n✅ ACCURACY: {{acc:.4f}} ({{acc*100:.1f}}%)")
	print(f" Precision: {{prec:.4f}} ({{prec*100:.1f}}%)")
	print(f" Recall: {{rec:.4f}} ({{rec*100:.1f}}%)")
	print(f" F1-Weighted: {{f1:.4f}} ({{f1*100:.1f}}%)")
	print(f" F1-Macro: {{f1_macro:.4f}} ({{f1_macro*100:.1f}}%)")

	# ROC-AUC
	if roc_auc_score is not None and model is not None and X_test is not None:
	try:
	n_classes = len(np.unique(y_test))
	if n_classes == 2 and hasattr(model, 'predict_proba'):
	y_proba = model.predict_proba(X_test)[:, 1]
	roc = roc_auc_score(y_test, y_proba)
	print(f" ROC-AUC: {{roc:.4f}} ({{roc*100:.1f}}%)")
	elif n_classes > 2 and hasattr(model, 'predict_proba'):
	y_proba = model.predict_proba(X_test)
	roc = roc_auc_score(y_test, y_proba, multi_class='ovr', average='weighted')
	print(f" ROC-AUC: {{roc:.4f}} ({{roc*100:.1f}}%)")
	elif n_classes == 2 and hasattr(model, 'decision_function'):
	y_scores = model.decision_function(X_test)
	roc = roc_auc_score(y_test, y_scores)
	print(f" ROC-AUC: {{roc:.4f}} ({{roc*100:.1f}}%)")
	except Exception:
	pass

	print("\\n📊 Classification Report:")
	if target_encoder is not None:
	try:
	print(classification_report(y_test, y_pred, target_names=target_encoder.classes_))
	except Exception:
	print(classification_report(y_test, y_pred))
	else:
	print(classification_report(y_test, y_pred))

	print("\\n📊 Confusion Matrix:")
	cm = confusion_matrix(y_test, y_pred)
	print(cm)


	def print_regression_metrics(y_test, y_pred):
	"""Print all regression metrics."""
	r2 = r2_score(y_test, y_pred)
	mae = mean_absolute_error(y_test, y_pred)
	mse = mean_squared_error(y_test, y_pred)
	rmse = np.sqrt(mse)

	print(f"\\n✅ R² SCORE: {{r2:.4f}}")
	print(f" MSE: {{mse:.4f}}")
	print(f" RMSE: {{rmse:.4f}}")
	print(f" MAE: {{mae:.4f}}")


	def load_dl_model(dl_path='deep_learning_model.pkl'):
	"""Load the Deep Learning model."""
	if not os.path.exists(dl_path):
	return None
	try:
	with open(dl_path, 'rb') as f:
	state = pickle.load(f)
	return state
	except Exception as e:
	print(f"⚠️ Failed to load DL model: {{e}}")
	return None


	def main():
	parser = argparse.ArgumentParser(description="Show your AutoML training results")
	parser.add_argument('--model', type=str, default='model.pkl')
	parser.add_argument('--dl-model', type=str, default='deep_learning_model.pkl', help='Path to DL model file')
	parser.add_argument('--mode', type=str, default='auto', choices=['auto', 'traditional', 'dl'], help='Evaluation mode')
	parser.add_argument('--data', type=str, default='data/cleaned_data.csv')
	parser.add_argument('--reeval', action='store_true', help='Re-evaluate with fresh train/test split (may differ from AutoML)')
	parser.add_argument('--cv', type=int, default=5, help='Number of cross-validation folds for --reeval')
	args = parser.parse_args()

	print("\\n" + "=" * 60)
	print("📊 DataVision AutoML — Your Training Results")
	print("=" * 60)

	# Decide which model to load based on BEST_MODE
	use_dl = False
	state = None

	if args.mode == 'dl' or (args.mode == 'auto' and BEST_MODE == 'deep_learning'):
	state = load_dl_model(args.dl_model)
	if state:
	use_dl = True
	else:
	print("⚠️ DL model not found, falling back to traditional ML model...")

	if not use_dl and args.mode != 'dl':
	if os.path.exists(args.model):
	with open(args.model, 'rb') as f:
	state = pickle.load(f)
	elif os.path.exists(args.dl_model):
	state = load_dl_model(args.dl_model)
	if state:
	use_dl = True

	if state is None:
	print("❌ No model found. Provide model.pkl or deep_learning_model.pkl")
	return

	model = state['model']
	task_type = state.get('task_type', TASK_TYPE)
	is_clf = 'classif' in task_type.lower()
	mode_display = 'deep_learning' if use_dl else state.get('mode', 'traditional')
	model_name = state.get('model_name', state.get('algorithm', 'Unknown'))

	print(f"\\n🏆 Model: {{model_name}}")
	print(f"🎯 Task: {{task_type}}")
	print(f"📊 Mode: {{mode_display}}")
	if use_dl:
	print(f"🧠 Architecture: {{state.get('algorithm', 'MLP')}}")
	print(f" Features: {{len(state.get('feature_columns', state.get('feature_metadata', [])))}}")

	# =========================================================================
	# OPTION 1: Use SAVED y_test/y_pred (EXACT same as AutoML showed)
	# =========================================================================
	y_test_saved = state.get('y_test')
	y_pred_saved = state.get('y_pred')

	if not args.reeval and y_test_saved is not None and y_pred_saved is not None:
	print("\\n" + "=" * 60)
	print("📋 YOUR AUTOML TRAINING RESULTS (from saved predictions)")
	print("=" * 60)

	y_test = np.array(y_test_saved)
	y_pred = np.array(y_pred_saved)

	# For ROC-AUC we need X_test and the model — try to reconstruct
	X_test_for_roc = None
	if use_dl:
	X_test_saved = state.get('X_test')
	if X_test_saved is not None:
	X_test_for_roc = np.array(X_test_saved)

	if is_clf:
	print_classification_metrics(
	y_test, y_pred, model=model,
	X_test=X_test_for_roc,
	target_encoder=state.get('target_encoder') if not use_dl else state.get('label_encoder')
	)
	else:
	print_regression_metrics(y_test, y_pred)

	# Also show stored metrics
	stored_metrics = state.get('metrics', {{}})
	if stored_metrics:
	print("\\n📋 Stored Metrics from Training:")
	for k, v in stored_metrics.items():
	if isinstance(v, float):
	print(f" {{k}}: {{v:.4f}}")
	else:
	print(f" {{k}}: {{v}}")

	print("\\n" + "=" * 60)
	print("✅ These are your EXACT AutoML results.")
	print(" Use --reeval to re-evaluate with fresh data split (may differ).")
	print("=" * 60)
	return

	# =========================================================================
	# OPTION 2: Re-evaluate (may produce different results due to random split)
	# =========================================================================
	print("\\n⚠️ Re-evaluating with fresh train/test split...")
	print(" Note: Results may differ from AutoML due to different random split.")

	if not os.path.exists(args.data):
	print(f"⚠️ Data file not found: {{args.data}}")
	print(" Showing stored metrics only:")
	for k, v in state.get('metrics', {{}}).items():
	print(f" {{k}}: {{v}}")
	return

	df = pd.read_csv(args.data)
	target_col = state.get('target_column', TARGET_COLUMN)
	feature_cols = state.get('feature_columns', [])

	if target_col not in df.columns:
	print(f"❌ Target '{{target_col}}' not in data")
	return

	df = df.dropna(subset=[target_col])

	if use_dl:
	# =====================================================================
	# DL Re-evaluation: Use the DL model's preprocessing pipeline
	# =====================================================================
	feature_metadata = state.get('feature_metadata', [])
	scaler = state.get('scaler')
	label_encoder = state.get('label_encoder')

	# Build feature matrix from metadata
	X_parts = []
	for meta in feature_metadata:
	col = meta['name']
	if meta.get('type') == 'numeric':
	if col in df.columns:
	X_parts.append(df[col].apply(pd.to_numeric, errors='coerce').fillna(meta.get('mean', 0)).values.reshape(-1, 1))
	else:
	X_parts.append(np.full((len(df), 1), meta.get('mean', 0)))
	elif meta.get('type') == 'categorical':
	if col in df.columns:
	encoding = meta.get('encoding', {{}})
	X_parts.append(df[col].astype(str).map(lambda v: encoding.get(v, 0)).values.reshape(-1, 1))
	else:
	X_parts.append(np.zeros((len(df), 1)))

	if X_parts:
	X = np.hstack(X_parts).astype(float)
	else:
	X = df.drop(columns=[target_col]).select_dtypes(include=[np.number]).fillna(0).values

	y = df[target_col].values
	if is_clf and label_encoder:
	try:
	y = label_encoder.transform(y)
	except Exception:
	le = LabelEncoder()
	y = le.fit_transform(y)

	X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

	# Scale AFTER split (same as training)
	if scaler:
	X_train = scaler.transform(X_train)
	X_test = scaler.transform(X_test)

	y_pred = model.predict(X_test)

	if is_clf:
	print(f"\\n📊 Deep Learning Classification Metrics (re-evaluated):")
	print_classification_metrics(y_test, y_pred, model=model, X_test=X_test,
	target_encoder=label_encoder)
	else:
	print(f"\\n📊 Deep Learning Regression Metrics (re-evaluated):")
	print_regression_metrics(y_test, y_pred)
	else:
	# =====================================================================
	# Traditional ML Re-evaluation
	# =====================================================================
	X = df.drop(columns=[target_col])
	y = df[target_col]

	target_encoder = state.get('target_encoder')
	if is_clf and target_encoder:
	y = pd.Series(target_encoder.transform(y))

	label_encoders = state.get('label_encoders', {{}})
	for col in X.select_dtypes(include=['object', 'category']).columns:
	if col in label_encoders:
	try:
	X[col] = label_encoders[col].transform(X[col].astype(str))
	except:
	le = LabelEncoder()
	X[col] = le.fit_transform(X[col].astype(str))
	else:
	le = LabelEncoder()
	X[col] = le.fit_transform(X[col].astype(str))

	X = X.fillna(0)

	scaler = state.get('scaler')
	if scaler and feature_cols:
	avail = [c for c in feature_cols if c in X.columns]
	if avail:
	X = X[avail]
	X = pd.DataFrame(scaler.transform(X), columns=avail)

	# Align columns with model's expected features
	if feature_cols:
	for col in feature_cols:
	if col not in X.columns:
	X[col] = 0
	X = X.reindex(columns=feature_cols, fill_value=0)

	X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

	y_pred = model.predict(X_test)
	y_pred_train = model.predict(X_train)

	if is_clf:
	print(f"\\n📊 Classification Metrics (re-evaluated):")
	print_classification_metrics(y_test, y_pred, model=model, X_test=X_test,
	target_encoder=state.get('target_encoder'))

	# Overfitting check
	train_acc = accuracy_score(y_train, y_pred_train)
	test_acc = accuracy_score(y_test, y_pred)
	gap = train_acc - test_acc
	if gap > 0.1:
	print(f"\\n⚠️ Overfitting detected (train-test gap: {{gap:.4f}})")
	elif gap < 0.01:
	print(f"\\n✅ Good generalization (train-test gap: {{gap:.4f}})")

	try:
	cv_scores = cross_val_score(model, X, y, cv=args.cv, scoring='accuracy')
	print(f"\\n🔄 {{args.cv}}-Fold CV Accuracy: {{cv_scores.mean():.4f}} ± {{cv_scores.std():.4f}}")
	except Exception as e:
	print(f" CV failed: {{e}}")
	else:
	print(f"\\n📊 Regression Metrics (re-evaluated):")
	print_regression_metrics(y_test, y_pred)

	try:
	cv_scores = cross_val_score(model, X, y, cv=args.cv, scoring='r2')
	print(f"\\n🔄 {{args.cv}}-Fold CV R²: {{cv_scores.mean():.4f}} ± {{cv_scores.std():.4f}}")
	except:
	pass

	print("\\n✅ Evaluation complete!")


	if __name__ == '__main__':
	main()
	'''


	# =============================================================================
	# API SERVER — DATA-AWARE
	# =============================================================================

	def _generate_api_server_script(config: Dict) -> str:
	target = config['target_column']
	task_type = config['task_type']
	features = config.get('feature_columns', [])
	best_model = config.get('best_model_name', 'Unknown')
	numeric_features = config.get('numeric_features', [])
	categorical_features = config.get('categorical_features', [])
	feature_metadata = config.get('feature_metadata', [])

	features_list = ", ".join(f"'{f}'" for f in features[:30])

	# Build raw feature info (original columns user understands)
	raw_feature_info_lines = []
	for fm in feature_metadata[:30]:
	name = fm.get('name', '')
	ftype = fm.get('type', 'numeric')
	if ftype == 'numeric':
	mn = fm.get('min', '?')
	mx = fm.get('max', '?')
	mean_val = fm.get('mean', 0)
	raw_feature_info_lines.append(f' "{name}": {{"type": "numeric", "min": {mn}, "max": {mx}, "default": {mean_val:.2f}}},')
	elif ftype == 'categorical':
	options = fm.get('options', [])
	opts_str = str(options[:10])
	raw_feature_info_lines.append(f' "{name}": {{"type": "categorical", "options": {opts_str}}},')
	elif ftype == 'text':
	raw_feature_info_lines.append(f' "{name}": {{"type": "text"}},')

	if not raw_feature_info_lines and features:
	for f in features[:30]:
	raw_feature_info_lines.append(f' "{f}": {{"type": "numeric", "default": 0}},')

	raw_features_block = "\n".join(raw_feature_info_lines)

	return f'''"""
	🌐 Flask API Server — Your Trained Model
	============================================
	Auto-generated by DataVision AI

	Model : {best_model}
	Target Column : {target}
	Task Type : {task_type}
	Features : {len(features)} columns

	Usage:
	python api_server.py
	python api_server.py --port 8080

	Endpoints:
	POST /predict — Make a prediction (JSON body with your feature names)
	GET /health — Health check
	GET /model-info — Model metadata and expected features
	"""

	import os
	import json
	import re
	import pickle
	import argparse
	import numpy as np
	import pandas as pd
	from flask import Flask, request, jsonify, render_template
	from sklearn.preprocessing import LabelEncoder, RobustScaler

	app = Flask(__name__)

	MODEL_STATE = None
	FEATURE_ENGINEER = None
	FEATURE_COLUMNS = [{features_list}]
	TARGET_COLUMN = '{target}'
	BEST_MODE = '{config.get("best_mode", "traditional")}' # Which mode won during AutoML

	# YOUR original column names with types and ranges (what you input)
	RAW_FEATURE_INFO = {{
	{raw_features_block}
	}}


	# =============================================================================
	# Standalone Feature Engineer — exact DataVision preprocessing
	# =============================================================================
	class StandaloneFeatureEngineer:
	def __init__(self, state):
	self.transformers = state.get('transformers', {{}})
	self.encoders = state.get('encoders', {{}})
	self.feature_names = state.get('feature_names', [])
	self.original_columns = state.get('original_columns', [])
	self.selected_feature_indices = state.get('selected_feature_indices', None)
	self.is_fitted = state.get('is_fitted', True)
	self.stopwords = set("i me my myself we our ours ourselves you your yours yourself yourselves he him his himself she her hers herself it its itself they them their theirs themselves what which who whom this that these those am is are was were be been being have has had having do does did doing a an the and but if or because as until while of at by for with about against between through during before after above below to from up down in out on off over under again further then once here there when where why how all both each few more most other some such no nor not only own same so than too very s t can will just don should now d ll m o re ve y ain aren couldn didn doesn hadn hasn haven isn ma mightn mustn needn shan shouldn wasn weren won wouldn".split())
	self.positive_words = set("good great excellent amazing wonderful fantastic brilliant outstanding superb love best perfect beautiful awesome incredible impressive magnificent exceptional fabulous terrific marvelous stellar remarkable extraordinary delightful gorgeous splendid spectacular phenomenal exquisite lovely charming pleasant nice fine happy joy enjoy satisfied pleased glad cheerful excited thrilled".split())
	self.negative_words = set("bad terrible horrible awful worst poor disappointing disgusting pathetic useless dreadful atrocious appalling abysmal mediocre inferior unpleasant nasty vile wretched miserable lousy horrendous hideous ghastly hideous deplorable lamentable pitiful woeful tragic sad angry frustrated annoyed irritated disappointed upset unhappy distressed sorrowful gloomy depressed".split())

	def _clean_text(self, text):
	text = str(text).lower()
	text = re.sub(r'<[^>]+>', ' ', text)
	text = re.sub(r'http\\S+', ' ', text)
	text = re.sub(r'[^a-z0-9\\s]', ' ', text)
	text = ' '.join(w for w in text.split() if w not in self.stopwords and len(w) > 1)
	return text if text.strip() else 'empty'

	def _text_statistics(self, text):
	text = str(text)
	words = text.split()
	wl = [len(w) for w in words] if words else [0]
	return [len(text), len(words), np.mean(wl), np.std(wl), max(wl) if wl else 0,
	len(set(words)) / max(len(words), 1), text.count('!'), text.count('?'),
	sum(1 for c in text if c.isupper()) / max(len(text), 1),
	len([w for w in words if len(w) > 6]) / max(len(words), 1)]

	def _sentiment_score(self, text):
	words = set(str(text).lower().split())
	pos = len(words & self.positive_words)
	neg = len(words & self.negative_words)
	total = max(pos + neg, 1)
	return [pos / total, neg / total, (pos - neg) / total, total]

	def transform(self, df):
	df = df.copy()
	cols = df.columns.tolist()
	seen = {{}}
	new_cols = []
	for col in cols:
	if col in seen:
	seen[col] += 1
	new_cols.append(f"{{col}}_{{seen[col]}}")
	else:
	seen[col] = 0
	new_cols.append(col)
	df.columns = new_cols
	feature_parts = []

	if 'numeric_cols' in self.transformers:
	numeric_cols = self.transformers['numeric_cols']
	for col in numeric_cols:
	if col not in df.columns:
	df[col] = 0
	try:
	numeric_data = df[numeric_cols].apply(pd.to_numeric, errors='coerce').fillna(0).values
	scaler = self.transformers['numeric_scaler']
	numeric_scaled = scaler.transform(numeric_data)
	numeric_scaled = np.nan_to_num(numeric_scaled, nan=0.0, posinf=0.0, neginf=0.0)
	feature_parts.append(numeric_scaled)
	if 2 <= len(numeric_cols) <= 10:
	interactions = []
	for i in range(min(len(numeric_cols), 4)):
	interactions.append((numeric_scaled[:, i] ** 2).reshape(-1, 1))
	for i in range(min(len(numeric_cols), 4)):
	for j in range(i + 1, min(len(numeric_cols), 4)):
	interactions.append((numeric_scaled[:, i] * numeric_scaled[:, j]).reshape(-1, 1))
	if interactions:
	feature_parts.append(np.hstack(interactions))
	except:
	feature_parts.append(np.zeros((len(df), len(numeric_cols))))

	for key, cols in self.transformers.items():
	if key.endswith('_onehot'):
	col_name = key.replace('_onehot', '')
	try:
	if col_name in df.columns:
	series = df[col_name].fillna('_MISSING_').astype(str).str.strip()
	dummies = pd.get_dummies(series, prefix=col_name)
	dummies = dummies.reindex(columns=cols, fill_value=0)
	feature_parts.append(dummies.values.astype(float))
	else:
	feature_parts.append(np.zeros((len(df), len(cols))))
	except:
	feature_parts.append(np.zeros((len(df), len(cols))))

	for col, le in self.encoders.items():
	try:
	if col in df.columns:
	series = df[col].fillna('_MISSING_').astype(str).str.strip()
	encoded = series.map(lambda s: le.transform([s])[0] if s in le.classes_ else -1)
	feature_parts.append(encoded.values.reshape(-1, 1).astype(float))
	freq_key = f'{{col}}_freq_map'
	if freq_key in self.transformers:
	freq_map = self.transformers[freq_key]
	freq_encoded = series.map(lambda s: freq_map.get(s, 0.0)).values.astype(float)
	feature_parts.append(freq_encoded.reshape(-1, 1))
	else:
	feature_parts.append(np.zeros((len(df), 1)))
	if f'{{col}}_freq_map' in self.transformers:
	feature_parts.append(np.zeros((len(df), 1)))
	except:
	feature_parts.append(np.zeros((len(df), 1)))

	if 'cat_interactions' in self.transformers:
	for key, value in self.transformers['cat_interactions'].items():
	if key.endswith('_freq_map'):
	continue
	try:
	col1, col2, le = value
	if col1 in df.columns and col2 in df.columns:
	combined = df[col1].fillna('_NA_').astype(str) + "_X_" + df[col2].fillna('_NA_').astype(str)
	encoded = combined.map(lambda s: le.transform([s])[0] if s in le.classes_ else -1)
	feature_parts.append(encoded.values.reshape(-1, 1).astype(float))
	fk = f"{{key}}_freq_map"
	if fk in self.transformers['cat_interactions']:
	fm = self.transformers['cat_interactions'][fk]
	feature_parts.append(combined.map(lambda s: fm.get(s, 0.0)).values.reshape(-1, 1).astype(float))
	else:
	feature_parts.append(np.zeros((len(df), 1)))
	except:
	pass

	if 'num_cat_aggs' in self.transformers:
	for key, mean_map in self.transformers['num_cat_aggs'].items():
	try:
	parts = key.rsplit('_by_', 1)
	if len(parts) == 2:
	num_col, cat_col = parts[0], parts[1].rsplit('_mean', 1)[0]
	if num_col in df.columns and cat_col in df.columns:
	cat_s = df[cat_col].fillna('_NA_').astype(str)
	dv = np.mean(list(mean_map.values())) if mean_map else 0
	agg = cat_s.map(lambda s: mean_map.get(s, dv)).values.astype(float)
	feature_parts.append(agg.reshape(-1, 1))
	feature_parts.append((df[num_col].fillna(0).values - agg).reshape(-1, 1))
	else:
	feature_parts.extend([np.zeros((len(df), 1))] * 2)
	except:
	pass

	for key, tfidf in self.transformers.items():
	if key.endswith('_tfidf'):
	col = key.replace('_tfidf', '')
	try:
	if col in df.columns:
	series = df[col].fillna('').astype(str)
	cleaned = series.apply(self._clean_text)
	tm = tfidf.transform(cleaned)
	svd_key = f'{{col}}_svd'
	feature_parts.append(self.transformers[svd_key].transform(tm) if svd_key in self.transformers else tm.toarray())
	feature_parts.append(np.array([self._text_statistics(t) for t in series]))
	feature_parts.append(np.array([self._sentiment_score(t) for t in series]))
	except:
	pass

	if not feature_parts:
	return np.zeros((len(df), len(self.feature_names)))
	X = np.hstack(feature_parts)
	X = np.nan_to_num(X, nan=0.0, posinf=0.0, neginf=0.0)

	if 'variance_selector' in self.transformers:
	sel = self.transformers['variance_selector']
	try:
	if X.shape[1] == sel.n_features_in_:
	X = sel.transform(X)
	elif self.selected_feature_indices is not None:
	vi = [i for i in self.selected_feature_indices if i < X.shape[1]]
	if vi:
	X = X[:, vi]
	except:
	pass

	if 'pca' in self.transformers:
	try:
	pca = self.transformers['pca']
	if X.shape[1] == pca.n_features_in_:
	X = pca.transform(X)
	except:
	pass

	if X.shape[1] != len(self.feature_names):
	if X.shape[1] < len(self.feature_names):
	X = np.hstack([X, np.zeros((len(df), len(self.feature_names) - X.shape[1]))])
	else:
	X = X[:, :len(self.feature_names)]
	return X

	def transform_single(self, data):
	for col in self.original_columns:
	if col not in data:
	data[col] = 0
	df = pd.DataFrame([data])
	X = self.transform(df)
	n = len(self.feature_names)
	if X.shape[1] < n:
	X = np.hstack([X, np.zeros((1, n - X.shape[1]))])
	elif X.shape[1] > n:
	X = X[:, :n]
	return X


	def load_model():
	global MODEL_STATE, FEATURE_ENGINEER

	# If best mode is deep_learning, load DL first as primary
	if BEST_MODE == 'deep_learning':
	load_dl_model()
	if DL_STATE is not None:
	print(f"🧠 Primary mode: Deep Learning (BEST_MODE={{BEST_MODE}})")
	# Also try loading ML model as fallback
	for path in ['model.pkl', 'nlp_model.pkl']:
	if os.path.exists(path):
	with open(path, 'rb') as f:
	MODEL_STATE = pickle.load(f)
	print(f" Also loaded fallback: {{path}}")
	break
	return True

	# Traditional/NLP: load ML model first
	for path in ['model.pkl', 'nlp_model.pkl']:
	if os.path.exists(path):
	with open(path, 'rb') as f:
	MODEL_STATE = pickle.load(f)
	print(f"✅ Loaded: {{path}} ({{MODEL_STATE.get('model_name', 'Unknown')}})")
	# Load feature engineer for exact preprocessing
	eng_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'feature_engineer.pkl')
	if os.path.exists(eng_path):
	try:
	with open(eng_path, 'rb') as ef:
	eng_state = pickle.load(ef)
	FEATURE_ENGINEER = StandaloneFeatureEngineer(eng_state)
	print(f" Pipeline: feature_engineer.pkl ({{len(FEATURE_ENGINEER.feature_names)}} features)")
	except Exception as e:
	print(f" Warning: Could not load feature_engineer.pkl: {{e}}")
	break

	# Also load DL model if available
	load_dl_model()

	if MODEL_STATE is None and DL_STATE is None:
	print("❌ No model file found")
	return False
	return True


	# =============================================================================
	# Deep Learning Model Support (sklearn MLPClassifier/MLPRegressor)
	# =============================================================================
	DL_STATE = None

	def load_dl_model():
	"""Load the Deep Learning model if available."""
	global DL_STATE
	dl_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'deep_learning_model.pkl')
	if os.path.exists(dl_path):
	try:
	with open(dl_path, 'rb') as f:
	DL_STATE = pickle.load(f)
	print(f"✅ Loaded DL model: {{DL_STATE.get('algorithm', 'MLP')}}")
	except Exception as e:
	print(f"⚠️ Failed to load DL model: {{e}}")


	def preprocess_dl_input(data):
	"""Preprocess input for Deep Learning model using feature_metadata."""
	feature_metadata = DL_STATE.get('feature_metadata', [])
	feature_values = []

	# Process numeric features
	for meta in feature_metadata:
	if meta.get('type') == 'numeric':
	col = meta['name']
	if col in data:
	try:
	feature_values.append(float(data[col]))
	except (ValueError, TypeError):
	feature_values.append(meta.get('mean', 0))
	else:
	feature_values.append(meta.get('mean', 0))

	# Process categorical features (one-hot)
	for meta in feature_metadata:
	if meta.get('type') == 'categorical':
	col = meta['name']
	value = str(data.get(col, ''))
	options = meta.get('options', [])
	for opt in options:
	feature_values.append(1.0 if value == opt else 0.0)
	feature_values.append(1.0 if not value or value == 'nan' else 0.0)

	# Fallback
	if not feature_values and DL_STATE.get('feature_columns'):
	for col in DL_STATE['feature_columns']:
	if col in data:
	try:
	feature_values.append(float(data[col]) if not isinstance(data[col], str) else 0)
	except:
	feature_values.append(0)
	else:
	feature_values.append(0)

	X = np.array([feature_values])

	scaler = DL_STATE.get('scaler')
	if scaler and hasattr(scaler, 'n_features_in_'):
	expected = scaler.n_features_in_
	actual = X.shape[1]
	if actual < expected:
	X = np.hstack([X, np.zeros((1, expected - actual))])
	elif actual > expected:
	X = X[:, :expected]

	if scaler:
	X = scaler.transform(X)

	return X


	def preprocess_input(data):
	"""Preprocess raw input using the EXACT same pipeline as DataVision AutoML."""
	global FEATURE_ENGINEER

	# Use feature engineer (exact AutoML pipeline)
	if FEATURE_ENGINEER is not None:
	try:
	return FEATURE_ENGINEER.transform_single(data)
	except Exception as e:
	print(f"Warning: Feature engineer failed ({{e}}), using fallback")

	# Fallback: basic label-encode + scale
	feature_cols = MODEL_STATE.get('feature_columns', FEATURE_COLUMNS)
	label_encoders = MODEL_STATE.get('label_encoders', {{}})
	scaler = MODEL_STATE.get('scaler')

	row = {{}}
	for col in feature_cols:
	val = data.get(col, 0)
	if col in label_encoders:
	le = label_encoders[col]
	val_str = str(val)
	if val_str in le.classes_:
	val = le.transform([val_str])[0]
	else:
	val = 0
	else:
	try:
	val = float(val)
	except (ValueError, TypeError):
	val = 0
	row[col] = val

	X = pd.DataFrame([row], columns=feature_cols)
	if scaler:
	try:
	X = pd.DataFrame(scaler.transform(X), columns=feature_cols)
	except:
	pass
	return X


	@app.route('/health', methods=['GET'])
	def health():
	return jsonify({{"status": "healthy", "model_loaded": MODEL_STATE is not None or DL_STATE is not None, "dl_model_loaded": DL_STATE is not None}})


	@app.route('/model-info', methods=['GET'])
	def model_info():
	if MODEL_STATE is None and DL_STATE is None:
	return jsonify({{"error": "No model loaded"}}), 404

	info = {{"best_mode": BEST_MODE}}
	if BEST_MODE == 'deep_learning' and DL_STATE:
	info.update({{
	"model_name": DL_STATE.get('algorithm', 'MLP Neural Network'),
	"task_type": DL_STATE.get('task_type', '{task_type}'),
	"target_column": TARGET_COLUMN,
	"input_features": list(RAW_FEATURE_INFO.keys()),
	"metrics": {{k: float(v) if isinstance(v, (int, float)) else str(v)
	for k, v in DL_STATE.get('metrics', {{}}).items()}},
	}})
	elif MODEL_STATE:
	info.update({{
	"model_name": MODEL_STATE.get('model_name', 'Unknown'),
	"task_type": MODEL_STATE.get('task_type', '{task_type}'),
	"target_column": TARGET_COLUMN,
	"input_features": list(RAW_FEATURE_INFO.keys()),
	"metrics": {{k: float(v) if isinstance(v, (int, float)) else str(v)
	for k, v in MODEL_STATE.get('metrics', {{}}).items()}},
	}})
	if DL_STATE:
	info["dl_model"] = {{
	"algorithm": DL_STATE.get('algorithm', 'MLP'),
	"task_type": DL_STATE.get('task_type', '{task_type}'),
	"metrics": {{k: float(v) if isinstance(v, (int, float)) else str(v)
	for k, v in DL_STATE.get('metrics', {{}}).items()}},
	}}
	return jsonify(info)


	@app.route('/predict', methods=['GET', 'POST'])
	def predict():
	if MODEL_STATE is None and DL_STATE is None:
	return jsonify({{"error": "No model loaded"}}), 500

	# GET request: show web form for interactive prediction
	if request.method == 'GET':
	if BEST_MODE == 'deep_learning' and DL_STATE:
	model_name = DL_STATE.get('algorithm', 'MLP Neural Network')
	task_type_val = DL_STATE.get('task_type', '{task_type}')
	metrics = DL_STATE.get('metrics', {{}})
	elif MODEL_STATE:
	model_name = MODEL_STATE.get('model_name', 'Unknown')
	task_type_val = MODEL_STATE.get('task_type', '{task_type}')
	metrics = MODEL_STATE.get('metrics', {{}})
	else:
	model_name = 'Unknown'
	task_type_val = '{task_type}'
	metrics = {{}}

	# Serve the UI using the extracted index.html template
	return render_template("index.html",
	model_name=model_name,
	target=TARGET_COLUMN,
	task_type_val=task_type_val,
	metrics=metrics,
	raw_feature_info=RAW_FEATURE_INFO)

	# POST request: JSON API prediction
	try:
	data = request.get_json()
	if not data:
	return jsonify({{"error": "No JSON body. Send features as JSON.", "expected_features": list(RAW_FEATURE_INFO.keys())}}), 400

	# Check if caller wants DL mode, or if BEST_MODE is deep_learning, or if only DL is available
	req_mode = data.pop('_mode', None)
	use_dl = (req_mode == 'dl') or (req_mode != 'ml' and BEST_MODE == 'deep_learning' and DL_STATE is not None) or (MODEL_STATE is None and DL_STATE is not None)

	if use_dl and DL_STATE:
	# Deep Learning prediction
	X = preprocess_dl_input(data)
	dl_model = DL_STATE['model']
	label_encoder = DL_STATE.get('label_encoder')
	task_type = DL_STATE.get('task_type', '{task_type}')
	classes = DL_STATE.get('classes', [])

	pred = dl_model.predict(X)[0]
	result = {{"raw_prediction": int(pred) if isinstance(pred, (np.integer,)) else float(pred)}}

	if 'classif' in task_type.lower() and label_encoder:
	try:
	result['prediction'] = str(label_encoder.inverse_transform([int(pred)])[0])
	except:
	result['prediction'] = str(pred)
	else:
	result['prediction'] = float(pred)

	if 'classif' in task_type.lower() and hasattr(dl_model, 'predict_proba'):
	try:
	proba = dl_model.predict_proba(X)[0]
	result['confidence'] = float(max(proba))
	if classes:
	result['probabilities'] = {{
	str(c): round(float(p), 4) for c, p in zip(classes, proba)
	}}
	except:
	pass

	result['mode'] = 'deep_learning'
	return jsonify(result)

	# Traditional/NLP prediction
	model = MODEL_STATE['model']
	target_encoder = MODEL_STATE.get('target_encoder')
	task_type = MODEL_STATE.get('task_type', '{task_type}')

	X = preprocess_input(data)

	prediction = model.predict(X)[0]
	result = {{"raw_prediction": int(prediction) if isinstance(prediction, (np.integer,)) else float(prediction)}}

	if 'classif' in task_type.lower() and target_encoder:
	try:
	result['prediction'] = str(target_encoder.inverse_transform([int(prediction)])[0])
	except:
	result['prediction'] = str(prediction)
	else:
	result['prediction'] = float(prediction)

	if 'classif' in task_type.lower() and hasattr(model, 'predict_proba'):
	try:
	proba = model.predict_proba(X)[0]
	result['confidence'] = float(max(proba))
	if target_encoder:
	result['probabilities'] = {{
	str(c): round(float(p), 4) for c, p in zip(target_encoder.classes_, proba)
	}}
	except:
	pass

	return jsonify(result)
	except Exception as e:
	return jsonify({{"error": str(e)}}), 500


	if __name__ == '__main__':
	parser = argparse.ArgumentParser()
	parser.add_argument('--port', type=int, default=5000)
	parser.add_argument('--host', type=str, default='0.0.0.0')
	args = parser.parse_args()

	if load_model():
	print(f"\\n🌐 API Server on http://{{args.host}}:{{args.port}}")
	print(f" POST /predict — Send JSON with your feature values")
	print(f" GET /health — Health check")
	print(f" GET /model-info — Your model details and expected features")
	app.run(host=args.host, port=args.port, debug=False)
	'''


	# =============================================================================
	# PREPROCESSING UTILS — DATA-AWARE
	# =============================================================================

	def _generate_preprocessing_utils(config: Dict) -> str:
	target = config['target_column']
	task_type = config['task_type']
	features = config.get('feature_columns', [])
	features_list = ", ".join(f"'{f}'" for f in features[:30])

	return f'''"""
	🔧 Preprocessing Utilities — Mirrors DataVision AutoML Pipeline
	===================================================================
	These utilities replicate the EXACT preprocessing pipeline used by
	DataVision's ProductionDataCleaner + ProductionFeatureEngineer.

	Pipeline steps:
	1. Remove useless columns (IDs, URLs, high-cardinality strings)
	2. Remove duplicates
	3. Smart type conversion (parse $, M/K units, commas)
	4. KNN imputation for numerics (median fallback)
	5. Mode fill for categoricals
	6. Replace inf/-inf with median
	7. Remove constant columns
	8. IQR-based outlier capping
	9. Log-transform skewed features
	10. RobustScaler for numeric features
	11. Categorical encoding (binary→label, low-card→one-hot, high-card→label+freq)
	12. VarianceThreshold to remove near-constant features
	"""

	import numpy as np
	import pandas as pd
	from sklearn.preprocessing import LabelEncoder, RobustScaler
	from sklearn.feature_selection import VarianceThreshold
	from sklearn.impute import KNNImputer

	TARGET_COLUMN = '{target}'
	TASK_TYPE = '{task_type}'
	FEATURE_COLUMNS = [{features_list}]


	def detect_task_type(y):
	"""Auto-detect classification vs regression."""
	if y.dtype == 'object' or y.nunique() < 20:
	return 'classification'
	return 'regression'


	def remove_useless_columns(df, target_col=TARGET_COLUMN):
	"""Remove ID-like, URL, and high-cardinality columns (same as DataVision)."""
	id_patterns = ['unnamed', 'index', '_id', 'guid', 'uuid', 'url', 'link', 'href', 'path']
	cols_to_drop = []
	for col in df.columns:
	if col == target_col:
	continue
	col_lower = col.lower().strip()
	if any(pat in col_lower for pat in id_patterns):
	cols_to_drop.append(col)
	elif df[col].dtype == 'object' and df[col].nunique() / max(len(df), 1) > 0.9:
	cols_to_drop.append(col)
	return df.drop(columns=cols_to_drop, errors='ignore') if cols_to_drop else df


	def smart_type_conversion(df, target_col=TARGET_COLUMN):
	"""Parse hidden numerics: $, M/K units, commas (same as DataVision)."""
	for col in df.select_dtypes(include=['object']).columns:
	if col == target_col:
	continue
	sample = df[col].dropna().head(20)
	if len(sample) == 0:
	continue
	converted = 0
	for val in sample:
	try:
	v = str(val).replace('$', '').replace(',', '').strip()
	if v.upper().endswith(('M', 'K', '+')):
	float(v[:-1])
	else:
	float(v)
	converted += 1
	except:
	pass
	if converted / max(len(sample), 1) > 0.6:
	def parse_val(val):
	if pd.isna(val):
	return np.nan
	v = str(val).replace('$', '').replace(',', '').strip()
	try:
	if v.upper().endswith('M'): return float(v[:-1]) * 1e6
	if v.upper().endswith('K'): return float(v[:-1]) * 1e3
	if v.endswith('+'): return float(v[:-1])
	return float(v)
	except:
	return np.nan
	df[col] = df[col].apply(parse_val)
	return df


	def impute_missing(df, target_col=TARGET_COLUMN):
	"""KNN imputation for numerics, mode fill for categoricals (same as DataVision)."""
	numeric_cols = [c for c in df.select_dtypes(include=[np.number]).columns if c != target_col]
	if numeric_cols and df[numeric_cols].isna().any().any():
	try:
	imputer = KNNImputer(n_neighbors=5, weights='distance')
	df[numeric_cols] = imputer.fit_transform(df[numeric_cols])
	except:
	for col in numeric_cols:
	df[col] = df[col].fillna(df[col].median())

	for col in df.select_dtypes(include=['object', 'category']).columns:
	if col == target_col:
	continue
	mode = df[col].mode()
	df[col] = df[col].fillna(mode.iloc[0] if len(mode) > 0 else '_MISSING_')
	return df


	def cap_outliers_iqr(df, target_col=TARGET_COLUMN):
	"""IQR-based outlier capping with [1st, 99th] percentile bounds (same as DataVision)."""
	for col in df.select_dtypes(include=[np.number]).columns:
	if col == target_col:
	continue
	Q1, Q3 = df[col].quantile(0.25), df[col].quantile(0.75)
	IQR = Q3 - Q1
	p1, p99 = df[col].quantile(0.01), df[col].quantile(0.99)
	lower = max(p1, Q1 - 3 * IQR)
	upper = min(p99, Q3 + 3 * IQR)
	df[col] = df[col].clip(lower=lower, upper=upper)
	return df


	def engineer_features(X):
	"""Feature engineering matching DataVision's ProductionFeatureEngineer."""
	label_encoders = {{}}
	numeric_cols = X.select_dtypes(include=[np.number]).columns.tolist()
	categorical_cols = X.select_dtypes(include=['object', 'category']).columns.tolist()

	# Log-transform skewed numerics (skewness > 2, non-negative)
	for col in numeric_cols:
	if (X[col] >= 0).all() and abs(X[col].skew()) > 2:
	X[f'{{col}}_log'] = np.log1p(X[col])

	# RobustScaler (same as DataVision, not StandardScaler)
	numeric_cols_all = X.select_dtypes(include=[np.number]).columns.tolist()
	scaler = RobustScaler()
	if numeric_cols_all:
	X[numeric_cols_all] = scaler.fit_transform(X[numeric_cols_all])

	# Polynomial features (top 4 numeric, squared + interactions)
	if 2 <= len(numeric_cols) <= 10:
	top_n = min(4, len(numeric_cols))
	top_cols = numeric_cols[:top_n]
	for col in top_cols:
	X[f'{{col}}_sq'] = X[col] ** 2
	for i in range(len(top_cols)):
	for j in range(i + 1, len(top_cols)):
	X[f'{{top_cols[i]}}_x_{{top_cols[j]}}'] = X[top_cols[i]] * X[top_cols[j]]

	# Categorical encoding
	for col in categorical_cols:
	nuniq = X[col].nunique()
	if nuniq <= 2: # Binary → LabelEncoder
	le = LabelEncoder()
	X[col] = le.fit_transform(X[col].astype(str))
	label_encoders[col] = le
	elif nuniq <= 10: # Low cardinality → one-hot
	dummies = pd.get_dummies(X[col], prefix=col, drop_first=False)
	X = pd.concat([X.drop(columns=[col]), dummies], axis=1)
	else: # High cardinality → label + frequency encoding
	le = LabelEncoder()
	X[f'{{col}}_label'] = le.fit_transform(X[col].astype(str))
	freq = X[col].value_counts(normalize=True)
	X[f'{{col}}_freq'] = X[col].map(freq)
	label_encoders[col] = le
	X = X.drop(columns=[col])

	# Force float + clean
	for col in X.columns:
	try:
	X[col] = X[col].astype(float)
	except:
	le = LabelEncoder()
	X[col] = le.fit_transform(X[col].astype(str)).astype(float)
	X = X.replace([np.inf, -np.inf], 0).fillna(0)

	# VarianceThreshold (remove near-constant, threshold=0.01)
	try:
	vt = VarianceThreshold(threshold=0.01)
	X = pd.DataFrame(vt.fit_transform(X), columns=X.columns[vt.get_support()], index=X.index)
	except:
	pass

	return X, label_encoders, scaler


	def full_preprocess(df, target_column=TARGET_COLUMN):
	"""Complete DataVision preprocessing pipeline.

	Combines ProductionDataCleaner.clean() + ProductionFeatureEngineer.fit_transform().
	"""
	# Step 1: Clean
	df = df.dropna(subset=[target_column])
	df = df.drop_duplicates()

	# Strip whitespace and replace string NaN variants
	for col in df.select_dtypes(include=['object']).columns:
	df[col] = df[col].str.strip()
	df[col] = df[col].replace(['nan', 'NaN', 'None', 'null', 'NULL', ''], np.nan)

	df = remove_useless_columns(df, target_column)
	df = smart_type_conversion(df, target_column)
	df = impute_missing(df, target_column)

	# Replace inf, remove constant columns
	for col in df.select_dtypes(include=[np.number]).columns:
	if col != target_column:
	df[col] = df[col].replace([np.inf, -np.inf], df[col].replace([np.inf, -np.inf], np.nan).median())
	const_cols = [c for c in df.columns if c != target_column and df[c].nunique() <= 1]
	df = df.drop(columns=const_cols, errors='ignore')

	df = cap_outliers_iqr(df, target_column)
	df[df.select_dtypes(include=[np.number]).columns] = df.select_dtypes(include=[np.number]).fillna(0)

	# Step 2: Split target
	X = df.drop(columns=[target_column])
	y = df[target_column]

	task_type = detect_task_type(y)
	target_encoder = None
	if task_type == 'classification':
	target_encoder = LabelEncoder()
	y = pd.Series(target_encoder.fit_transform(y))

	# Step 3: Engineer features
	X, label_encoders, scaler = engineer_features(X)
	feature_cols = X.columns.tolist()

	return X, y, task_type, target_encoder, label_encoders, scaler, feature_cols
	'''


	# =============================================================================
	# DOCKERFILE
	# =============================================================================

	def _generate_dockerfile(config: Dict) -> str:
	return '''FROM python:3.10-slim

	WORKDIR /app

	COPY requirements.txt .
	RUN pip install --no-cache-dir -r requirements.txt

	COPY . .

	EXPOSE 5000

	CMD ["python", "api_server.py", "--port", "5000"]
	'''


	# =============================================================================
	# CLUSTERING / UNSUPERVISED LEARNING ZIP GENERATOR
	# =============================================================================

	def generate_clustering_code_zip(
	pkl_path: Path,
	cleaned_data_path: Optional[Path] = None,
	charts_data: Optional[Dict[str, str]] = None,
	clustering_meta: Optional[Dict[str, Any]] = None,
	) -> io.BytesIO:
	"""
	Generate a complete Python clustering project as a ZIP file.
	Similar to supervised ZIP but tailored for unsupervised learning.

	Args:
	pkl_path: Path to clustering_model_*.pkl
	cleaned_data_path: Path to clustered_data_*.csv
	charts_data: Dict of chart_name -> base64 data URI strings
	clustering_meta: Dict with algorithm, n_clusters, silhouette_score, feature_columns, etc.
	"""
	import pickle

	buf = io.BytesIO()
	meta = clustering_meta or {}

	algorithm = meta.get('algorithm', 'kmeans')
	n_clusters = meta.get('n_clusters', 3)
	silhouette = meta.get('silhouette_score', 0)
	feature_columns = meta.get('feature_columns', [])
	cluster_profiles = meta.get('cluster_profiles', {})
	n_samples = meta.get('n_samples', 0)

	with zipfile.ZipFile(buf, 'w', zipfile.ZIP_DEFLATED) as zf:

	# 1. Model PKL
	if pkl_path and pkl_path.exists():
	zf.write(pkl_path, 'clustering_model.pkl')

	# 2. Cleaned data with cluster assignments
	if cleaned_data_path and cleaned_data_path.exists():
	zf.writestr('data/clustered_data.csv', cleaned_data_path.read_bytes())

	# 3. config.json
	config_json = {
	"task_type": "clustering",
	"algorithm": algorithm,
	"n_clusters": n_clusters,
	"silhouette_score": silhouette,
	"feature_columns": feature_columns,
	"n_samples": n_samples,
	"generated_by": "DataVision AutoML",
	}
	zf.writestr('config.json', json.dumps(config_json, indent=2, default=str))

	# 4. predict_cluster.py
	zf.writestr('predict_cluster.py', _generate_clustering_predict_script(
	algorithm, n_clusters, feature_columns
	))

	# 5. train_clustering.py
	zf.writestr('train_clustering.py', _generate_clustering_train_script(
	algorithm, n_clusters, feature_columns
	))

	# 6. visualize_clusters.py
	zf.writestr('visualize_clusters.py', _generate_clustering_visualize_script(
	algorithm, n_clusters, feature_columns
	))

	# 7. requirements.txt
	reqs = (
	"scikit-learn>=1.3.0\n"
	"pandas>=2.0.0\n"
	"numpy>=1.24.0\n"
	"matplotlib>=3.7.0\n"
	"seaborn>=0.12.0\n"
	"flask>=2.3.0\n"
	"joblib>=1.3.0\n"
	)
	zf.writestr('requirements.txt', reqs)

	# 8. api_server.py
	zf.writestr('api_server.py', _generate_clustering_api_script(
	algorithm, n_clusters, feature_columns
	))

	# 9. README.md
	zf.writestr('README.md', _generate_clustering_readme(
	algorithm, n_clusters, silhouette, feature_columns, n_samples, cluster_profiles
	))

	# 10. Dockerfile
	zf.writestr('Dockerfile', _generate_dockerfile({}))

	# 11. Bundle training charts as PNG
	if charts_data:
	charts_added = 0
	for chart_name, chart_value in charts_data.items():
	try:
	if isinstance(chart_value, str) and 'base64,' in chart_value:
	b64_data = chart_value.split('base64,', 1)[1]
	img_bytes = base64.b64decode(b64_data)
	safe_name = chart_name.replace(' ', '_').replace('/', '_').lower()
	if not safe_name.endswith('.png'):
	safe_name += '.png'
	zf.writestr(f'charts/{safe_name}', img_bytes)
	charts_added += 1
	except Exception as e:
	logger.warning(f"Could not bundle chart '{chart_name}': {e}")
	if charts_added > 0:
	logger.info(f"Added {charts_added} clustering charts to charts/ directory")

	buf.seek(0)
	logger.info(f"Generated clustering ZIP: {buf.getbuffer().nbytes / 1024:.1f} KB")
	return buf


	def _generate_clustering_predict_script(algorithm: str, n_clusters: int, feature_columns: List[str]) -> str:
	features_list = ", ".join(f'"{f}"' for f in feature_columns[:20])
	return f'''#!/usr/bin/env python3
	"""
	Predict Cluster Assignment for New Data
	========================================
	Generated by DataVision AutoML - Unsupervised Learning

	Usage:
	python predict_cluster.py # Interactive mode
	python predict_cluster.py --csv input.csv # Batch mode
	"""

	import pickle
	import pandas as pd
	import numpy as np
	import argparse
	import json
	import sys
	from pathlib import Path


	FEATURE_COLUMNS = [{features_list}]


	def load_model(model_path="clustering_model.pkl"):
	"""Load the trained clustering model"""
	with open(model_path, "rb") as f:
	model_data = pickle.load(f)
	return model_data


	def predict_cluster(model_data, features_dict):
	"""Predict cluster for a single data point"""
	scaler = model_data.get("scaler")
	centroids = model_data.get("centroids_scaled")
	feature_cols = model_data.get("feature_columns", FEATURE_COLUMNS)

	# Build feature vector
	values = []
	for col in feature_cols:
	val = features_dict.get(col, 0)
	try:
	values.append(float(val))
	except (ValueError, TypeError):
	values.append(0.0)

	X = np.array(values).reshape(1, -1)

	# Scale
	if scaler is not None:
	X_scaled = scaler.transform(X)
	else:
	X_scaled = X

	# Find nearest centroid
	if centroids is not None:
	centroids_arr = np.array(centroids)
	distances = np.linalg.norm(X_scaled - centroids_arr, axis=1)
	cluster = int(np.argmin(distances))
	confidence = 1.0 / (1.0 + distances[cluster])
	else:
	cluster = 0
	confidence = 0.0

	return {{
	"cluster": cluster,
	"cluster_name": f"Cluster_{{cluster}}",
	"confidence": round(confidence, 4),
	}}


	def predict_batch(model_data, df):
	"""Predict clusters for a batch of data"""
	results = []
	for _, row in df.iterrows():
	features = row.to_dict()
	result = predict_cluster(model_data, features)
	results.append(result)
	return results


	def main():
	parser = argparse.ArgumentParser(description="Predict cluster assignment")
	parser.add_argument("--csv", type=str, help="Path to CSV file for batch prediction")
	parser.add_argument("--model", type=str, default="clustering_model.pkl", help="Model path")
	parser.add_argument("--output", type=str, default="predictions.csv", help="Output CSV path")
	args = parser.parse_args()

	model_data = load_model(args.model)
	feature_cols = model_data.get("feature_columns", FEATURE_COLUMNS)

	if args.csv:
	# Batch mode
	df = pd.read_csv(args.csv)
	print(f"Predicting clusters for {{len(df)}} samples...")
	results = predict_batch(model_data, df)
	df["Predicted_Cluster"] = [r["cluster"] for r in results]
	df["Cluster_Name"] = [r["cluster_name"] for r in results]
	df["Confidence"] = [r["confidence"] for r in results]
	df.to_csv(args.output, index=False)
	print(f"Results saved to {{args.output}}")
	else:
	# Interactive mode
	print("\\n=== Cluster Prediction (Interactive) ===")
	print(f"Algorithm: {algorithm.upper()}, Clusters: {n_clusters}")
	print(f"Features: {{len(feature_cols)}}")
	print()

	features = {{}}
	for col in feature_cols:
	val = input(f" {{col}}: ")
	features[col] = val

	result = predict_cluster(model_data, features)
	print(f"\\n Predicted: {{result[\'cluster_name\']}}")
	print(f" Confidence: {{result[\'confidence\']:.1%}}")


	if __name__ == "__main__":
	main()
	'''


	def _generate_clustering_train_script(algorithm: str, n_clusters: int, feature_columns: List[str]) -> str:
	features_list = ", ".join(f'"{f}"' for f in feature_columns[:20])
	return f'''#!/usr/bin/env python3
	"""
	Re-Train Clustering Model
	==========================
	Generated by DataVision AutoML - Unsupervised Learning

	Reproduces the EXACT clustering from DataVision training.
	Algorithm: {algorithm.upper()}, Clusters: {n_clusters}

	Usage:
	python train_clustering.py # Use default data
	python train_clustering.py --data my_data.csv # Custom data
	python train_clustering.py --algorithm dbscan # Different algorithm
	"""

	import pandas as pd
	import numpy as np
	from sklearn.cluster import KMeans, DBSCAN, AgglomerativeClustering, SpectralClustering
	from sklearn.mixture import GaussianMixture
	from sklearn.preprocessing import StandardScaler
	from sklearn.decomposition import PCA
	from sklearn.metrics import silhouette_score, calinski_harabasz_score, davies_bouldin_score
	import pickle
	import argparse
	import json
	from pathlib import Path


	FEATURE_COLUMNS = [{features_list}]
	DEFAULT_ALGORITHM = "{algorithm}"
	DEFAULT_N_CLUSTERS = {n_clusters}


	def train_clustering(data_path="data/clustered_data.csv", algorithm=DEFAULT_ALGORITHM,
	n_clusters=DEFAULT_N_CLUSTERS, output_path="clustering_model.pkl"):
	"""Train a clustering model"""
	print(f"\\n{'='*60}")
	print(f" CLUSTERING TRAINING")
	print(f" Algorithm: {{algorithm.upper()}}")
	print(f" Target clusters: {{n_clusters}}")
	print(f"{'='*60}\\n")

	# Load data
	df = pd.read_csv(data_path)
	print(f"Loaded {{len(df)}} samples")

	# Select numeric features
	feature_cols = [c for c in FEATURE_COLUMNS if c in df.columns]
	if not feature_cols:
	feature_cols = df.select_dtypes(include=[np.number]).columns.tolist()
	# Remove cluster assignment columns if present
	feature_cols = [c for c in feature_cols if c not in ["Cluster", "PCA_1", "PCA_2"]]

	print(f"Using {{len(feature_cols)}} features: {{feature_cols[:10]}}")

	X = df[feature_cols].fillna(0).values

	# Scale
	scaler = StandardScaler()
	X_scaled = scaler.fit_transform(X)

	# Train model
	if algorithm.lower() == "kmeans":
	model = KMeans(n_clusters=n_clusters, random_state=42, n_init=10)
	labels = model.fit_predict(X_scaled)
	elif algorithm.lower() == "dbscan":
	model = DBSCAN(eps=0.5, min_samples=5)
	labels = model.fit_predict(X_scaled)
	n_clusters = len(set(labels)) - (1 if -1 in labels else 0)
	elif algorithm.lower() == "hierarchical":
	model = AgglomerativeClustering(n_clusters=n_clusters)
	labels = model.fit_predict(X_scaled)
	elif algorithm.lower() == "gmm":
	model = GaussianMixture(n_components=n_clusters, random_state=42)
	labels = model.fit_predict(X_scaled)
	elif algorithm.lower() == "spectral":
	model = SpectralClustering(n_clusters=n_clusters, random_state=42)
	labels = model.fit_predict(X_scaled)
	else:
	print(f"Unknown algorithm: {{algorithm}}. Using KMeans.")
	model = KMeans(n_clusters=n_clusters, random_state=42, n_init=10)
	labels = model.fit_predict(X_scaled)

	# Compute centroids
	centroids = []
	for i in range(max(labels) + 1):
	mask = labels == i
	if mask.sum() > 0:
	centroids.append(X_scaled[mask].mean(axis=0))
	centroids = np.array(centroids) if centroids else np.array([])

	# Metrics
	if len(set(labels)) > 1 and -1 not in labels:
	sil = silhouette_score(X_scaled, labels)
	cal = calinski_harabasz_score(X_scaled, labels)
	dav = davies_bouldin_score(X_scaled, labels)
	else:
	sil, cal, dav = 0, 0, 0

	print(f"\\n Clusters found: {{len(set(labels) - {{-1}})}}")
	print(f" Silhouette Score: {{sil:.4f}}")
	print(f" Calinski-Harabasz: {{cal:.2f}}")
	print(f" Davies-Bouldin: {{dav:.4f}}")

	# Save model
	pkl_data = {{
	"algorithm": algorithm,
	"n_clusters": int(max(labels) + 1) if len(labels) > 0 else n_clusters,
	"scaler": scaler,
	"feature_columns": feature_cols,
	"centroids_scaled": centroids,
	"labels": labels,
	"silhouette_score": sil,
	}}
	with open(output_path, "wb") as f:
	pickle.dump(pkl_data, f)
	print(f"\\n Model saved: {{output_path}}")

	# Save clustered data
	df_out = df.copy()
	df_out["Cluster"] = labels
	pca = PCA(n_components=2)
	X_2d = pca.fit_transform(X_scaled)
	df_out["PCA_1"] = X_2d[:, 0]
	df_out["PCA_2"] = X_2d[:, 1]
	df_out.to_csv("data/clustered_data.csv", index=False)
	print(f" Clustered data saved: data/clustered_data.csv")

	return pkl_data


	def main():
	parser = argparse.ArgumentParser(description="Train clustering model")
	parser.add_argument("--data", type=str, default="data/clustered_data.csv")
	parser.add_argument("--algorithm", type=str, default=DEFAULT_ALGORITHM)
	parser.add_argument("--n-clusters", type=int, default=DEFAULT_N_CLUSTERS)
	parser.add_argument("--output", type=str, default="clustering_model.pkl")
	args = parser.parse_args()

	train_clustering(args.data, args.algorithm, args.n_clusters, args.output)


	if __name__ == "__main__":
	main()
	'''


	def _generate_clustering_visualize_script(algorithm: str, n_clusters: int, feature_columns: List[str]) -> str:
	return f'''#!/usr/bin/env python3
	"""
	Clustering Visualization
	=========================
	Generated by DataVision AutoML - Unsupervised Learning

	Generates:
	- PCA 2D scatter plot colored by cluster
	- Cluster distribution bar chart
	- Silhouette analysis
	- Feature importance for clustering
	- Elbow method chart

	Usage:
	python visualize_clusters.py
	python visualize_clusters.py --data data/clustered_data.csv
	"""

	import pandas as pd
	import numpy as np
	import matplotlib
	matplotlib.use("Agg")
	import matplotlib.pyplot as plt
	import seaborn as sns
	from sklearn.cluster import KMeans
	from sklearn.preprocessing import StandardScaler
	from sklearn.decomposition import PCA
	from sklearn.metrics import silhouette_score, silhouette_samples
	import pickle
	import argparse
	from pathlib import Path


	def generate_all_charts(data_path="data/clustered_data.csv", model_path="clustering_model.pkl",
	output_dir="charts"):
	"""Generate all clustering visualization charts"""
	Path(output_dir).mkdir(parents=True, exist_ok=True)

	# Load data
	df = pd.read_csv(data_path)
	print(f"Loaded {{len(df)}} samples")

	# Load model
	model_data = None
	if Path(model_path).exists():
	with open(model_path, "rb") as f:
	model_data = pickle.load(f)

	feature_cols = model_data.get("feature_columns", []) if model_data else []
	if not feature_cols:
	feature_cols = df.select_dtypes(include=[np.number]).columns.tolist()
	feature_cols = [c for c in feature_cols if c not in ["Cluster", "PCA_1", "PCA_2"]]

	labels = df["Cluster"].values if "Cluster" in df.columns else None

	# Prepare scaled data
	X = df[feature_cols].fillna(0).values
	scaler = StandardScaler()
	X_scaled = scaler.fit_transform(X)

	charts_generated = 0

	# 1. PCA Scatter Plot
	try:
	pca = PCA(n_components=2)
	X_2d = pca.fit_transform(X_scaled)
	fig, ax = plt.subplots(figsize=(10, 8))
	scatter = ax.scatter(X_2d[:, 0], X_2d[:, 1], c=labels, cmap="viridis",
	alpha=0.6, s=20, edgecolors="white", linewidth=0.3)
	plt.colorbar(scatter, ax=ax, label="Cluster")
	ax.set_xlabel(f"PC1 ({{pca.explained_variance_ratio_[0]*100:.1f}}%)")
	ax.set_ylabel(f"PC2 ({{pca.explained_variance_ratio_[1]*100:.1f}}%)")
	ax.set_title("Cluster Scatter Plot (PCA 2D Projection)")
	plt.tight_layout()
	plt.savefig(f"{{output_dir}}/cluster_scatter.png", dpi=150, bbox_inches="tight")
	plt.close()
	charts_generated += 1
	print(f" [1] cluster_scatter.png")
	except Exception as e:
	print(f" [!] scatter plot failed: {{e}}")

	# 2. Cluster Distribution
	try:
	if labels is not None:
	unique, counts = np.unique(labels[labels >= 0], return_counts=True)
	fig, ax = plt.subplots(figsize=(8, 5))
	colors = plt.cm.viridis(np.linspace(0.2, 0.8, len(unique)))
	bars = ax.bar([f"Cluster {{u}}" for u in unique], counts, color=colors, edgecolor="white")
	for bar, count in zip(bars, counts):
	ax.text(bar.get_x() + bar.get_width()/2., bar.get_height() + 0.5,
	str(count), ha="center", va="bottom", fontweight="bold")
	ax.set_xlabel("Cluster")
	ax.set_ylabel("Number of Samples")
	ax.set_title("Cluster Size Distribution")
	plt.tight_layout()
	plt.savefig(f"{{output_dir}}/cluster_distribution.png", dpi=150, bbox_inches="tight")
	plt.close()
	charts_generated += 1
	print(f" [2] cluster_distribution.png")
	except Exception as e:
	print(f" [!] distribution chart failed: {{e}}")

	# 3. Silhouette Analysis
	try:
	if labels is not None and len(set(labels)) > 1 and -1 not in labels:
	sample_silhouette = silhouette_samples(X_scaled, labels)
	fig, ax = plt.subplots(figsize=(10, 7))
	y_lower = 10
	unique_labels = sorted(set(labels))
	colors = plt.cm.viridis(np.linspace(0.2, 0.8, len(unique_labels)))
	for i, label in enumerate(unique_labels):
	cluster_sil = sample_silhouette[labels == label]
	cluster_sil.sort()
	size = cluster_sil.shape[0]
	y_upper = y_lower + size
	ax.fill_betweenx(np.arange(y_lower, y_upper), 0, cluster_sil,
	facecolor=colors[i], alpha=0.7, edgecolor="none")
	ax.text(-0.05, y_lower + 0.5 * size, f"C{{label}}", fontsize=10)
	y_lower = y_upper + 10
	avg_sil = silhouette_score(X_scaled, labels)
	ax.axvline(x=avg_sil, color="red", linestyle="--", label=f"Avg: {{avg_sil:.3f}}")
	ax.set_xlabel("Silhouette Coefficient")
	ax.set_ylabel("Cluster")
	ax.set_title("Silhouette Analysis")
	ax.legend()
	plt.tight_layout()
	plt.savefig(f"{{output_dir}}/silhouette_analysis.png", dpi=150, bbox_inches="tight")
	plt.close()
	charts_generated += 1
	print(f" [3] silhouette_analysis.png")
	except Exception as e:
	print(f" [!] silhouette analysis failed: {{e}}")

	# 4. Elbow Method
	try:
	max_k = min(10, len(X_scaled) - 1)
	if max_k >= 2:
	inertias = []
	sil_scores = []
	K_range = range(2, max_k + 1)
	for k in K_range:
	km = KMeans(n_clusters=k, random_state=42, n_init=10)
	km.fit(X_scaled)
	inertias.append(km.inertia_)
	sil_scores.append(silhouette_score(X_scaled, km.labels_))

	fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 5))
	ax1.plot(list(K_range), inertias, "bo-", linewidth=2)
	ax1.set_xlabel("Number of Clusters (k)")
	ax1.set_ylabel("Inertia")
	ax1.set_title("Elbow Method")
	ax1.grid(alpha=0.3)

	ax2.plot(list(K_range), sil_scores, "ro-", linewidth=2)
	ax2.set_xlabel("Number of Clusters (k)")
	ax2.set_ylabel("Silhouette Score")
	ax2.set_title("Silhouette Score vs k")
	ax2.grid(alpha=0.3)

	plt.tight_layout()
	plt.savefig(f"{{output_dir}}/elbow_method.png", dpi=150, bbox_inches="tight")
	plt.close()
	charts_generated += 1
	print(f" [4] elbow_method.png")
	except Exception as e:
	print(f" [!] elbow method failed: {{e}}")

	# 5. Feature Importance (variance-based)
	try:
	if labels is not None and len(feature_cols) > 1:
	importances = []
	for i, col in enumerate(feature_cols):
	between_var = 0
	overall_mean = X_scaled[:, i].mean()
	for label in set(labels):
	if label < 0:
	continue
	mask = labels == label
	cluster_mean = X_scaled[mask, i].mean()
	between_var += mask.sum() * (cluster_mean - overall_mean) ** 2
	importances.append(between_var)

	importances = np.array(importances)
	if importances.sum() > 0:
	importances = importances / importances.sum()

	# Sort
	sorted_idx = np.argsort(importances)[::-1][:15]
	fig, ax = plt.subplots(figsize=(10, 6))
	bars = ax.barh(
	[feature_cols[i] for i in sorted_idx][::-1],
	importances[sorted_idx][::-1],
	color=plt.cm.viridis(np.linspace(0.3, 0.8, len(sorted_idx))),
	edgecolor="white"
	)
	ax.set_xlabel("Importance (Between-cluster Variance Ratio)")
	ax.set_title("Feature Importance for Clustering")
	plt.tight_layout()
	plt.savefig(f"{{output_dir}}/feature_importance.png", dpi=150, bbox_inches="tight")
	plt.close()
	charts_generated += 1
	print(f" [5] feature_importance.png")
	except Exception as e:
	print(f" [!] feature importance failed: {{e}}")

	print(f"\\n Total charts generated: {{charts_generated}}")
	print(f" Saved to: {{output_dir}}/")


	def main():
	parser = argparse.ArgumentParser(description="Generate clustering charts")
	parser.add_argument("--data", type=str, default="data/clustered_data.csv")
	parser.add_argument("--model", type=str, default="clustering_model.pkl")
	parser.add_argument("--output", type=str, default="charts")
	args = parser.parse_args()

	generate_all_charts(args.data, args.model, args.output)


	if __name__ == "__main__":
	main()
	'''


	def _generate_clustering_api_script(algorithm: str, n_clusters: int, feature_columns: List[str]) -> str:
	features_list = ", ".join(f'"{f}"' for f in feature_columns[:20])
	return f'''#!/usr/bin/env python3
	"""
	Clustering API Server
	======================
	Generated by DataVision AutoML - Unsupervised Learning

	Endpoints:
	GET /predict - Web form for interactive cluster prediction
	POST /predict - Predict cluster for new data (JSON API)
	GET /health - Health check
	GET /info - Model info

	Usage:
	python api_server.py
	python api_server.py --port 8000
	"""

	from flask import Flask, request, jsonify, render_template
	import pickle
	import numpy as np
	import argparse


	app = Flask(__name__)
	FEATURE_COLUMNS = [{features_list}]

	# Load model at startup
	with open("clustering_model.pkl", "rb") as f:
	MODEL_DATA = pickle.load(f)


	@app.route("/health", methods=["GET"])
	def health():
	return jsonify({{"status": "healthy", "model": "clustering", "algorithm": "{algorithm}"}})


	@app.route("/info", methods=["GET"])
	def info():
	return jsonify({{
	"algorithm": MODEL_DATA.get("algorithm", "{algorithm}"),
	"n_clusters": MODEL_DATA.get("n_clusters", {n_clusters}),
	"feature_columns": MODEL_DATA.get("feature_columns", FEATURE_COLUMNS),
	"silhouette_score": MODEL_DATA.get("silhouette_score", 0),
	}})


	@app.route("/predict", methods=["GET", "POST"])
	def predict():
	# GET request: show web form for interactive cluster prediction
	if request.method == "GET":
	form_fields = ""
	for col in FEATURE_COLUMNS:
	form_fields += '<div style="margin:6px 0"><label style="display:inline-block;width:260px;font-weight:500">' + col + '</label><input name="' + col + '" type="text" value="0" placeholder="Enter value" style="padding:6px 10px;border:1px solid #d1d5db;border-radius:6px;width:200px"></div>\\n'

	algo = MODEL_DATA.get("algorithm", "{algorithm}")
	n_cl = MODEL_DATA.get("n_clusters", {n_clusters})
	sil = MODEL_DATA.get("silhouette_score", 0)

	html = '<!DOCTYPE html><html><head><title>DataVision Clustering API</title>'
	html += '<style>'
	html += "body " + chr(123) + " font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif; max-width: 800px; margin: 40px auto; padding: 0 20px; background: #f9fafb; color: #111827; " + chr(125)
	html += 'h1 ' + chr(123) + ' color: #7C3AED; ' + chr(125) + ' h2 ' + chr(123) + ' color: #1f2937; border-bottom: 2px solid #e5e7eb; padding-bottom: 8px; ' + chr(125)
	html += '.card ' + chr(123) + ' background: white; border-radius: 12px; padding: 24px; margin: 16px 0; box-shadow: 0 1px 3px rgba(0,0,0,0.1); ' + chr(125)
	html += 'button ' + chr(123) + ' background: #7C3AED; color: white; border: none; padding: 12px 32px; border-radius: 8px; font-size: 16px; cursor: pointer; margin-top: 12px; ' + chr(125)
	html += 'button:hover ' + chr(123) + ' background: #6D28D9; ' + chr(125)
	html += '#result ' + chr(123) + ' margin-top: 20px; padding: 16px; background: #f5f3ff; border-radius: 8px; display: none; ' + chr(125)
	html += '</style></head><body>'
	html += '<h1>DataVision Clustering API</h1>'
	html += '<div class="card"><h2>Model Info</h2>'
	html += '<p><b>Algorithm:</b> ' + str(algo).upper() + ' \| <b>Clusters:</b> ' + str(n_cl) + ' \| <b>Silhouette:</b> ' + format(sil, '.4f') + ' \| <b>Features:</b> ' + str(len(FEATURE_COLUMNS)) + '</p></div>'
	html += '<div class="card"><h2>Predict Cluster</h2>'
	html += '<form id="predForm">' + form_fields + '<button type="submit">Predict Cluster</button></form>'
	html += '<div id="result"></div></div>'
	html += '<div class="card"><h2>API Usage</h2>'
	html += '<pre style="background:#1f2937;color:#e5e7eb;padding:16px;border-radius:8px;overflow-x:auto">'
	html += 'POST /predict\\nContent-Type: application/json\\n\\n' + chr(123) + '"feature1": val, "feature2": val' + chr(125)
	html += '\\n\\nGET /health - Health check\\nGET /info - Model metadata</pre></div>'
	html += '<script>'
	html += 'document.getElementById("predForm").onsubmit=function(e)' + chr(123)
	html += 'e.preventDefault();var fd=new FormData(this);var data=' + chr(123) + chr(125) + ';'
	html += 'fd.forEach(function(v,k)' + chr(123) + 'data[k]=isNaN(v)\|\|v===""?v:parseFloat(v)' + chr(125) + ');'
	html += 'fetch("/predict",' + chr(123) + 'method:"POST",headers:' + chr(123) + '"Content-Type":"application/json"' + chr(125) + ',body:JSON.stringify(' + chr(123) + 'features:data' + chr(125) + ')' + chr(125) + ')'
	html += '.then(function(r)' + chr(123) + 'return r.json()' + chr(125) + ')'
	html += '.then(function(j)' + chr(123)
	html += 'var h="<h3>Assigned Cluster: "+j.cluster_name+"</h3>";'
	html += 'h+="<p>Cluster ID: "+j.cluster+"</p>";'
	html += 'if(j.confidence)h+="<p>Confidence: "+(j.confidence*100).toFixed(1)+"%</p>";'
	html += 'document.getElementById("result").innerHTML=h;document.getElementById("result").style.display="block"'
	html += chr(125) + ')' + chr(125) + ';'
	html += '</script></body></html>'
	return html

	# POST request: JSON API prediction
	try:
	data = request.get_json()
	if not data:
	return jsonify({{"error": "No JSON body"}}), 400

	features = data.get("features", data)
	scaler = MODEL_DATA.get("scaler")
	centroids = MODEL_DATA.get("centroids_scaled")
	feature_cols = MODEL_DATA.get("feature_columns", FEATURE_COLUMNS)

	values = []
	for col in feature_cols:
	val = features.get(col, 0)
	try:
	values.append(float(val))
	except (ValueError, TypeError):
	values.append(0.0)

	X = np.array(values).reshape(1, -1)
	if scaler is not None:
	X = scaler.transform(X)

	if centroids is not None:
	centroids_arr = np.array(centroids)
	distances = np.linalg.norm(X - centroids_arr, axis=1)
	cluster = int(np.argmin(distances))
	confidence = float(1.0 / (1.0 + distances[cluster]))
	else:
	cluster = 0
	confidence = 0.0

	return jsonify({{
	"cluster": cluster,
	"cluster_name": f"Cluster_{{cluster}}",
	"confidence": round(confidence, 4),
	}})

	except Exception as e:
	return jsonify({{"error": str(e)}}), 500


	if __name__ == "__main__":
	parser = argparse.ArgumentParser()
	parser.add_argument("--port", type=int, default=5000)
	args = parser.parse_args()
	print("")
	print("Clustering API running on http://localhost:" + str(args.port))
	print(" GET /predict - Web form for predictions")
	print(" POST /predict - Send JSON with feature values")
	print(" GET /health - Health check")
	print(" GET /info - Model metadata")
	app.run(host="0.0.0.0", port=args.port, debug=False)
	'''


	def _generate_clustering_readme(algorithm: str, n_clusters: int, silhouette: float,
	feature_columns: List[str], n_samples: int,
	cluster_profiles: Dict) -> str:
	features_block = "\\n".join(f"\| `{{f}}` \| numeric \|" for f in feature_columns[:15])
	profiles_block = ""
	if cluster_profiles:
	for cname, cdata in cluster_profiles.items():
	size = cdata.get("size", "?")
	pct = cdata.get("percentage", 0)
	profiles_block += f"\| {{cname}} \| {{size}} samples ({{pct:.1f}}%) \|\\n"

	return f"""# Clustering Model - DataVision AutoML

	## Overview
	\| Property \| Value \|
	\|---\|---\|
	\| Task \| Unsupervised Clustering \|
	\| Algorithm \| {algorithm.upper()} \|
	\| Clusters \| {n_clusters} \|
	\| Silhouette Score \| {silhouette:.4f} \|
	\| Samples \| {n_samples:,} \|
	\| Features \| {len(feature_columns)} \|

	## Project Structure
	```
	├── clustering_model.pkl # Trained clustering model
	├── data/
	│ └── clustered_data.csv # Data with cluster assignments
	├── predict_cluster.py # Predict cluster for new data
	├── train_clustering.py # Re-train the clustering model
	├── visualize_clusters.py # Generate clustering charts
	├── api_server.py # Flask REST API
	├── charts/ # Pre-generated training charts (PNG)
	├── config.json # Clustering configuration
	├── requirements.txt # Python dependencies
	├── Dockerfile # Docker deployment
	└── README.md # This file
	```

	## Quick Start

	### 1. Install Dependencies
	```bash
	pip install -r requirements.txt
	```

	### 2. Predict Cluster
	```bash
	# Interactive
	python predict_cluster.py

	# Batch
	python predict_cluster.py --csv new_data.csv
	```

	### 3. Generate Charts
	```bash
	python visualize_clusters.py
	```

	### 4. Re-Train Model
	```bash
	python train_clustering.py --data data/clustered_data.csv --algorithm {algorithm} --n-clusters {n_clusters}
	```

	### 5. Start API Server
	```bash
	python api_server.py --port 5000

	# Test
	curl -X POST http://localhost:5000/predict \\
	-H "Content-Type: application/json" \\
	-d '{{"features": {{}}}}'
	```

	## Features Used
	\| Feature \| Type \|
	\|---\|---\|
	{features_block}

	## Cluster Profiles
	\| Cluster \| Size \|
	\|---\|---\|
	{profiles_block}

	## Charts
	\| Chart \| Description \|
	\|---\|---\|
	\| `cluster_scatter.png` \| PCA 2D scatter plot colored by cluster \|
	\| `cluster_distribution.png` \| Cluster size distribution \|
	\| `silhouette_analysis.png` \| Per-sample silhouette coefficients \|
	\| `elbow_method.png` \| Optimal k detection (inertia + silhouette) \|
	\| `feature_importance.png` \| Feature importance for cluster separation \|

	---
	Generated by DataVision AutoML - Unsupervised Learning
	"""