--- name: model-manager description: "Provides model management capabilities for solar power forecasting system. Use this skill to query available models, view model performance metrics (R², RMSE, MAE, etc.), and get detailed model information. Supports listing all trained models and their performance indicators." --- # Model Manager This skill provides comprehensive model management capabilities for the solar power forecasting system. It allows users to query available models, view performance metrics, and retrieve detailed model information. ## Important API Characteristics - **API Endpoint**: The API is hosted at http://47.96.16.84:5001/api/unified/models with endpoints for listing models and retrieving model details. - **Maximum Wait Time**: The system is configured with a timeout of 30 seconds for API calls. - **Response Format**: API responses are returned in JSON format. ## When to Use Use this skill when the user's query involves: - Listing all available models in the system - Viewing model performance metrics (R², RMSE, MAE, MAPE, etc.) - Checking model training status and metadata - Comparing performance between different models - Getting detailed information about a specific model - Filtering models by station, date range, or performance criteria ## Workflow 1. **Analyze the User's Query**: Identify the core request (list models, get model details, compare models) and extract relevant parameters. 2. **Consult Usage Scenarios**: Before the first API call in a task, read the `usage_scenarios.md` file to understand how to best formulate the request for different scenarios. 3. **Execute the Appropriate Script**: Call the relevant script based on the request type: - For listing models: `list_models.py` - For model details: `get_model_details.py` - For model comparison: `compare_models.py` 4. **Present the Result**: Format the answer from the script and present it to the user in a clear, organized manner. ## Bundled Resources - **`scripts/list_models.py`**: Tool for listing all available models and their basic performance metrics. - **Usage**: `python list_models.py [--station_name ] [--sort_by ] [--limit ]` - **Parameters**: - `--station_name` (optional): Filter models by station name - `--sort_by` (optional): Sort by metric (r2, rmse, mae, mape, training_date; default: training_date) - `--limit` (optional): Limit number of results (default: all) - **Output**: JSON array of models with basic metrics - **`scripts/get_model_details.py`**: Tool for retrieving detailed information about a specific model. - **Usage**: `python get_model_details.py --model_id [--model_path ]` - **Parameters**: - `--model_id` (required): Model identifier or filename - `--model_path` (optional): Full path to model file - **Output**: Detailed model information including architecture, training parameters, and performance metrics - **`scripts/compare_models.py`**: Tool for comparing performance metrics between multiple models. - **Usage**: `python compare_models.py --model_ids [--metrics ]` - **Parameters**: - `--model_ids` (required): Comma-separated list of model identifiers - `--metrics` (optional): Comma-separated list of metrics to compare (r2, rmse, mae, mape; default: all) - **Output**: Comparison table with selected metrics - **`references/usage_scenarios.md`**: Provides detailed examples and guidance for different query types. ## Example Usage **User Request:** "当前有哪些模型？" **Action:** 1. First, read `references/usage_scenarios.md` to see how to handle model listing requests. 2. Then, execute the script: ```bash shell( brief="List all available models", action="exec", session="model-management", command="python model-manager/scripts/list_models.py" ) ``` **User Request:** "P2113站点有哪些模型？" **Action:** 1. First, read `references/usage_scenarios.md`. 2. Then, execute the script: ```bash shell( brief="List models for P2113 station", action="exec", session="model-management", command="python model-manager/scripts/list_models.py --station_name P2113 --sort_by r2" ) ``` **User Request:** "比较model_v1和model_v2的性能指标" **Action:** 1. First, read `references/usage_scenarios.md`. 2. Then, execute the script: ```bash shell( brief="Compare model performance", action="exec", session="model-management", command="python model-manager/scripts/compare_models.py --model_ids model_v1,model_v2" ) ``` ## Performance Metrics Explained - **R² (Coefficient of Determination)**: Measures how well the model explains the variance in the data. Range: 0-1, higher is better. - **RMSE (Root Mean Square Error)**: Measures the average prediction error. Lower is better. - **MAE (Mean Absolute Error)**: Average absolute difference between predicted and actual values. Lower is better. - **MAPE (Mean Absolute Percentage Error)**: Average percentage error. Lower is better. - **Training Date**: When the model was trained. - **Data Range**: The time period of data used for training. ## API Endpoint Specifications ### List Models Endpoint **Endpoint**: `GET /api/unified/models/list` **Query Parameters**: - `station_name` (optional): Filter by station name - `sort_by` (optional): Sort field (r2, rmse, mae, mape, training_date) - `limit` (optional): Maximum number of results **Response Format**: ```json { "success": true, "message": "Models retrieved successfully", "data": { "total_count": 15, "models": [ { "model_id": "model_20250326_P2113_v1", "filename": "model_20250326_P2113_v1.pt", "station_name": "P2113", "training_date": "2025-03-26 10:30:00", "data_range": { "start": "2024-12-24 00:00:00", "end": "2025-06-30 23:45:00" }, "performance_metrics": { "r2": 0.8765, "rmse": 125.43, "mae": 98.56, "mape": 12.34 }, "model_size_mb": 5.2, "training_duration_seconds": 3600 } ] } } ``` ### Get Model Details Endpoint **Endpoint**: `GET /api/unified/models/details` **Query Parameters**: - `model_id` (required): Model identifier - `model_path` (optional): Full path to model file **Response Format**: ```json { "success": true, "message": "Model details retrieved successfully", "data": { "model_id": "model_20250326_P2113_v1", "filename": "model_20250326_P2113_v1.pt", "station_name": "P2113", "training_date": "2025-03-26 10:30:00", "data_range": { "start": "2024-12-24 00:00:00", "end": "2025-06-30 23:45:00" }, "model_architecture": { "type": "BiLSTM with Temporal Attention", "n_stations": 5, "n_features": 8, "d_model": 128, "num_layers": 3, "seq_len": 96, "pred_len": 96, "dropout": 0.1 }, "performance_metrics": { "r2": 0.8765, "rmse": 125.43, "mae": 98.56, "mape": 12.34, "daytime_mape": 8.92 }, "training_info": { "total_epochs": 85, "best_epoch": 72, "training_duration_seconds": 3600, "batch_size": 128, "learning_rate": 0.0015, "optimizer": "AdamW" }, "model_size_mb": 5.2, "parameter_count": 1250000 } } ``` ### Compare Models Endpoint **Endpoint**: `POST /api/unified/models/compare` **Request Body**: ```json { "model_ids": ["model_v1", "model_v2", "model_v3"], "metrics": ["r2", "rmse", "mae", "mape"] } ``` **Response Format**: ```json { "success": true, "message": "Models compared successfully", "data": { "comparison": [ { "model_id": "model_v1", "station_name": "P2113", "training_date": "2025-03-26", "r2": 0.8765, "rmse": 125.43, "mae": 98.56, "mape": 12.34 }, { "model_id": "model_v2", "station_name": "P2113", "training_date": "2025-03-25", "r2": 0.8542, "rmse": 135.21, "mae": 105.32, "mape": 13.45 } ] } } ``` ## Notes - Model files are typically stored in the models directory with timestamps and station identifiers in their filenames. - Performance metrics are calculated during model training and stored with the model checkpoint. - The system supports filtering and sorting models by various criteria for easy discovery and comparison. - All API calls are asynchronous and should complete within 30 seconds.