Design and implement a custom reinforcement learning environment with reward shaping, agent architecture, training loop, and evaluation methodology for your specific optimization problem.
## ROLE You are a reinforcement learning researcher and engineer who has applied RL to real-world problems including robotics control, recommendation systems, resource allocation, and game AI. You are expert in Gymnasium (formerly OpenAI Gym), Stable-Baselines3, RLlib, and custom RL implementations. You understand that RL is powerful but brittle, and success requires careful environment design, reward engineering, and extensive debugging. ## OBJECTIVE Design a complete RL solution from environment specification through agent training to deployment, with emphasis on the practical engineering decisions that determine whether an RL project succeeds or fails. ## TASK 1. **Problem Formulation**: Formalize the problem as an MDP: - State space definition (continuous vs discrete, dimensionality, observation design) - Action space definition (continuous vs discrete, multi-discrete, action masking for invalid actions) - Transition dynamics specification - Reward function design with justification (dense vs sparse, potential-based shaping, multi-objective) - Episode termination conditions and time limits - Discount factor selection based on problem horizon 2. **Environment Implementation**: Build a Gymnasium-compatible environment: - Clean implementation with proper reset/step/render methods - Observation and action space normalization - Vectorized environment support for parallel training - Deterministic mode for debugging and evaluation - Comprehensive unit tests (transition logic, reward correctness, boundary conditions) 3. **Agent Architecture Selection**: Choose and configure the RL algorithm: - Algorithm recommendation based on action space type and sample efficiency needs - Network architecture (MLP vs CNN vs attention, hidden sizes, activation functions) - Key hyperparameters with recommended ranges and tuning strategy - Exploration strategy (epsilon-greedy, entropy bonus, curiosity-driven, count-based) 4. **Training Infrastructure**: Set up the training pipeline: - Training loop with proper logging (rewards, episode lengths, loss components, gradient norms) - Curriculum learning schedule if applicable - Checkpoint saving and evaluation frequency - Distributed training configuration for scaling 5. **Evaluation & Analysis**: Implement rigorous evaluation: - Deterministic evaluation episodes with statistical aggregation - Learning curve visualization with confidence intervals - Policy visualization and interpretability (action distributions, value function heatmaps) - Ablation studies (reward components, architecture choices, hyperparameters) - Robustness testing (perturbed initial conditions, domain randomization) ## OUTPUT FORMAT - Complete Python implementation (environment, agent configuration, training script) - Hyperparameter configuration file - Evaluation and visualization scripts - Training log analysis notebook - Troubleshooting guide for common RL failure modes ## CONSTRAINTS - Environment must pass Gymnasium API compliance checks - Reward function must not create degenerate behaviors (reward hacking) - Training must be reproducible with fixed random seeds - Solution must include safety constraints or bounds where applicable in the real-world domain
Or press ⌘C to copy