File size: 3,869 Bytes
18ddfe2 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 |
Code for several RL algorithms used in the following papers:
* "Improving Policy Gradient by Exploring Under-appreciated Rewards" by
Ofir Nachum, Mohammad Norouzi, and Dale Schuurmans.
* "Bridging the Gap Between Value and Policy Based Reinforcement Learning" by
Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans.
* "Trust-PCL: An Off-Policy Trust Region Method for Continuous Control" by
Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans.
Available algorithms:
* Actor Critic
* TRPO
* PCL
* Unified PCL
* Trust-PCL
* PCL + Constraint Trust Region (un-published)
* REINFORCE
* UREX
Requirements:
* TensorFlow (see http://www.tensorflow.org for how to install/upgrade)
* OpenAI Gym (see http://gym.openai.com/docs)
* NumPy (see http://www.numpy.org/)
* SciPy (see http://www.scipy.org/)
Quick Start:
Run UREX on a simple environment:
```
python trainer.py --logtostderr --batch_size=400 --env=DuplicatedInput-v0 \
--validation_frequency=25 --tau=0.1 --clip_norm=50 \
--num_samples=10 --objective=urex
```
Run REINFORCE on a simple environment:
```
python trainer.py --logtostderr --batch_size=400 --env=DuplicatedInput-v0 \
--validation_frequency=25 --tau=0.01 --clip_norm=50 \
--num_samples=10 --objective=reinforce
```
Run PCL on a simple environment:
```
python trainer.py --logtostderr --batch_size=400 --env=DuplicatedInput-v0 \
--validation_frequency=25 --tau=0.025 --rollout=10 --critic_weight=1.0 \
--gamma=0.9 --clip_norm=10 --replay_buffer_freq=1 --objective=pcl
```
Run PCL with expert trajectories on a simple environment:
```
python trainer.py --logtostderr --batch_size=400 --env=DuplicatedInput-v0 \
--validation_frequency=25 --tau=0.025 --rollout=10 --critic_weight=1.0 \
--gamma=0.9 --clip_norm=10 --replay_buffer_freq=1 --objective=pcl \
--num_expert_paths=10
```
Run Mujoco task with TRPO:
```
python trainer.py --logtostderr --batch_size=25 --env=HalfCheetah-v1 \
--validation_frequency=5 --rollout=10 --gamma=0.995 \
--max_step=1000 --cutoff_agent=1000 \
--objective=trpo --norecurrent --internal_dim=64 --trust_region_p \
--max_divergence=0.05 --value_opt=best_fit --critic_weight=0.0 \
```
To run Mujoco task using Trust-PCL (off-policy) use the below command.
It should work well across all environments, given that you
search sufficiently among
(1) max_divergence (0.001, 0.0005, 0.002 are good values),
(2) rollout (1, 5, 10 are good values),
(3) tf_seed (need to average over enough random seeds).
```
python trainer.py --logtostderr --batch_size=1 --env=HalfCheetah-v1 \
--validation_frequency=250 --rollout=1 --critic_weight=1.0 --gamma=0.995 \
--clip_norm=40 --learning_rate=0.0001 --replay_buffer_freq=1 \
--replay_buffer_size=5000 --replay_buffer_alpha=0.001 --norecurrent \
--objective=pcl --max_step=10 --cutoff_agent=1000 --tau=0.0 --eviction=fifo \
--max_divergence=0.001 --internal_dim=256 --replay_batch_size=64 \
--nouse_online_batch --batch_by_steps --value_hidden_layers=2 \
--update_eps_lambda --nounify_episodes --target_network_lag=0.99 \
--sample_from=online --clip_adv=1 --prioritize_by=step --num_steps=1000000 \
--noinput_prev_actions --use_target_values --tf_seed=57
```
Run Mujoco task with PCL constraint trust region:
```
python trainer.py --logtostderr --batch_size=25 --env=HalfCheetah-v1 \
--validation_frequency=5 --tau=0.001 --rollout=50 --gamma=0.99 \
--max_step=1000 --cutoff_agent=1000 \
--objective=pcl --norecurrent --internal_dim=64 --trust_region_p \
--max_divergence=0.01 --value_opt=best_fit --critic_weight=0.0 \
--tau_decay=0.1 --tau_start=0.1
```
Maintained by Ofir Nachum (ofirnachum).
|