Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,295 @@
+---
+language: en
+license: apache-2.0
+library_name: pytorch
+tags:
+- deep-reinforcement-learning
+- reinforcement-learning
+- DI-engine
+- TicTacToe-play-with-bot
+benchmark_name: OpenAI/Gym/Atari
+task_name: TicTacToe-play-with-bot
+pipeline_tag: reinforcement-learning
+model-index:
+- name: SampledAlphaZero
+  results:
+  - task:
+      type: reinforcement-learning
+      name: reinforcement-learning
+    dataset:
+      name: TicTacToe-play-with-bot
+      type: TicTacToe-play-with-bot
+    metrics:
+    - type: mean_reward
+      value: 0.3 +/- 0.64
+      name: mean_reward
+---
+
+# Play **TicTacToe-play-with-bot** with **SampledAlphaZero** Policy
+
+## Model Description
+<!-- Provide a longer summary of what this model is. -->
+
+This implementation applies **SampledAlphaZero** to the OpenAI/Gym/Atari **TicTacToe-play-with-bot** environment using [LightZero](https://github.com/opendilab/LightZero) and [DI-engine](https://github.com/opendilab/di-engine).
+
+**LightZero** is an efficient, easy-to-understand open-source toolkit that merges Monte Carlo Tree Search (MCTS) with Deep Reinforcement Learning (RL), simplifying their integration for developers and researchers. More details are in paper [LightZero: A Unified Benchmark for Monte Carlo Tree Search in General Sequential Decision Scenarios](https://huggingface.co/papers/2310.08348).
+
+## Model Usage
+### Install the Dependencies
+<details close>
+<summary>(Click for Details)</summary>
+
+```shell
+# install huggingface_ding
+git clone https://github.com/opendilab/huggingface_ding.git
+pip3 install -e ./huggingface_ding/
+# install environment dependencies if needed
+
+pip3 install DI-engine[common_env,video]
+pip3 install LightZero
+
+```
+</details>
+
+### Git Clone from Huggingface and Run the Model
+
+<details close>
+<summary>(Click for Details)</summary>
+
+```shell
+# running with trained model
+python3 -u run.py
+```
+**run.py**
+```python
+from lzero.agent import SampledAlphaZeroAgent
+from ding.config import Config
+from easydict import EasyDict
+import torch
+
+# Pull model from files which are git cloned from huggingface
+policy_state_dict = torch.load("pytorch_model.bin", map_location=torch.device("cpu"))
+cfg = EasyDict(Config.file_to_dict("policy_config.py").cfg_dict)
+# Instantiate the agent
+agent = SampledAlphaZeroAgent(
+    env_id="TicTacToe-play-with-bot", exp_name="TicTacToe-play-with-bot-SampledAlphaZero", cfg=cfg.exp_config, policy_state_dict=policy_state_dict
+)
+# Continue training
+agent.train(step=5000)
+# Render the new agent performance
+agent.deploy(enable_save_replay=True)
+
+```
+</details>
+
+### Run Model by Using Huggingface_ding
+
+<details close>
+<summary>(Click for Details)</summary>
+
+```shell
+# running with trained model
+python3 -u run.py
+```
+**run.py**
+```python
+from lzero.agent import SampledAlphaZeroAgent
+from huggingface_ding import pull_model_from_hub
+
+# Pull model from Hugggingface hub
+policy_state_dict, cfg = pull_model_from_hub(repo_id="OpenDILabCommunity/TicTacToe-play-with-bot-SampledAlphaZero")
+# Instantiate the agent
+agent = SampledAlphaZeroAgent(
+    env_id="TicTacToe-play-with-bot", exp_name="TicTacToe-play-with-bot-SampledAlphaZero", cfg=cfg.exp_config, policy_state_dict=policy_state_dict
+)
+# Continue training
+agent.train(step=5000)
+# Render the new agent performance
+agent.deploy(enable_save_replay=True)
+
+```
+</details>
+
+## Model Training
+
+### Train the Model and Push to Huggingface_hub
+
+<details close>
+<summary>(Click for Details)</summary>
+
+```shell
+#Training Your Own Agent
+python3 -u train.py
+```
+**train.py**
+```python
+from lzero.agent import SampledAlphaZeroAgent
+from huggingface_ding import push_model_to_hub
+
+# Instantiate the agent
+agent = SampledAlphaZeroAgent(env_id="TicTacToe-play-with-bot", exp_name="TicTacToe-play-with-bot-SampledAlphaZero")
+# Train the agent
+return_ = agent.train(step=int(500000))
+# Push model to huggingface hub
+push_model_to_hub(
+    agent=agent.best,
+    env_name="OpenAI/Gym/Atari",
+    task_name="TicTacToe-play-with-bot",
+    algo_name="SampledAlphaZero",
+    github_repo_url="https://github.com/opendilab/LightZero",
+    github_doc_model_url=None,
+    github_doc_env_url=None,
+    installation_guide='''
+pip3 install DI-engine[common_env,video]
+pip3 install LightZero
+''',
+    usage_file_by_git_clone="./sampled_alphazero/tictactoe_play_with_bot_sampled_alphazero_deploy.py",
+    usage_file_by_huggingface_ding="./sampled_alphazero/tictactoe_play_with_bot_sampled_alphazero_download.py",
+    train_file="./sampled_alphazero/tictactoe_play_with_bot_sampled_alphazero.py",
+    repo_id="OpenDILabCommunity/TicTacToe-play-with-bot-SampledAlphaZero",
+    platform_info="[LightZero](https://github.com/opendilab/LightZero) and [DI-engine](https://github.com/opendilab/di-engine)",
+    model_description="**LightZero** is an efficient, easy-to-understand open-source toolkit that merges Monte Carlo Tree Search (MCTS) with Deep Reinforcement Learning (RL), simplifying their integration for developers and researchers. More details are in paper [LightZero: A Unified Benchmark for Monte Carlo Tree Search in General Sequential Decision Scenarios](https://huggingface.co/papers/2310.08348).",
+    create_repo=True
+)
+
+```
+</details>
+
+**Configuration**
+<details close>
+<summary>(Click for Details)</summary>
+
+
+```python
+exp_config = {
+    'main_config': {
+        'exp_name': 'TicTacToe-play-with-bot-SampledAlphaZero',
+        'seed': 0,
+        'env': {
+            'env_id': 'TicTacToe-play-with-bot',
+            'board_size': 3,
+            'battle_mode': 'play_with_bot_mode',
+            'bot_action_type': 'v0',
+            'channel_last': False,
+            'collector_env_num': 8,
+            'evaluator_env_num': 5,
+            'n_evaluator_episode': 5,
+            'manager': {
+                'shared_memory': False
+            },
+            'agent_vs_human': False,
+            'prob_random_agent': 0,
+            'prob_expert_agent': 0,
+            'scale': True,
+            'alphazero_mcts_ctree': False,
+            'save_replay_gif': False,
+            'replay_path_gif': './replay_gif'
+        },
+        'policy': {
+            'on_policy': False,
+            'cuda': True,
+            'multi_gpu': False,
+            'bp_update_sync': True,
+            'traj_len_inf': False,
+            'model': {
+                'observation_shape': [3, 3, 3],
+                'action_space_size': 9,
+                'num_res_blocks': 1,
+                'num_channels': 16,
+                'fc_value_layers': [8],
+                'fc_policy_layers': [8]
+            },
+            'torch_compile': False,
+            'tensor_float_32': False,
+            'sampled_algo': False,
+            'gumbel_algo': False,
+            'update_per_collect': 50,
+            'model_update_ratio': 0.1,
+            'batch_size': 256,
+            'optim_type': 'Adam',
+            'learning_rate': 0.003,
+            'weight_decay': 0.0001,
+            'momentum': 0.9,
+            'grad_clip_value': 0.5,
+            'value_weight': 1.0,
+            'collector_env_num': 8,
+            'evaluator_env_num': 5,
+            'lr_piecewise_constant_decay': False,
+            'threshold_training_steps_for_final_lr': 500000,
+            'manual_temperature_decay': False,
+            'threshold_training_steps_for_final_temperature': 100000,
+            'fixed_temperature_value': 0.25,
+            'mcts': {
+                'num_simulations': 25
+            },
+            'other': {
+                'replay_buffer': {
+                    'replay_buffer_size': 1000000,
+                    'save_episode': False
+                }
+            },
+            'cfg_type': 'AlphaZeroPolicyDict',
+            'mcts_ctree': False,
+            'simulation_env_name': 'tictactoe',
+            'simulation_env_config_type': 'play_with_bot',
+            'board_size': 3,
+            'entropy_weight': 0.0,
+            'n_episode': 8,
+            'eval_freq': 2000
+        },
+        'wandb_logger': {
+            'gradient_logger': False,
+            'video_logger': False,
+            'plot_logger': False,
+            'action_logger': False,
+            'return_logger': False
+        }
+    },
+    'create_config': {
+        'env': {
+            'type': 'tictactoe',
+            'import_names': ['zoo.board_games.tictactoe.envs.tictactoe_env']
+        },
+        'env_manager': {
+            'type': 'subprocess'
+        },
+        'policy': {
+            'type': 'alphazero',
+            'import_names': ['lzero.policy.alphazero']
+        },
+        'collector': {
+            'type': 'episode_alphazero',
+            'import_names': ['lzero.worker.alphazero_collector']
+        },
+        'evaluator': {
+            'type': 'alphazero',
+            'import_names': ['lzero.worker.alphazero_evaluator']
+        }
+    }
+}
+
+```
+</details>
+
+**Training Procedure** 
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+- **Weights & Biases (wandb):** [monitor link](<TODO>)
+
+## Model Information
+<!-- Provide the basic links for the model. -->
+- **Github Repository:** [repo link](https://github.com/opendilab/LightZero)
+- **Doc**: [Algorithm link](<TODO>)
+- **Configuration:** [config link](https://huggingface.co/OpenDILabCommunity/TicTacToe-play-with-bot-SampledAlphaZero/blob/main/policy_config.py)
+- **Demo:** [video](https://huggingface.co/OpenDILabCommunity/TicTacToe-play-with-bot-SampledAlphaZero/blob/main/replay.mp4)
+<!-- Provide the size information for the model. -->
+- **Parameters total size:** 51.13 KB
+- **Last Update Date:** 2024-02-01
+
+## Environments
+<!-- Address questions around what environment the model is intended to be trained and deployed at, including the necessary information needed to be provided for future users. -->
+- **Benchmark:** OpenAI/Gym/Atari
+- **Task:** TicTacToe-play-with-bot
+- **Gym version:** 0.25.1
+- **DI-engine version:** v0.5.0
+- **PyTorch version:** 2.0.1+cu117
+- **Doc**: [Environments link](<TODO>)