rag_on_client - v1.11/Readme.md

1. Please download python 3.9 and git before you run install.bat.
2. Open Command Prompt then run below commands to install environment.

   cd RAG_on_client
   install.bat
   

3. Open Command Prompt then run below commands to run demo.

   run_demo.bat
   

Notes

• Change excution device for models

  • Embedding model Please comment out line 349 to define execution device of embedding model

        embedding_device = "NPU"
    

  • Language model Please comment out line 433 to define execution device of language model.

        device = "gpu"
    

  Note: LNL GPU enabling is ongoing, recommend to use CPU for validation.

• Get embedding model for NPU Please following below command to execute it. sh rag_on_client\Scripts\activate python get_npu_embedding_model.py Note: Below log is expected, you can ignore it and you can find the model saved at directory, bge-small-en-v1.5-npu.

  (rag_on_client) C:\Users\Public>python get_npu_embedding_model.py
  Framework not specified. Using pt to export the model.
  Using framework PyTorch: 2.3.1+cpu
  Overriding 1 configuration item(s)
          - use_cache -> False
  Compiling the model to CPU ...
  Traceback (most recent call last):
  File "C:\Program Files\Python39\lib\shutil.py", line 627, in _rmtree_unsafe
      os.unlink(fullname)
  PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\LOCAL_~1\\AppData\\Local\\Temp\\tmpj_ftexji\\openvino_model.bin'
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
  File "C:\Program Files\Python39\lib\tempfile.py", line 805, in onerror
      _os.unlink(path)
  PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\LOCAL_~1\\AppData\\Local\\Temp\\tmpj_ftexji\\openvino_model.bin'
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
  File "C:\Program Files\Python39\lib\weakref.py", line 667, in _exitfunc
      f()
  File "C:\Program Files\Python39\lib\weakref.py", line 591, in __call__
      return info.func(*info.args, **(info.kwargs or {}))
  File "C:\Program Files\Python39\lib\tempfile.py", line 820, in _cleanup
      cls._rmtree(name)
  File "C:\Program Files\Python39\lib\tempfile.py", line 816, in _rmtree
      _shutil.rmtree(name, onerror=onerror)
  File "C:\Program Files\Python39\lib\shutil.py", line 759, in rmtree
      return _rmtree_unsafe(path, onerror)
  File "C:\Program Files\Python39\lib\shutil.py", line 629, in _rmtree_unsafe
      onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Program Files\Python39\lib\tempfile.py", line 808, in onerror
      cls._rmtree(path)
  File "C:\Program Files\Python39\lib\tempfile.py", line 816, in _rmtree
      _shutil.rmtree(name, onerror=onerror)
  File "C:\Program Files\Python39\lib\shutil.py", line 759, in rmtree
      return _rmtree_unsafe(path, onerror)
  File "C:\Program Files\Python39\lib\shutil.py", line 610, in _rmtree_unsafe
      onerror(os.scandir, path, sys.exc_info())
  File "C:\Program Files\Python39\lib\shutil.py", line 607, in _rmtree_unsafe
      with os.scandir(path) as scandir_it:
  NotADirectoryError: [WinError 267] The directory name is invalid: 'C:\\Users\\LOCAL_~1\\AppData\\Local\\Temp\\tmpj_ftexji\\openvino_model.bin'
  

• Replace embedding model Modify model_id = "BAAI/bge-small-en-v1.5" in get_embedding_model.py to another model, then run below command in command prompt.

  rag_on_client\Scripts\activate
  python get_embedding_model.py
  

  After it, you can see the model is saved in the same directory.