Spaces:
Sleeping
Sleeping
Update chunking.py
Browse files- chunking.py +1 -0
chunking.py
CHANGED
@@ -4,6 +4,7 @@ from langchain.schema import Document
|
|
4 |
def chunk_pythoncode_and_add_metadata(code_files_content, code_files_path):
|
5 |
chunks = []
|
6 |
for code_file_content, code_file_path in zip(code_files_content, code_files_path):
|
|
|
7 |
"""
|
8 |
Custom made python code splitter, algorithm iterates through child nodes of ast-tree(max child depth = 2)
|
9 |
aims to have full body of methods along signature (+ can handle decorators) in a chunk and adds method specific metadata
|
|
|
4 |
def chunk_pythoncode_and_add_metadata(code_files_content, code_files_path):
|
5 |
chunks = []
|
6 |
for code_file_content, code_file_path in zip(code_files_content, code_files_path):
|
7 |
+
print("-----------------------------------------------------", code_file_content)
|
8 |
"""
|
9 |
Custom made python code splitter, algorithm iterates through child nodes of ast-tree(max child depth = 2)
|
10 |
aims to have full body of methods along signature (+ can handle decorators) in a chunk and adds method specific metadata
|