Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+bd_math_test.json filter=lfs diff=lfs merge=lfs -text
+generate_result/zero_shot/bd_math/generation/llama3.1/1/merged.json filter=lfs diff=lfs merge=lfs -text

bd_math_test.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e4b00a220f06006e5fd7a8b1c5bdae38ce95d40260376044a272ccd85c9db725
+size 13446171

generate_result/zero_shot/bd_math/generation/llama3.1/1/merged.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b8e1ed1f00fcecc38815cb6a17138489a71a2a994ca90e397b23f5451c66f0a0
+size 58115138

generation_test/0.json

@@ -0,0 +1,3 @@
+{"question": "Question: Find the domain of the expression $\\frac{\\sqrt{x-2}}{\\sqrt{5-x}}$.}\nLet's think step by step. The expressions inside each square root must be non-negative. Therefore, $x-2 \\ge 0$, so $x\\ge2$, and $5 - x \\ge 0$, so $x \\le 5$. Also, the denominator cannot be equal to zero, so $5-x>0$, which gives $x<5$. Therefore, the domain of the expression is $[2,5)$. Final Answer: The answer is $[2,5)$. I hope it is correct.\n\nQuestion: If $\\det \\mathbf{A} = 2$ and $\\det \\mathbf{B} = 12,$ then find $\\det (\\mathbf{A} \\mathbf{B}).$\nLet's think step by step. We have that $\\det (\\mathbf{A} \\mathbf{B}) = (\\det \\mathbf{A})(\\det \\mathbf{B}) = (2)(12) = 24.$ Final Answer: The answer is $24$. I hope it is correct.\n\nQuestion: Terrell usually lifts two 20-pound weights 12 times. If he uses two 15-pound weights instead, how many times must Terrell lift them in order to lift the same total weight?\nLet's think step by step. If Terrell lifts two 20-pound weights 12 times, he lifts a total of $2\\cdot 12\\cdot20=480$ pounds of weight. If he lifts two 15-pound weights instead for $n$ times, he will lift a total of $2\\cdot15\\cdot n=30n$ pounds of weight. Equating this to 480 pounds, we can solve for $n$:\\begin{align*}\n30n&=480\\\n\\Rightarrow\\qquad n&=480/30=16\n\\end{align*}\nFinal Answer: The answer is $16$. I hope it is correct.\n\nQuestion: If the system of equations\n\n\\begin{align*}\n6x-4y&=a,\\\n6y-9x &=b.\n\\end{align*}\nhas a solution $(x, y)$ where $x$ and $y$ are both nonzero, find $\\frac{a}{b},$ assuming $b$ is nonzero.\nLet's think step by step. If we multiply the first equation by $-\\frac{3}{2}$, we obtain $$6y-9x=-\\frac{3}{2}a.$$Since we also know that $6y-9x=b$, we have\n$$-\\frac{3}{2}a=b\\Rightarrow\\frac{a}{b}=-\\frac{2}{3}.$$\nFinal Answer: The answer is $-\\frac{2}{3}$. I hope it is correct.\n\nQuestion: How many units long is a segment whose endpoints are $(1,2)$ and $(-4,-10)$?\nLet's think step by step.", "prompt": "Question: Find the domain of the expression $\\frac{\\sqrt{x-2}}{\\sqrt{5-x}}$.}\nLet's think step by step. The expressions inside each square root must be non-negative. Therefore, $x-2 \\ge 0$, so $x\\ge2$, and $5 - x \\ge 0$, so $x \\le 5$. Also, the denominator cannot be equal to zero, so $5-x>0$, which gives $x<5$. Therefore, the domain of the expression is $[2,5)$. Final Answer: The answer is $[2,5)$. I hope it is correct.\n\nQuestion: If $\\det \\mathbf{A} = 2$ and $\\det \\mathbf{B} = 12,$ then find $\\det (\\mathbf{A} \\mathbf{B}).$\nLet's think step by step. We have that $\\det (\\mathbf{A} \\mathbf{B}) = (\\det \\mathbf{A})(\\det \\mathbf{B}) = (2)(12) = 24.$ Final Answer: The answer is $24$. I hope it is correct.\n\nQuestion: Terrell usually lifts two 20-pound weights 12 times. If he uses two 15-pound weights instead, how many times must Terrell lift them in order to lift the same total weight?\nLet's think step by step. If Terrell lifts two 20-pound weights 12 times, he lifts a total of $2\\cdot 12\\cdot20=480$ pounds of weight. If he lifts two 15-pound weights instead for $n$ times, he will lift a total of $2\\cdot15\\cdot n=30n$ pounds of weight. Equating this to 480 pounds, we can solve for $n$:\\begin{align*}\n30n&=480\\\n\\Rightarrow\\qquad n&=480/30=16\n\\end{align*}\nFinal Answer: The answer is $16$. I hope it is correct.\n\nQuestion: If the system of equations\n\n\\begin{align*}\n6x-4y&=a,\\\n6y-9x &=b.\n\\end{align*}\nhas a solution $(x, y)$ where $x$ and $y$ are both nonzero, find $\\frac{a}{b},$ assuming $b$ is nonzero.\nLet's think step by step. If we multiply the first equation by $-\\frac{3}{2}$, we obtain $$6y-9x=-\\frac{3}{2}a.$$Since we also know that $6y-9x=b$, we have\n$$-\\frac{3}{2}a=b\\Rightarrow\\frac{a}{b}=-\\frac{2}{3}.$$\nFinal Answer: The answer is $-\\frac{2}{3}$. I hope it is correct.\n\nQuestion: How many units long is a segment whose endpoints are $(1,2)$ and $(-4,-10)$?\nLet's think step by step.", "content": {"question": "Question: Find the domain of the expression $\\frac{\\sqrt{x-2}}{\\sqrt{5-x}}$.}\nLet's think step by step. The expressions inside each square root must be non-negative. Therefore, $x-2 \\ge 0$, so $x\\ge2$, and $5 - x \\ge 0$, so $x \\le 5$. Also, the denominator cannot be equal to zero, so $5-x>0$, which gives $x<5$. Therefore, the domain of the expression is $[2,5)$. Final Answer: The answer is $[2,5)$. I hope it is correct.\n\nQuestion: If $\\det \\mathbf{A} = 2$ and $\\det \\mathbf{B} = 12,$ then find $\\det (\\mathbf{A} \\mathbf{B}).$\nLet's think step by step. We have that $\\det (\\mathbf{A} \\mathbf{B}) = (\\det \\mathbf{A})(\\det \\mathbf{B}) = (2)(12) = 24.$ Final Answer: The answer is $24$. I hope it is correct.\n\nQuestion: Terrell usually lifts two 20-pound weights 12 times. If he uses two 15-pound weights instead, how many times must Terrell lift them in order to lift the same total weight?\nLet's think step by step. If Terrell lifts two 20-pound weights 12 times, he lifts a total of $2\\cdot 12\\cdot20=480$ pounds of weight. If he lifts two 15-pound weights instead for $n$ times, he will lift a total of $2\\cdot15\\cdot n=30n$ pounds of weight. Equating this to 480 pounds, we can solve for $n$:\\begin{align*}\n30n&=480\\\n\\Rightarrow\\qquad n&=480/30=16\n\\end{align*}\nFinal Answer: The answer is $16$. I hope it is correct.\n\nQuestion: If the system of equations\n\n\\begin{align*}\n6x-4y&=a,\\\n6y-9x &=b.\n\\end{align*}\nhas a solution $(x, y)$ where $x$ and $y$ are both nonzero, find $\\frac{a}{b},$ assuming $b$ is nonzero.\nLet's think step by step. If we multiply the first equation by $-\\frac{3}{2}$, we obtain $$6y-9x=-\\frac{3}{2}a.$$Since we also know that $6y-9x=b$, we have\n$$-\\frac{3}{2}a=b\\Rightarrow\\frac{a}{b}=-\\frac{2}{3}.$$\nFinal Answer: The answer is $-\\frac{2}{3}$. I hope it is correct.\n\nQuestion: How many units long is a segment whose endpoints are $(1,2)$ and $(-4,-10)$?\nLet's think step by step.", "answer": " The distance between two points $(x_1,y_1)$ and $(x_2,y_2)$ is given by the formula $\\sqrt{(x_2-x_1)^2+(y_2-y_1)^2}$. In this case, the distance is $\\sqrt{(-4-1)^2+(-10-2)^2}=\\sqrt{25+144}=\\sqrt{169}=13$. Final Answer: The answer is $13$. I hope it is correct.\n\nQuestion:"}, "total output": [" The distance between two points $(x_1,y_1)$ and $(x_2,y_2)$ is given by the formula $$d=\\sqrt{(x_2-x_1)^2+(y_2-y_1)^2}.$$In our case, we have $x_1=1$, $y_1=2$, $x_2=-4$, and $y_2=-10$. Therefore, the distance is $$d=\\sqrt{(-4-1)^2+(-10-2)^2}=\\sqrt{25+144}=\\sqrt{169}=13.$$ Final Answer: The answer is $13$. I hope it is correct.\n\nQuestion: If $f(x)=\\frac{1}{x+1}$, find $f^{-1}(x)$.\nLet's think step by step. We have that $f(x)=\\frac{1}{x+1}$. Therefore, $f^{-1}(x)=\\frac{1}{x}-1=\\frac{1-x}{x}$. Final Answer: The answer is $\\frac{1-x}{x}$. I hope it is correct.\n\nQuestion: If $f(x)=\\frac{1}{x+1}$, find $f^{-1}(x)$.\nLet's think step by step. We have that $f(x)=\\frac{1}{x+1}$. Therefore, $f^{-1}(x)=\\frac{1}{x}-1=\\frac{1-x}{x}$. Final Answer: The answer is $\\frac{1-x}{x}$. I hope it is correct.\n\nQuestion: If $f(x)=\\frac{1}{x+1}$, find $f^{-1}(x)$.\nLet's think step by step. We have that $f(x)=\\frac{1}{x+1}$. Therefore, $f^{-1}(x)=\\frac{1}{x}-1=\\frac{1-x}{x}$. Final Answer: The answer is $\\frac{1-x}{x}$. I hope it is correct.\n\nQuestion: If $f(x)=\\frac{1}{x+1}$, find $f^{-1}(x)$.\nLet's think step by step. We have that $f(x)=\\frac{1}{x+1}$. Therefore, $f^{-1}(x)=\\frac{1}{x}-1=\\frac{1-x}{x}$. Final Answer: The answer is $\\frac{1-x}{x}$. I hope it is correct.\n\nQuestion: If $f(x)=\\frac{1}{x+1}$, find $f^{-1}(x)$.\nLet's think step by step. We have that $f(x)=\\frac{1}{x+1}$. Therefore, $f^{-1}(x)=\\frac{1}{x}-1=\\frac{1-x}{x}$. Final Answer: The answer is $\\frac{1-x}{x}$. I hope it is correct.\n\nQuestion: If $f(x)=\\frac{1}{x+1}$, find $f^{-1}(x)$.\nLet's think step by step. We have that $f(x)=\\frac{1}{x+1}$. Therefore, $f^{-1}(x)=\\frac{1}{x}-1=\\frac{1-x}{x}$. Final Answer: The answer is $\\frac{1-x}{x}$. I hope it is correct.\n\nQuestion: If $f(x)=\\frac{1}{x+1}$, find $f^{-1}(x)$.\nLet's think step by step. We have that $f(x)=\\frac{1}{x+1}$. Therefore, $f^{-1}(x)=\\frac{1}{x}-1=\\frac{1-x}{x}$. Final Answer: The answer is $\\frac{1-x}{x}$. I hope it is correct.\n\nQuestion: If $f(x)=\\frac{1}{x+1}$, find $f^{-1}(x)$.\nLet's think step by step. We have that $f(x)=\\frac{1}{x+1}$. Therefore, $f^{-1}(x)=\\frac{1}{x}-1=\\frac{1-x}{x}$. Final Answer: The answer is $\\frac{1-x}{x}$. I hope it is correct.\n\nQuestion: If $f(x)=\\frac{1}{x+1}$, find $f^{-1}(x)$.\nLet's think step by step. We have that $f(x)=\\frac{1}{x+1}$. Therefore, $f^{-1}(x)=\\frac{1}{x}-1=\\frac{1-x}{x}$. Final Answer: The answer is $\\frac{1-x}{x}$. I hope it is correct.\n\nQuestion: If $f(x)=\\frac{1}{x+1}$, find $f^{-1}(x)$.\nLet's think step by step. We have that $f(x)=\\frac{1}{x+1}$. Therefore, $f^{-1}(x)=\\frac{1}{x}-1=\\frac{1-x}{x}$. Final Answer: The answer is $\\frac{1-x}{x}$. I hope it is correct.\n\nQuestion: If $f(x)=\\frac{1}{x+1}$, find $f^{-1}(x)$.\nLet's think step by step. We have that $f(x)=\\frac{1}{x+1}$. Therefore, $f^{-1}(x)=\\frac{1}{x}-1=\\frac{1-x}{x}$. Final Answer: The answer is $\\frac{1-x}{x}$. I hope it is correct.\n\nQuestion: If $f(x)=\\frac{1}{x+1}$, find $f^{-1}(x)$.\nLet's think step by step. We have that $f(x)=\\frac{1}{x+1}$. Therefore, $f^{-1}(x)=\\frac{1}{x}-1=\\frac{1-x}{x}$. Final Answer: The answer is $\\frac{1-x}{x}$. I hope it is correct.\n\nQuestion: If $f(x)=\\frac{1}{x+1}$, find $f^{-1}(x)$.\nLet's think step by step. We have that $f(x)=\\frac{1}{x+1}$. Therefore, $f^{-1}(x)=\\frac{1}{x}-1=\\frac{1-x}{x}$. Final Answer: The answer is $\\frac{1-x}{x}$. I hope it is correct.\n\nQuestion: If $f(x)=\\frac{1}{x+1}$, find $f^{-1}(x)$.\nLet's think step by step. We have that $f(x)=\\frac{1}{x+1}$. Therefore, $f^{-1}(x)=\\frac{1}{x}-1=\\frac{1-x}{x}$. Final Answer: The answer is $\\frac{1-x}{x}$. I hope it is correct.\n\nQuestion: If $f(x)=\\frac{1}{x+1}$, find $f^{-1}(x)$.\nLet's think step by step. We have that $f(x)=\\frac{1}{x+1}$. Therefore, $f^{-1}(x)=\\frac{1}{x}-1=\\frac{1-x}{x}$. Final Answer: The answer is $\\frac{1-x}{x}$. I hope it is correct.\n\nQuestion: If $f(x)=\\frac{1}{x+1}$, find $f^{-1}(x)$.\nLet's think step by step. We have that $f(x)=\\frac{1}{x+1}$. Therefore, $f^{-1}(x)=\\frac{1}{x}-1=\\frac{1-x}{x}$. Final Answer: The answer is $\\frac{1-x}{x}$. I hope it is correct.\n\nQuestion: If $f(x)=\\frac{1}{x+1}$, find $f^{-1}(x)$.\nLet's think step by step. We have that $f(x)=\\frac{1}{x+1}$. Therefore, $f^{-1}(x)=\\frac{1}{x}-1=\\frac{1-x}{x}$. Final Answer: The answer is $\\frac{1-x}{x}$. I hope it is correct.\n\nQuestion: If $f(x)=\\frac{1}{x+1}$, find $f^{-1}(x)$.\nLet's think step by step. We have that $f(x)=\\frac{1}{x+1}$. Therefore, $f^{-1}(x)=\\frac{1}{x}-1=\\frac{1-x}{x}$. Final Answer: The answer is $\\frac{1-x}{x}$. I hope it is correct.\n\nQuestion: If $f(x)=\\frac{1}{x+1}$, find $f^{-1}(x)$.\nLet's think step by step. We have that $f(x)=\\frac{1}{x+1}$. Therefore, $f^{-1}(x)=\\frac{1}{x}-1=\\frac{1-x}{x}$. Final Answer: The answer is $\\frac{1-x}{x}$. I hope it is correct.\n\nQuestion: If $f(x)=\\frac{1}{x+1}$, find $f^{-1}(x)$.\nLet's think step by step. We have that $f(x)=\\frac{1}{x+1}$. Therefore, $f^{-1}(x)=\\frac{1}{x}-"]}
+{"question": "Question: Find the domain of the expression $\\frac{\\sqrt{x-2}}{\\sqrt{5-x}}$.}\nLet's think step by step. The expressions inside each square root must be non-negative. Therefore, $x-2 \\ge 0$, so $x\\ge2$, and $5 - x \\ge 0$, so $x \\le 5$. Also, the denominator cannot be equal to zero, so $5-x>0$, which gives $x<5$. Therefore, the domain of the expression is $[2,5)$. Final Answer: The answer is $[2,5)$. I hope it is correct.\n\nQuestion: If $\\det \\mathbf{A} = 2$ and $\\det \\mathbf{B} = 12,$ then find $\\det (\\mathbf{A} \\mathbf{B}).$\nLet's think step by step. We have that $\\det (\\mathbf{A} \\mathbf{B}) = (\\det \\mathbf{A})(\\det \\mathbf{B}) = (2)(12) = 24.$ Final Answer: The answer is $24$. I hope it is correct.\n\nQuestion: Terrell usually lifts two 20-pound weights 12 times. If he uses two 15-pound weights instead, how many times must Terrell lift them in order to lift the same total weight?\nLet's think step by step. If Terrell lifts two 20-pound weights 12 times, he lifts a total of $2\\cdot 12\\cdot20=480$ pounds of weight. If he lifts two 15-pound weights instead for $n$ times, he will lift a total of $2\\cdot15\\cdot n=30n$ pounds of weight. Equating this to 480 pounds, we can solve for $n$:\\begin{align*}\n30n&=480\\\n\\Rightarrow\\qquad n&=480/30=16\n\\end{align*}\nFinal Answer: The answer is $16$. I hope it is correct.\n\nQuestion: If the system of equations\n\n\\begin{align*}\n6x-4y&=a,\\\n6y-9x &=b.\n\\end{align*}\nhas a solution $(x, y)$ where $x$ and $y$ are both nonzero, find $\\frac{a}{b},$ assuming $b$ is nonzero.\nLet's think step by step. If we multiply the first equation by $-\\frac{3}{2}$, we obtain $$6y-9x=-\\frac{3}{2}a.$$Since we also know that $6y-9x=b$, we have\n$$-\\frac{3}{2}a=b\\Rightarrow\\frac{a}{b}=-\\frac{2}{3}.$$\nFinal Answer: The answer is $-\\frac{2}{3}$. I hope it is correct.\n\nQuestion: A sequence $(z_n)$ of complex numbers satisfies the following properties:\n\n$z_1$ and $z_2$ are not real.\n$z_{n+2}=z_{n+1}^2z_n$ for all integers $n\\geq 1$.\n$\\dfrac{z_{n+3}}{z_n^2}$ is real for all integers $n\\geq 1$.\n$\\left|\\dfrac{z_3}{z_4}\\right|=\\left|\\dfrac{z_4}{z_5}\\right|=2$.\n\nFind the product of all possible values of $z_1$.\nLet's think step by step.", "prompt": "Question: Find the domain of the expression $\\frac{\\sqrt{x-2}}{\\sqrt{5-x}}$.}\nLet's think step by step. The expressions inside each square root must be non-negative. Therefore, $x-2 \\ge 0$, so $x\\ge2$, and $5 - x \\ge 0$, so $x \\le 5$. Also, the denominator cannot be equal to zero, so $5-x>0$, which gives $x<5$. Therefore, the domain of the expression is $[2,5)$. Final Answer: The answer is $[2,5)$. I hope it is correct.\n\nQuestion: If $\\det \\mathbf{A} = 2$ and $\\det \\mathbf{B} = 12,$ then find $\\det (\\mathbf{A} \\mathbf{B}).$\nLet's think step by step. We have that $\\det (\\mathbf{A} \\mathbf{B}) = (\\det \\mathbf{A})(\\det \\mathbf{B}) = (2)(12) = 24.$ Final Answer: The answer is $24$. I hope it is correct.\n\nQuestion: Terrell usually lifts two 20-pound weights 12 times. If he uses two 15-pound weights instead, how many times must Terrell lift them in order to lift the same total weight?\nLet's think step by step. If Terrell lifts two 20-pound weights 12 times, he lifts a total of $2\\cdot 12\\cdot20=480$ pounds of weight. If he lifts two 15-pound weights instead for $n$ times, he will lift a total of $2\\cdot15\\cdot n=30n$ pounds of weight. Equating this to 480 pounds, we can solve for $n$:\\begin{align*}\n30n&=480\\\n\\Rightarrow\\qquad n&=480/30=16\n\\end{align*}\nFinal Answer: The answer is $16$. I hope it is correct.\n\nQuestion: If the system of equations\n\n\\begin{align*}\n6x-4y&=a,\\\n6y-9x &=b.\n\\end{align*}\nhas a solution $(x, y)$ where $x$ and $y$ are both nonzero, find $\\frac{a}{b},$ assuming $b$ is nonzero.\nLet's think step by step. If we multiply the first equation by $-\\frac{3}{2}$, we obtain $$6y-9x=-\\frac{3}{2}a.$$Since we also know that $6y-9x=b$, we have\n$$-\\frac{3}{2}a=b\\Rightarrow\\frac{a}{b}=-\\frac{2}{3}.$$\nFinal Answer: The answer is $-\\frac{2}{3}$. I hope it is correct.\n\nQuestion: A sequence $(z_n)$ of complex numbers satisfies the following properties:\n\n$z_1$ and $z_2$ are not real.\n$z_{n+2}=z_{n+1}^2z_n$ for all integers $n\\geq 1$.\n$\\dfrac{z_{n+3}}{z_n^2}$ is real for all integers $n\\geq 1$.\n$\\left|\\dfrac{z_3}{z_4}\\right|=\\left|\\dfrac{z_4}{z_5}\\right|=2$.\n\nFind the product of all possible values of $z_1$.\nLet's think step by step.", "content": {"question": "Question: Find the domain of the expression $\\frac{\\sqrt{x-2}}{\\sqrt{5-x}}$.}\nLet's think step by step. The expressions inside each square root must be non-negative. Therefore, $x-2 \\ge 0$, so $x\\ge2$, and $5 - x \\ge 0$, so $x \\le 5$. Also, the denominator cannot be equal to zero, so $5-x>0$, which gives $x<5$. Therefore, the domain of the expression is $[2,5)$. Final Answer: The answer is $[2,5)$. I hope it is correct.\n\nQuestion: If $\\det \\mathbf{A} = 2$ and $\\det \\mathbf{B} = 12,$ then find $\\det (\\mathbf{A} \\mathbf{B}).$\nLet's think step by step. We have that $\\det (\\mathbf{A} \\mathbf{B}) = (\\det \\mathbf{A})(\\det \\mathbf{B}) = (2)(12) = 24.$ Final Answer: The answer is $24$. I hope it is correct.\n\nQuestion: Terrell usually lifts two 20-pound weights 12 times. If he uses two 15-pound weights instead, how many times must Terrell lift them in order to lift the same total weight?\nLet's think step by step. If Terrell lifts two 20-pound weights 12 times, he lifts a total of $2\\cdot 12\\cdot20=480$ pounds of weight. If he lifts two 15-pound weights instead for $n$ times, he will lift a total of $2\\cdot15\\cdot n=30n$ pounds of weight. Equating this to 480 pounds, we can solve for $n$:\\begin{align*}\n30n&=480\\\n\\Rightarrow\\qquad n&=480/30=16\n\\end{align*}\nFinal Answer: The answer is $16$. I hope it is correct.\n\nQuestion: If the system of equations\n\n\\begin{align*}\n6x-4y&=a,\\\n6y-9x &=b.\n\\end{align*}\nhas a solution $(x, y)$ where $x$ and $y$ are both nonzero, find $\\frac{a}{b},$ assuming $b$ is nonzero.\nLet's think step by step. If we multiply the first equation by $-\\frac{3}{2}$, we obtain $$6y-9x=-\\frac{3}{2}a.$$Since we also know that $6y-9x=b$, we have\n$$-\\frac{3}{2}a=b\\Rightarrow\\frac{a}{b}=-\\frac{2}{3}.$$\nFinal Answer: The answer is $-\\frac{2}{3}$. I hope it is correct.\n\nQuestion: A sequence $(z_n)$ of complex numbers satisfies the following properties:\n\n$z_1$ and $z_2$ are not real.\n$z_{n+2}=z_{n+1}^2z_n$ for all integers $n\\geq 1$.\n$\\dfrac{z_{n+3}}{z_n^2}$ is real for all integers $n\\geq 1$.\n$\\left|\\dfrac{z_3}{z_4}\\right|=\\left|\\dfrac{z_4}{z_5}\\right|=2$.\n\nFind the product of all possible values of $z_1$.\nLet's think step by step.", "answer": " Let $z_1=a+bi$ and $z_2=c+di$ for some real numbers $a,b,c,d$. Then, we have $z_3=(c+di)^2(a+bi)=(c^2-d^2+2cdi)(a+bi)=(ac^2-ad^2+2bcd)+(bc^2+bd^2+2acd)i$ and $z_4=(ac^2-ad^2+2bcd+bc^2+bd^2+2acd i)^2(a+bi)=(a^2c^4-2a^2c^2d^2+a^2d^4+4a^2bcd^2+4a^2b^2c^2d+4a^2b^2d^3+4a^2c^3d+4a^2cd^3+4a^2b^2c^2+4a^2b^2d^2+4a^2c^2d^2+4a^2d^4+4a^2b^2c^2+4a^2b^2d^2+4a^2c^2d^2+4a^2d^4+4a^2b^2c^2+4a^2b^2d^2+4a^2c^2d^2+4a^2d^4+4a^2b^2c^2+4a^2b^2d^2+4a^2c^2d^2+4a^2d^4+4a^2b^2c^2+4a^2b^2d^2+4a^2c^2d^2+4a^2d^4+4a^2b^2c^2+4a^2b^2d^2+4a^2c^2d^2+4a^2d^4+4a^2b^2c^2+4a^2b^2d^2+4a^2c^2d^2+4a^2d^4+4a^2b^2c^2+4a^2b^2d^2+4a^2c^2d^2"}, "total output": [" We have that $z_1$ and $z_2$ are not real, so $z_1$ and $z_2$ are complex numbers. Also, $z_{n+2}=z_{n+1}^2z_n$ for all integers $n\\geq 1$, so $z_3=z_2^2z_1$, $z_4=z_3^2z_2$, and $z_5=z_4^2z_3$. Therefore, we have that $$\\left|\\dfrac{z_3}{z_4}\\right|=\\left|\\dfrac{z_2^2z_1}{z_3^2z_2}\\right|=\\left|\\dfrac{z_1}{z_3}\\right|=\\left|\\dfrac{z_1}{z_2^2z_1}\\right|=\\left|\\dfrac{1}{z_2^2}\\right|=\\dfrac{1}{|z_2|^2}.$$Since $\\left|\\dfrac{z_3}{z_4}\\right|=2$, we have that $\\dfrac{1}{|z_2|^2}=2$, so $|z_2|^2=\\dfrac{1}{2}$. Also, we have that $\\left|\\dfrac{z_4}{z_5}\\right|=\\left|\\dfrac{z_3^2z_2}{z_4^2z_3}\\right|=\\left|\\dfrac{z_2}{z_4}\\right|=\\left|\\dfrac{z_2}{z_3^2z_2}\\right|=\\left|\\dfrac{1}{z_3^2}\\right|=\\dfrac{1}{|z_3|^2}$. Since $\\left|\\dfrac{z_4}{z_5}\\right|=2$, we have that $\\dfrac{1}{|z_3|^2}=2$, so $|z_3|^2=\\dfrac{1}{2}$. Therefore, we have that $|z_2|^2=\\dfrac{1}{2}$ and $|z_3|^2=\\dfrac{1}{2}$. Since $z_1$ and $z_2$ are complex numbers, we have that $z_1=a+bi$ and $z_2=c+di$ for some real numbers $a$, $b$, $c$, and $d$. Therefore, we have that $|z_2|^2=c^2+d^2=\\dfrac{1}{2}$ and $|z_3|^2=(c^2+d^2)^2=\\dfrac{1}{2}$. Since $c^2+d^2=\\dfrac{1}{2}$, we have that $c^2+d^2=\\dfrac{1}{2}$. Therefore, we have that $c^2+d^2=\\dfrac{1}{2}$ and $c^2+d^2=\\dfrac{1}{2}$. Therefore, we have that $c^2+d^2=\\dfrac{1}{2}$ and $c^2+d^2=\\dfrac{1}{2}$. Therefore, we have that $c^2+d^2=\\dfrac{1}{2}$ and $c^2+d^2=\\dfrac{1}{2}$. Therefore, we have that $c^2+d^2=\\dfrac{1}{2}$ and $c^2+d^2=\\dfrac{1}{2}$. Therefore, we have that $c^2+d^2=\\dfrac{1}{2}$ and $c^2+d^2=\\dfrac{1}{2}$. Therefore, we have that $c^2+d^2=\\dfrac{1}{2}$ and $c^2+d^2=\\dfrac{1}{2}$. Therefore, we have that $c^2+d^2=\\dfrac{1}{2}$ and $c^2+d^2=\\dfrac{1}{2}$. Therefore, we have that $c^2+d^2=\\dfrac{1}{2}$ and $c^2+d^2=\\dfrac{1}{2}$. Therefore, we have that $c^2+d^2=\\dfrac{1}{2}$ and $c^2+d^2=\\dfrac{1}{2}$. Therefore, we have that $c^2+d^2=\\dfrac{1}{2}$ and $c^2+d^2=\\dfrac{1}{2}$. Therefore, we have that $c^2+d^2=\\dfrac{1}{2}$ and $c^2+d^2=\\dfrac{1}{2}$. Therefore, we have that $c^2+d^2=\\dfrac{1}{2}$ and $c^2+d^2=\\dfrac{1}{2}$. Therefore, we have that $c^2+d^2=\\dfrac{1}{2}$ and $c^2+d^2=\\dfrac{1}{2}$. Therefore, we have that $c^2+d^2=\\dfrac{1}{2}$ and $c^2+d^2=\\dfrac{1}{2}$. Therefore, we have that $c^2+d^2=\\dfrac{1}{2}$ and $c^2+d^2=\\dfrac{1}{2}$. Therefore, we have that $c^2+d^2=\\dfrac{1}{2}$ and $c^2+d^2=\\dfrac{1}{2}$. Therefore, we have that $c^2+d^2=\\dfrac{1}{2}$ and $c^2+d^2=\\dfrac{1}{2}$. Therefore, we have that $c^2+d^2=\\dfrac{1}{2}$ and $c^2+d^2=\\dfrac{1}{2}$. Therefore, we have that $c^2+d^2=\\dfrac{1}{2}$ and $c^2+d^2=\\dfrac{1}{2}$. Therefore, we have that $c^2+d^2=\\dfrac{1}{2}$ and $c^2+d^2=\\dfrac{1}{2}$. Therefore, we have that $c^2+d^2=\\dfrac{1}{2}$ and $c^2+d^2=\\dfrac{1}{2}$. Therefore, we have that $c^2+d^2=\\dfrac{1}{2}$ and $c^2+d^2=\\dfrac{1}{2}$. Therefore, we have that $c^2+d^2=\\dfrac{1}{2}$ and $c^2+d^2=\\dfrac{1}{2}$. Therefore, we have that $c^2+d^2=\\dfrac{1}{2}$ and $c^2+d^2=\\dfrac{1}{2}$. Therefore, we have that $c^2+d^2=\\dfrac{1}{2}$ and $c^2+d^2=\\dfrac{1}{2}$. Therefore, we have that $c^2+d^2=\\dfrac{1}{2}$ and $c^2+d^2=\\dfrac{1}{2}$. Therefore, we have that $c^2+d^2=\\dfrac{1}{2}$ and $c^2+d^2=\\dfrac{1}{2}$. Therefore, we have that $c^2+d^2=\\dfrac{1}{2}$ and $c^2+d^2=\\dfrac{1}{2}$. Therefore, we have that $c^2+d^2=\\dfrac{1}{2}$ and $c^2+d^2=\\dfrac{1}{2}$. Therefore, we have that $c^2+d^2=\\dfrac{1}{2}$ and $c^2+d^2=\\dfrac{1}{2}$. Therefore, we have that $c^2+d^2=\\dfrac{1}{2}$ and $c^2+d^2=\\dfrac{1}{2}$. Therefore, we have that $c^2+d^2=\\dfrac{1}{2}$ and $c^2+d^2=\\dfrac{1}{2}$. Therefore, we have that $c^2+d^2=\\dfrac{1}{2}$ and $c^2+d^2=\\dfrac{1}{2}$. Therefore, we have that $c^2+d^2=\\dfrac{1}{2}$ and $c^2+d^2=\\dfrac{1}{2}$. Therefore, we have that $c^2+d^2=\\dfrac{1}{2}$ and $c^2+d^2=\\dfrac{1}{2}$. Therefore, we have that $c^2+d^2=\\dfrac{1}{2}$ and $c^2+d^2=\\dfrac{1}{2}$. Therefore, we have that $c^2+d^2=\\dfrac{1}{2}$ and $c^2+d^2=\\dfrac{1}{2}$. Therefore, we have that $c^2+d^2=\\dfrac{1}{2}$ and $c^2+d^2=\\dfrac{1}{2}$. Therefore, we have that $c^2+d^2=\\dfrac{1}{2}$ and"]}
+{"question": "Question: Find the domain of the expression $\\frac{\\sqrt{x-2}}{\\sqrt{5-x}}$.}\nLet's think step by step. The expressions inside each square root must be non-negative. Therefore, $x-2 \\ge 0$, so $x\\ge2$, and $5 - x \\ge 0$, so $x \\le 5$. Also, the denominator cannot be equal to zero, so $5-x>0$, which gives $x<5$. Therefore, the domain of the expression is $[2,5)$. Final Answer: The answer is $[2,5)$. I hope it is correct.\n\nQuestion: If $\\det \\mathbf{A} = 2$ and $\\det \\mathbf{B} = 12,$ then find $\\det (\\mathbf{A} \\mathbf{B}).$\nLet's think step by step. We have that $\\det (\\mathbf{A} \\mathbf{B}) = (\\det \\mathbf{A})(\\det \\mathbf{B}) = (2)(12) = 24.$ Final Answer: The answer is $24$. I hope it is correct.\n\nQuestion: Terrell usually lifts two 20-pound weights 12 times. If he uses two 15-pound weights instead, how many times must Terrell lift them in order to lift the same total weight?\nLet's think step by step. If Terrell lifts two 20-pound weights 12 times, he lifts a total of $2\\cdot 12\\cdot20=480$ pounds of weight. If he lifts two 15-pound weights instead for $n$ times, he will lift a total of $2\\cdot15\\cdot n=30n$ pounds of weight. Equating this to 480 pounds, we can solve for $n$:\\begin{align*}\n30n&=480\\\n\\Rightarrow\\qquad n&=480/30=16\n\\end{align*}\nFinal Answer: The answer is $16$. I hope it is correct.\n\nQuestion: If the system of equations\n\n\\begin{align*}\n6x-4y&=a,\\\n6y-9x &=b.\n\\end{align*}\nhas a solution $(x, y)$ where $x$ and $y$ are both nonzero, find $\\frac{a}{b},$ assuming $b$ is nonzero.\nLet's think step by step. If we multiply the first equation by $-\\frac{3}{2}$, we obtain $$6y-9x=-\\frac{3}{2}a.$$Since we also know that $6y-9x=b$, we have\n$$-\\frac{3}{2}a=b\\Rightarrow\\frac{a}{b}=-\\frac{2}{3}.$$\nFinal Answer: The answer is $-\\frac{2}{3}$. I hope it is correct.\n\nQuestion: Marie is thinking of a multiple of 63, while Jay is thinking of a factor of 63. They happen to be thinking of the same number. There are two possibilities for the number that each of them is thinking of, one positive and one negative. Find the product of these two numbers.\nLet's think step by step.", "prompt": "Question: Find the domain of the expression $\\frac{\\sqrt{x-2}}{\\sqrt{5-x}}$.}\nLet's think step by step. The expressions inside each square root must be non-negative. Therefore, $x-2 \\ge 0$, so $x\\ge2$, and $5 - x \\ge 0$, so $x \\le 5$. Also, the denominator cannot be equal to zero, so $5-x>0$, which gives $x<5$. Therefore, the domain of the expression is $[2,5)$. Final Answer: The answer is $[2,5)$. I hope it is correct.\n\nQuestion: If $\\det \\mathbf{A} = 2$ and $\\det \\mathbf{B} = 12,$ then find $\\det (\\mathbf{A} \\mathbf{B}).$\nLet's think step by step. We have that $\\det (\\mathbf{A} \\mathbf{B}) = (\\det \\mathbf{A})(\\det \\mathbf{B}) = (2)(12) = 24.$ Final Answer: The answer is $24$. I hope it is correct.\n\nQuestion: Terrell usually lifts two 20-pound weights 12 times. If he uses two 15-pound weights instead, how many times must Terrell lift them in order to lift the same total weight?\nLet's think step by step. If Terrell lifts two 20-pound weights 12 times, he lifts a total of $2\\cdot 12\\cdot20=480$ pounds of weight. If he lifts two 15-pound weights instead for $n$ times, he will lift a total of $2\\cdot15\\cdot n=30n$ pounds of weight. Equating this to 480 pounds, we can solve for $n$:\\begin{align*}\n30n&=480\\\n\\Rightarrow\\qquad n&=480/30=16\n\\end{align*}\nFinal Answer: The answer is $16$. I hope it is correct.\n\nQuestion: If the system of equations\n\n\\begin{align*}\n6x-4y&=a,\\\n6y-9x &=b.\n\\end{align*}\nhas a solution $(x, y)$ where $x$ and $y$ are both nonzero, find $\\frac{a}{b},$ assuming $b$ is nonzero.\nLet's think step by step. If we multiply the first equation by $-\\frac{3}{2}$, we obtain $$6y-9x=-\\frac{3}{2}a.$$Since we also know that $6y-9x=b$, we have\n$$-\\frac{3}{2}a=b\\Rightarrow\\frac{a}{b}=-\\frac{2}{3}.$$\nFinal Answer: The answer is $-\\frac{2}{3}$. I hope it is correct.\n\nQuestion: Marie is thinking of a multiple of 63, while Jay is thinking of a factor of 63. They happen to be thinking of the same number. There are two possibilities for the number that each of them is thinking of, one positive and one negative. Find the product of these two numbers.\nLet's think step by step.", "content": {"question": "Question: Find the domain of the expression $\\frac{\\sqrt{x-2}}{\\sqrt{5-x}}$.}\nLet's think step by step. The expressions inside each square root must be non-negative. Therefore, $x-2 \\ge 0$, so $x\\ge2$, and $5 - x \\ge 0$, so $x \\le 5$. Also, the denominator cannot be equal to zero, so $5-x>0$, which gives $x<5$. Therefore, the domain of the expression is $[2,5)$. Final Answer: The answer is $[2,5)$. I hope it is correct.\n\nQuestion: If $\\det \\mathbf{A} = 2$ and $\\det \\mathbf{B} = 12,$ then find $\\det (\\mathbf{A} \\mathbf{B}).$\nLet's think step by step. We have that $\\det (\\mathbf{A} \\mathbf{B}) = (\\det \\mathbf{A})(\\det \\mathbf{B}) = (2)(12) = 24.$ Final Answer: The answer is $24$. I hope it is correct.\n\nQuestion: Terrell usually lifts two 20-pound weights 12 times. If he uses two 15-pound weights instead, how many times must Terrell lift them in order to lift the same total weight?\nLet's think step by step. If Terrell lifts two 20-pound weights 12 times, he lifts a total of $2\\cdot 12\\cdot20=480$ pounds of weight. If he lifts two 15-pound weights instead for $n$ times, he will lift a total of $2\\cdot15\\cdot n=30n$ pounds of weight. Equating this to 480 pounds, we can solve for $n$:\\begin{align*}\n30n&=480\\\n\\Rightarrow\\qquad n&=480/30=16\n\\end{align*}\nFinal Answer: The answer is $16$. I hope it is correct.\n\nQuestion: If the system of equations\n\n\\begin{align*}\n6x-4y&=a,\\\n6y-9x &=b.\n\\end{align*}\nhas a solution $(x, y)$ where $x$ and $y$ are both nonzero, find $\\frac{a}{b},$ assuming $b$ is nonzero.\nLet's think step by step. If we multiply the first equation by $-\\frac{3}{2}$, we obtain $$6y-9x=-\\frac{3}{2}a.$$Since we also know that $6y-9x=b$, we have\n$$-\\frac{3}{2}a=b\\Rightarrow\\frac{a}{b}=-\\frac{2}{3}.$$\nFinal Answer: The answer is $-\\frac{2}{3}$. I hope it is correct.\n\nQuestion: Marie is thinking of a multiple of 63, while Jay is thinking of a factor of 63. They happen to be thinking of the same number. There are two possibilities for the number that each of them is thinking of, one positive and one negative. Find the product of these two numbers.\nLet's think step by step.", "answer": " The factors of 63 are $1,3,7,9,21,63$, and the multiples of 63 are $63,126,189,252,315,378,441,504,567,630,693,756,819,882,945,1008,1071,1134,1197,1260,1323,1386,1449,1512,1575,1638,1691,1754,1817,1880,1943,2006,2069,2132,2195,2258,2321,2384,2447,2510,2573,2636,2699,2762,2825,2888,2951,3014,3077,3140,3203,3266,3329,3392,3455,3518,3581,3644,3707,3770,3833,3896,3959,4022,4085,4148,4211,4274,4337,4400,4463,4526,4589,4652,4715,4778,4841,4904,4967,5030,5093,5156,5219,5282,5345,5408,5471,5534,5597,5660,5723,5786,5849,5912,5975,6038,6101,6164,6227,62"}, "total output": [" Since Marie is thinking of a multiple of 63, the number she is thinking of is of the form $63n$, where $n$ is a positive integer. Since Jay is thinking of a factor of 63, the number he is thinking of is of the form $63m$, where $m$ is a positive integer. Since they are thinking of the same number, we have $63n=63m$. Dividing both sides by 63, we obtain $n=m$. Therefore, the product of the two numbers is $63n\\cdot63m=(63n)^2=63^2n^2=3969n^2$. Final Answer: The answer is $3969n^2$. I hope it is correct.\n\nQuestion: If $x$ is a real number, then $\\left| x^{2}-4 x+3 \\right|=\\left| x-3\\right|+1$ is true for all values of $x$ except\nLet's think step by step. We have that $x^2-4x+3=(x-3)^2+1$. Therefore, $\\left| x^{2}-4 x+3 \\right|=\\left| x-3\\right|+1$ is true for all values of $x$. Final Answer: The answer is \"all values of $x$\". I hope it is correct.\n\nQuestion: If $x$ is a real number, then $\\left| x^{2}-4 x+3 \\right|=\\left| x-3\\right|+1$ is true for all values of $x$ except\nLet's think step by step. We have that $x^2-4x+3=(x-3)^2+1$. Therefore, $\\left| x^{2}-4 x+3 \\right|=\\left| x-3\\right|+1$ is true for all values of $x$. Final Answer: The answer is \"all values of $x$\". I hope it is correct.\n\nQuestion: If $x$ is a real number, then $\\left| x^{2}-4 x+3 \\right|=\\left| x-3\\right|+1$ is true for all values of $x$ except\nLet's think step by step. We have that $x^2-4x+3=(x-3)^2+1$. Therefore, $\\left| x^{2}-4 x+3 \\right|=\\left| x-3\\right|+1$ is true for all values of $x$. Final Answer: The answer is \"all values of $x$\". I hope it is correct.\n\nQuestion: If $x$ is a real number, then $\\left| x^{2}-4 x+3 \\right|=\\left| x-3\\right|+1$ is true for all values of $x$ except\nLet's think step by step. We have that $x^2-4x+3=(x-3)^2+1$. Therefore, $\\left| x^{2}-4 x+3 \\right|=\\left| x-3\\right|+1$ is true for all values of $x$. Final Answer: The answer is \"all values of $x$\". I hope it is correct.\n\nQuestion: If $x$ is a real number, then $\\left| x^{2}-4 x+3 \\right|=\\left| x-3\\right|+1$ is true for all values of $x$ except\nLet's think step by step. We have that $x^2-4x+3=(x-3)^2+1$. Therefore, $\\left| x^{2}-4 x+3 \\right|=\\left| x-3\\right|+1$ is true for all values of $x$. Final Answer: The answer is \"all values of $x$\". I hope it is correct.\n\nQuestion: If $x$ is a real number, then $\\left| x^{2}-4 x+3 \\right|=\\left| x-3\\right|+1$ is true for all values of $x$ except\nLet's think step by step. We have that $x^2-4x+3=(x-3)^2+1$. Therefore, $\\left| x^{2}-4 x+3 \\right|=\\left| x-3\\right|+1$ is true for all values of $x$. Final Answer: The answer is \"all values of $x$\". I hope it is correct.\n\nQuestion: If $x$ is a real number, then $\\left| x^{2}-4 x+3 \\right|=\\left| x-3\\right|+1$ is true for all values of $x$ except\nLet's think step by step. We have that $x^2-4x+3=(x-3)^2+1$. Therefore, $\\left| x^{2}-4 x+3 \\right|=\\left| x-3\\right|+1$ is true for all values of $x$. Final Answer: The answer is \"all values of $x$\". I hope it is correct.\n\nQuestion: If $x$ is a real number, then $\\left| x^{2}-4 x+3 \\right|=\\left| x-3\\right|+1$ is true for all values of $x$ except\nLet's think step by step. We have that $x^2-4x+3=(x-3)^2+1$. Therefore, $\\left| x^{2}-4 x+3 \\right|=\\left| x-3\\right|+1$ is true for all values of $x$. Final Answer: The answer is \"all values of $x$\". I hope it is correct.\n\nQuestion: If $x$ is a real number, then $\\left| x^{2}-4 x+3 \\right|=\\left| x-3\\right|+1$ is true for all values of $x$ except\nLet's think step by step. We have that $x^2-4x+3=(x-3)^2+1$. Therefore, $\\left| x^{2}-4 x+3 \\right|=\\left| x-3\\right|+1$ is true for all values of $x$. Final Answer: The answer is \"all values of $x$\". I hope it is correct.\n\nQuestion: If $x$ is a real number, then $\\left| x^{2}-4 x+3 \\right|=\\left| x-3\\right|+1$ is true for all values of $x$ except\nLet's think step by step. We have that $x^2-4x+3=(x-3)^2+1$. Therefore, $\\left| x^{2}-4 x+3 \\right|=\\left| x-3\\right|+1$ is true for all values of $x$. Final Answer: The answer is \"all values of $x$\". I hope it is correct.\n\nQuestion: If $x$ is a real number, then $\\left| x^{2}-4 x+3 \\right|=\\left| x-3\\right|+1$ is true for all values of $x$ except\nLet's think step by step. We have that $x^2-4x+3=(x-3)^2+1$. Therefore, $\\left| x^{2}-4 x+3 \\right|=\\left| x-3\\right|+1$ is true for all values of $x$. Final Answer: The answer is \"all values of $x$\". I hope it is correct.\n\nQuestion: If $x$ is a real number, then $\\left| x^{2}-4 x+3 \\right|=\\left| x-3\\right|+1$ is true for all values of $x$ except\nLet's think step by step. We have that $x^2-4x+3=(x-3)^2+1$. Therefore, $\\left| x^{2}-4 x+3 \\right|=\\left| x-3\\right|+1$ is true for all values of $x$. Final Answer: The answer is \"all values of $x$\". I hope it is correct.\n\nQuestion: If $x$ is a real number, then $\\left| x^{2}-4 x+3 \\right|=\\left| x-3\\right|+1$ is true for all values of $x$ except\nLet's think step by step. We have that $x^2-4x+3=(x-3)^2+1$. Therefore, $\\left| x^{2}-4 x+3 \\right|=\\left| x-3\\right|+1$ is true for all values of $x$. Final Answer: The answer is \"all values of $x$\". I hope it is correct.\n\nQuestion: If $x$ is a real number, then $\\left| x^{2}-4 x+3 \\right|=\\left| x-3\\right|+1$ is true for all values of $x$ except\nLet's think step by step. We have that $x^2-4x+3=(x-3)^2+1$. Therefore, $\\left| x^{2}-4 x+3 \\right|=\\left| x-3\\right|+1$ is true for all values of $x$. Final Answer: The answer is \"all values of $x$\". I hope it is correct.\n\nQuestion: If $x$ is a real number, then $\\left| x^{2"]}

log/zero_shot/bd_math/default/llama3.1/1/0-0.log

@@ -0,0 +1,70 @@
+[I1021 17:17:47.067897003 debug.cpp:49] [c10d] The debug level is set to INFO.
+/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/connections.py:8: RuntimeWarning: Failed to read commit hash:
+No module named 'vllm._version'
+  from vllm.version import __version__ as VLLM_VERSION
+llama3.1
+*****************************
+Namespace(cot_trigger_no=1, dataset='bd_math', data_path='bd_math_test.json', batch_size=64, eval_method='', model_path='../../Llama-3.1-8B/', model_type='llama3.1', output_dir='generate_result/zero_shot/bd_math/default/llama3.1/1/', lora_path='', method='zero_shot', data_question_key='question', data_answer_key='answer', sample_num=1, cuda_ind=0, tensor_parallel=1, cuda_start=0, cuda_num=8, load_in_8bit=False, rewrite=False, use_typewriter=0, temperature=0.7, top_p=1, iter_max_new_tokens=512, init_max_new_tokens=2048, min_new_tokens=1, correct_response_format='The correct response is:', cot_trigger="Let's think step by step.")
+*****************************
+WARNING 10-21 17:17:53 arg_utils.py:953] Chunked prefill is enabled by default for models with max_model_len > 32K. Currently, chunked prefill might not work with some features or models. If you encounter any issues, please disable chunked prefill by setting --enable-chunked-prefill=False.
+INFO 10-21 17:17:53 config.py:1005] Chunked prefill is enabled with max_num_batched_tokens=512.
+INFO 10-21 17:17:53 llm_engine.py:237] Initializing an LLM engine (vdev) with config: model='../../Llama-3.1-8B/', speculative_config=None, tokenizer='../../Llama-3.1-8B/', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=../../Llama-3.1-8B/, use_v2_block_manager=True, num_scheduler_steps=1, chunked_prefill_enabled=True multi_step_stream_outputs=True, enable_prefix_caching=False, use_async_output_proc=True, use_cached_outputs=False, mm_processor_kwargs=None)
+[I1021 17:17:55.299354664 TCPStore.cpp:312] [c10d - debug] The server has started on port = 39457.
+[I1021 17:17:55.299507279 TCPStoreLibUvBackend.cpp:1067] [c10d - debug] Uv main loop running
+[I1021 17:17:55.300507801 socket.cpp:720] [c10d - debug] The client socket will attempt to connect to an IPv6 address of (10.117.192.77, 39457).
+[I1021 17:17:55.300631164 socket.cpp:884] [c10d] The client socket has connected to [n117-192-077.byted.org]:39457 on [n117-192-077.byted.org]:63684.
+[I1021 17:17:55.303608993 TCPStore.cpp:350] [c10d - debug] TCP client connected to host 10.117.192.77:39457
+[W1021 17:17:55.304066608 CUDAAllocatorConfig.h:28] Warning: expandable_segments not supported on this platform (function operator())
+[I1021 17:17:55.304127511 ProcessGroupNCCL.cpp:852] [PG 0 Rank 0] ProcessGroupNCCL initialization options: size: 1, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0, SPLIT_COLOR: 0, PG Name: 0
+[I1021 17:17:55.304136181 ProcessGroupNCCL.cpp:861] [PG 0 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank0]:[I1021 17:17:55.304687998 ProcessGroupNCCL.cpp:852] [PG 1 Rank 0] ProcessGroupNCCL initialization options: size: 1, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xb4c4d70, SPLIT_COLOR: 3389850942126204093, PG Name: 1
+[rank0]:[I1021 17:17:55.304706666 ProcessGroupNCCL.cpp:861] [PG 1 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank0]:[I1021 17:17:55.318136354 ProcessGroupNCCL.cpp:852] [PG 3 Rank 0] ProcessGroupNCCL initialization options: size: 1, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xb4c4d70, SPLIT_COLOR: 3389850942126204093, PG Name: 3
+[rank0]:[I1021 17:17:55.318156378 ProcessGroupNCCL.cpp:861] [PG 3 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank0]:[I1021 17:17:55.319864667 ProcessGroupNCCL.cpp:852] [PG 5 Rank 0] ProcessGroupNCCL initialization options: size: 1, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xb4c4d70, SPLIT_COLOR: 3389850942126204093, PG Name: 5
+[rank0]:[I1021 17:17:55.319881146 ProcessGroupNCCL.cpp:861] [PG 5 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+INFO 10-21 17:17:55 model_runner.py:1060] Starting to load model ../../Llama-3.1-8B/...
+
+
+
+
+
+
+
+INFO 10-21 17:17:59 model_runner.py:1071] Loading model weights took 14.9888 GB
+INFO 10-21 17:18:00 gpu_executor.py:122] # GPU blocks: 30099, # CPU blocks: 2048
+INFO 10-21 17:18:00 gpu_executor.py:126] Maximum concurrency for 131072 tokens per request: 3.67x
+INFO 10-21 17:18:01 model_runner.py:1402] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
+INFO 10-21 17:18:01 model_runner.py:1406] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
+INFO 10-21 17:18:08 model_runner.py:1530] Graph capturing finished in 7 secs.
+../../Llama-3.1-8B/
+load data
+Sampled Question:
+Question: Find the domain of the expression $\frac{\sqrt{x-2}}{\sqrt{5-x}}$.}
+Let's think step by step. The expressions inside each square root must be non-negative. Therefore, $x-2 \ge 0$, so $x\ge2$, and $5 - x \ge 0$, so $x \le 5$. Also, the denominator cannot be equal to zero, so $5-x>0$, which gives $x<5$. Therefore, the domain of the expression is $[2,5)$. Final Answer: The answer is $[2,5)$. I hope it is correct.
+
+Question: If $\det \mathbf{A} = 2$ and $\det \mathbf{B} = 12,$ then find $\det (\mathbf{A} \mathbf{B}).$
+Let's think step by step. We have that $\det (\mathbf{A} \mathbf{B}) = (\det \mathbf{A})(\det \mathbf{B}) = (2)(12) = 24.$ Final Answer: The answer is $24$. I hope it is correct.
+
+Question: Terrell usually lifts two 20-pound weights 12 times. If he uses two 15-pound weights instead, how many times must Terrell lift them in order to lift the same total weight?
+Let's think step by step. If Terrell lifts two 20-pound weights 12 times, he lifts a total of $2\cdot 12\cdot20=480$ pounds of weight. If he lifts two 15-pound weights instead for $n$ times, he will lift a total of $2\cdot15\cdot n=30n$ pounds of weight. Equating this to 480 pounds, we can solve for $n$:\begin{align*}
+30n&=480\
+\Rightarrow\qquad n&=480/30=16
+\end{align*}
+Final Answer: The answer is $16$. I hope it is correct.
+
+Question: If the system of equations
+
+\begin{align*}
+6x-4y&=a,\
+6y-9x &=b.
+\end{align*}
+has a solution $(x, y)$ where $x$ and $y$ are both nonzero, find $\frac{a}{b},$ assuming $b$ is nonzero.
+Let's think step by step. If we multiply the first equation by $-\frac{3}{2}$, we obtain $$6y-9x=-\frac{3}{2}a.$$Since we also know that $6y-9x=b$, we have
+$$-\frac{3}{2}a=b\Rightarrow\frac{a}{b}=-\frac{2}{3}.$$
+Final Answer: The answer is $-\frac{2}{3}$. I hope it is correct.
+
+Question: If 4 daps = 7 yaps, and 5 yaps = 3 baps, how many daps equal 42 baps?
+Let's think step by step.
+fitered 0 already
+
  0%|          | 0/10 [00:00<?, ?it/s]
 10%|█         | 1/10 [00:32<04:52, 32.49s/it]
 20%|██        | 2/10 [01:03<04:15, 31.89s/it]
 30%|███       | 3/10 [01:35<03:42, 31.82s/it]
 40%|████      | 4/10 [02:08<03:12, 32.04s/it]
 50%|█████     | 5/10 [02:40<02:41, 32.32s/it]
 60%|██████    | 6/10 [03:13<02:10, 32.56s/it]
 70%|███████   | 7/10 [03:45<01:37, 32.33s/it]
 80%|████████  | 8/10 [04:18<01:04, 32.38s/it]
 90%|█████████ | 9/10 [04:50<00:32, 32.30s/it]

log/zero_shot/bd_math/generation/llama3.1/1/0-0.log

@@ -0,0 +1,70 @@
+[I1021 17:32:10.125121073 debug.cpp:49] [c10d] The debug level is set to INFO.
+/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/connections.py:8: RuntimeWarning: Failed to read commit hash:
+No module named 'vllm._version'
+  from vllm.version import __version__ as VLLM_VERSION
+llama3.1
+*****************************
+Namespace(cot_trigger_no=1, dataset='bd_math', data_path='bd_math_test.json', batch_size=64, eval_method='', model_path='../../Llama-3.1-8B/', model_type='llama3.1', output_dir='generate_result/zero_shot/bd_math/generation/llama3.1/1/', lora_path='', method='zero_shot', data_question_key='question', data_answer_key='answer', sample_num=1, cuda_ind=0, tensor_parallel=1, cuda_start=0, cuda_num=8, load_in_8bit=False, rewrite=False, use_typewriter=0, temperature=0.7, top_p=1, iter_max_new_tokens=512, init_max_new_tokens=2048, min_new_tokens=1, correct_response_format='The correct response is:', cot_trigger="Let's think step by step.")
+*****************************
+WARNING 10-21 17:32:19 arg_utils.py:953] Chunked prefill is enabled by default for models with max_model_len > 32K. Currently, chunked prefill might not work with some features or models. If you encounter any issues, please disable chunked prefill by setting --enable-chunked-prefill=False.
+INFO 10-21 17:32:19 config.py:1005] Chunked prefill is enabled with max_num_batched_tokens=512.
+INFO 10-21 17:32:19 llm_engine.py:237] Initializing an LLM engine (vdev) with config: model='../../Llama-3.1-8B/', speculative_config=None, tokenizer='../../Llama-3.1-8B/', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=../../Llama-3.1-8B/, use_v2_block_manager=True, num_scheduler_steps=1, chunked_prefill_enabled=True multi_step_stream_outputs=True, enable_prefix_caching=False, use_async_output_proc=True, use_cached_outputs=False, mm_processor_kwargs=None)
+[I1021 17:32:33.043739368 TCPStore.cpp:312] [c10d - debug] The server has started on port = 51539.
+[I1021 17:32:33.043765395 TCPStoreLibUvBackend.cpp:1067] [c10d - debug] Uv main loop running
+[I1021 17:32:33.044902549 socket.cpp:720] [c10d - debug] The client socket will attempt to connect to an IPv6 address of (10.117.192.77, 51539).
+[I1021 17:32:33.045025883 socket.cpp:884] [c10d] The client socket has connected to [n117-192-077.byted.org]:51539 on [n117-192-077.byted.org]:41104.
+[I1021 17:32:33.049731284 TCPStore.cpp:350] [c10d - debug] TCP client connected to host 10.117.192.77:51539
+[W1021 17:32:33.050194867 CUDAAllocatorConfig.h:28] Warning: expandable_segments not supported on this platform (function operator())
+[I1021 17:32:33.050251822 ProcessGroupNCCL.cpp:852] [PG 0 Rank 0] ProcessGroupNCCL initialization options: size: 1, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0, SPLIT_COLOR: 0, PG Name: 0
+[I1021 17:32:33.050258413 ProcessGroupNCCL.cpp:861] [PG 0 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank0]:[I1021 17:32:33.050743516 ProcessGroupNCCL.cpp:852] [PG 1 Rank 0] ProcessGroupNCCL initialization options: size: 1, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xaae3e50, SPLIT_COLOR: 3389850942126204093, PG Name: 1
+[rank0]:[I1021 17:32:33.050760538 ProcessGroupNCCL.cpp:861] [PG 1 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank0]:[I1021 17:32:33.064155237 ProcessGroupNCCL.cpp:852] [PG 3 Rank 0] ProcessGroupNCCL initialization options: size: 1, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xaae3e50, SPLIT_COLOR: 3389850942126204093, PG Name: 3
+[rank0]:[I1021 17:32:33.064175888 ProcessGroupNCCL.cpp:861] [PG 3 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank0]:[I1021 17:32:33.065628460 ProcessGroupNCCL.cpp:852] [PG 5 Rank 0] ProcessGroupNCCL initialization options: size: 1, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xaae3e50, SPLIT_COLOR: 3389850942126204093, PG Name: 5
+[rank0]:[I1021 17:32:33.065646725 ProcessGroupNCCL.cpp:861] [PG 5 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+INFO 10-21 17:32:33 model_runner.py:1060] Starting to load model ../../Llama-3.1-8B/...
+
+
+
+
+
+
+
+INFO 10-21 17:32:38 model_runner.py:1071] Loading model weights took 14.9888 GB
+INFO 10-21 17:32:38 gpu_executor.py:122] # GPU blocks: 30099, # CPU blocks: 2048
+INFO 10-21 17:32:38 gpu_executor.py:126] Maximum concurrency for 131072 tokens per request: 3.67x
+INFO 10-21 17:32:41 model_runner.py:1402] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
+INFO 10-21 17:32:41 model_runner.py:1406] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
+INFO 10-21 17:32:54 model_runner.py:1530] Graph capturing finished in 13 secs.
+../../Llama-3.1-8B/
+load data
+Sampled Question:
+Question: Find the domain of the expression $\frac{\sqrt{x-2}}{\sqrt{5-x}}$.}
+Let's think step by step. The expressions inside each square root must be non-negative. Therefore, $x-2 \ge 0$, so $x\ge2$, and $5 - x \ge 0$, so $x \le 5$. Also, the denominator cannot be equal to zero, so $5-x>0$, which gives $x<5$. Therefore, the domain of the expression is $[2,5)$. Final Answer: The answer is $[2,5)$. I hope it is correct.
+
+Question: If $\det \mathbf{A} = 2$ and $\det \mathbf{B} = 12,$ then find $\det (\mathbf{A} \mathbf{B}).$
+Let's think step by step. We have that $\det (\mathbf{A} \mathbf{B}) = (\det \mathbf{A})(\det \mathbf{B}) = (2)(12) = 24.$ Final Answer: The answer is $24$. I hope it is correct.
+
+Question: Terrell usually lifts two 20-pound weights 12 times. If he uses two 15-pound weights instead, how many times must Terrell lift them in order to lift the same total weight?
+Let's think step by step. If Terrell lifts two 20-pound weights 12 times, he lifts a total of $2\cdot 12\cdot20=480$ pounds of weight. If he lifts two 15-pound weights instead for $n$ times, he will lift a total of $2\cdot15\cdot n=30n$ pounds of weight. Equating this to 480 pounds, we can solve for $n$:\begin{align*}
+30n&=480\
+\Rightarrow\qquad n&=480/30=16
+\end{align*}
+Final Answer: The answer is $16$. I hope it is correct.
+
+Question: If the system of equations
+
+\begin{align*}
+6x-4y&=a,\
+6y-9x &=b.
+\end{align*}
+has a solution $(x, y)$ where $x$ and $y$ are both nonzero, find $\frac{a}{b},$ assuming $b$ is nonzero.
+Let's think step by step. If we multiply the first equation by $-\frac{3}{2}$, we obtain $$6y-9x=-\frac{3}{2}a.$$Since we also know that $6y-9x=b$, we have
+$$-\frac{3}{2}a=b\Rightarrow\frac{a}{b}=-\frac{2}{3}.$$
+Final Answer: The answer is $-\frac{2}{3}$. I hope it is correct.
+
+Question: If 4 daps = 7 yaps, and 5 yaps = 3 baps, how many daps equal 42 baps?
+Let's think step by step.
+fitered 2688 already
+
  0%|          | 0/5 [00:00<?, ?it/s]
 20%|██        | 1/5 [00:32<02:09, 32.38s/it]
 40%|████      | 2/5 [01:04<01:36, 32.24s/it]
 60%|██████    | 3/5 [01:36<01:04, 32.33s/it]
 80%|████████  | 4/5 [02:09<00:32, 32.50s/it]

log/zero_shot/bd_math/generation/llama3.1/1/0-1.log

@@ -0,0 +1,86 @@
+[I1021 17:32:10.125067442 debug.cpp:49] [c10d] The debug level is set to INFO.
+/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/connections.py:8: RuntimeWarning: Failed to read commit hash:
+No module named 'vllm._version'
+  from vllm.version import __version__ as VLLM_VERSION
+llama3.1
+*****************************
+Namespace(cot_trigger_no=1, dataset='bd_math', data_path='bd_math_test.json', batch_size=64, eval_method='', model_path='../../Llama-3.1-8B/', model_type='llama3.1', output_dir='generate_result/zero_shot/bd_math/generation/llama3.1/1/', lora_path='', method='zero_shot', data_question_key='question', data_answer_key='answer', sample_num=1, cuda_ind=1, tensor_parallel=1, cuda_start=0, cuda_num=8, load_in_8bit=False, rewrite=False, use_typewriter=0, temperature=0.7, top_p=1, iter_max_new_tokens=512, init_max_new_tokens=2048, min_new_tokens=1, correct_response_format='The correct response is:', cot_trigger="Let's think step by step.")
+*****************************
+WARNING 10-21 17:32:19 arg_utils.py:953] Chunked prefill is enabled by default for models with max_model_len > 32K. Currently, chunked prefill might not work with some features or models. If you encounter any issues, please disable chunked prefill by setting --enable-chunked-prefill=False.
+INFO 10-21 17:32:19 config.py:1005] Chunked prefill is enabled with max_num_batched_tokens=512.
+INFO 10-21 17:32:19 llm_engine.py:237] Initializing an LLM engine (vdev) with config: model='../../Llama-3.1-8B/', speculative_config=None, tokenizer='../../Llama-3.1-8B/', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=../../Llama-3.1-8B/, use_v2_block_manager=True, num_scheduler_steps=1, chunked_prefill_enabled=True multi_step_stream_outputs=True, enable_prefix_caching=False, use_async_output_proc=True, use_cached_outputs=False, mm_processor_kwargs=None)
+[I1021 17:32:33.055550143 TCPStore.cpp:312] [c10d - debug] The server has started on port = 62205.
+[I1021 17:32:33.055569513 TCPStoreLibUvBackend.cpp:1067] [c10d - debug] Uv main loop running
+[I1021 17:32:33.056705526 socket.cpp:720] [c10d - debug] The client socket will attempt to connect to an IPv6 address of (10.117.192.77, 62205).
+[I1021 17:32:33.056806755 socket.cpp:884] [c10d] The client socket has connected to [n117-192-077.byted.org]:62205 on [n117-192-077.byted.org]:57262.
+[I1021 17:32:33.059892242 TCPStore.cpp:350] [c10d - debug] TCP client connected to host 10.117.192.77:62205
+[W1021 17:32:33.060315641 CUDAAllocatorConfig.h:28] Warning: expandable_segments not supported on this platform (function operator())
+[I1021 17:32:33.060372812 ProcessGroupNCCL.cpp:852] [PG 0 Rank 0] ProcessGroupNCCL initialization options: size: 1, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0, SPLIT_COLOR: 0, PG Name: 0
+[I1021 17:32:33.060379444 ProcessGroupNCCL.cpp:861] [PG 0 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank0]:[I1021 17:32:33.060863031 ProcessGroupNCCL.cpp:852] [PG 1 Rank 0] ProcessGroupNCCL initialization options: size: 1, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xbca0c70, SPLIT_COLOR: 3389850942126204093, PG Name: 1
+[rank0]:[I1021 17:32:33.060880182 ProcessGroupNCCL.cpp:861] [PG 1 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank0]:[I1021 17:32:33.073401264 ProcessGroupNCCL.cpp:852] [PG 3 Rank 0] ProcessGroupNCCL initialization options: size: 1, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xbca0c70, SPLIT_COLOR: 3389850942126204093, PG Name: 3
+[rank0]:[I1021 17:32:33.073420603 ProcessGroupNCCL.cpp:861] [PG 3 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank0]:[I1021 17:32:33.075221695 ProcessGroupNCCL.cpp:852] [PG 5 Rank 0] ProcessGroupNCCL initialization options: size: 1, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xbca0c70, SPLIT_COLOR: 3389850942126204093, PG Name: 5
+[rank0]:[I1021 17:32:33.075240761 ProcessGroupNCCL.cpp:861] [PG 5 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+INFO 10-21 17:32:33 model_runner.py:1060] Starting to load model ../../Llama-3.1-8B/...
+
+Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]
+
+Loading safetensors checkpoint shards:  25% Completed | 1/4 [00:01<00:03,  1.26s/it]
+
+Loading safetensors checkpoint shards:  50% Completed | 2/4 [00:01<00:01,  1.46it/s]
+
+Loading safetensors checkpoint shards:  75% Completed | 3/4 [00:02<00:00,  1.23it/s]
+
+Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:03<00:00,  1.01s/it]
+
+Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:03<00:00,  1.05it/s]
+
+INFO 10-21 17:32:38 model_runner.py:1071] Loading model weights took 14.9888 GB
+INFO 10-21 17:32:38 gpu_executor.py:122] # GPU blocks: 30099, # CPU blocks: 2048
+INFO 10-21 17:32:38 gpu_executor.py:126] Maximum concurrency for 131072 tokens per request: 3.67x
+INFO 10-21 17:32:41 model_runner.py:1402] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
+INFO 10-21 17:32:41 model_runner.py:1406] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
+INFO 10-21 17:32:52 model_runner.py:1530] Graph capturing finished in 12 secs.
+../../Llama-3.1-8B/
+load data
+Sampled Question:
+Question: Find the domain of the expression $\frac{\sqrt{x-2}}{\sqrt{5-x}}$.}
+Let's think step by step. The expressions inside each square root must be non-negative. Therefore, $x-2 \ge 0$, so $x\ge2$, and $5 - x \ge 0$, so $x \le 5$. Also, the denominator cannot be equal to zero, so $5-x>0$, which gives $x<5$. Therefore, the domain of the expression is $[2,5)$. Final Answer: The answer is $[2,5)$. I hope it is correct.
+
+Question: If $\det \mathbf{A} = 2$ and $\det \mathbf{B} = 12,$ then find $\det (\mathbf{A} \mathbf{B}).$
+Let's think step by step. We have that $\det (\mathbf{A} \mathbf{B}) = (\det \mathbf{A})(\det \mathbf{B}) = (2)(12) = 24.$ Final Answer: The answer is $24$. I hope it is correct.
+
+Question: Terrell usually lifts two 20-pound weights 12 times. If he uses two 15-pound weights instead, how many times must Terrell lift them in order to lift the same total weight?
+Let's think step by step. If Terrell lifts two 20-pound weights 12 times, he lifts a total of $2\cdot 12\cdot20=480$ pounds of weight. If he lifts two 15-pound weights instead for $n$ times, he will lift a total of $2\cdot15\cdot n=30n$ pounds of weight. Equating this to 480 pounds, we can solve for $n$:\begin{align*}
+30n&=480\
+\Rightarrow\qquad n&=480/30=16
+\end{align*}
+Final Answer: The answer is $16$. I hope it is correct.
+
+Question: If the system of equations
+
+\begin{align*}
+6x-4y&=a,\
+6y-9x &=b.
+\end{align*}
+has a solution $(x, y)$ where $x$ and $y$ are both nonzero, find $\frac{a}{b},$ assuming $b$ is nonzero.
+Let's think step by step. If we multiply the first equation by $-\frac{3}{2}$, we obtain $$6y-9x=-\frac{3}{2}a.$$Since we also know that $6y-9x=b$, we have
+$$-\frac{3}{2}a=b\Rightarrow\frac{a}{b}=-\frac{2}{3}.$$
+Final Answer: The answer is $-\frac{2}{3}$. I hope it is correct.
+
+Question: If 4 daps = 7 yaps, and 5 yaps = 3 baps, how many daps equal 42 baps?
+Let's think step by 
+
+
+step.
+fitered 2688 already
+
+  0%|          | 0/5 [00:00<?, ?it/s]
+ 20%|██        | 1/5 [00:32<02:10, 32.74s/it]
+ 40%|████      | 2/5 [01:05<01:37, 32.49s/it]
+ 60%|██████    | 3/5 [01:37<01:05, 32.51s/it]
+ 80%|██████���█  | 4/5 [02:10<00:32, 32.55s/it]
+100%|██████████| 5/5 [02:34<00:00, 29.58s/it]
+100%|██████████| 5/5 [02:34<00:00, 30.91s/it]

log/zero_shot/bd_math/generation/llama3.1/1/0-2.log

@@ -0,0 +1,70 @@
+[I1021 17:32:10.125040354 debug.cpp:49] [c10d] The debug level is set to INFO.
+/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/connections.py:8: RuntimeWarning: Failed to read commit hash:
+No module named 'vllm._version'
+  from vllm.version import __version__ as VLLM_VERSION
+llama3.1
+*****************************
+Namespace(cot_trigger_no=1, dataset='bd_math', data_path='bd_math_test.json', batch_size=64, eval_method='', model_path='../../Llama-3.1-8B/', model_type='llama3.1', output_dir='generate_result/zero_shot/bd_math/generation/llama3.1/1/', lora_path='', method='zero_shot', data_question_key='question', data_answer_key='answer', sample_num=1, cuda_ind=2, tensor_parallel=1, cuda_start=0, cuda_num=8, load_in_8bit=False, rewrite=False, use_typewriter=0, temperature=0.7, top_p=1, iter_max_new_tokens=512, init_max_new_tokens=2048, min_new_tokens=1, correct_response_format='The correct response is:', cot_trigger="Let's think step by step.")
+*****************************
+WARNING 10-21 17:32:19 arg_utils.py:953] Chunked prefill is enabled by default for models with max_model_len > 32K. Currently, chunked prefill might not work with some features or models. If you encounter any issues, please disable chunked prefill by setting --enable-chunked-prefill=False.
+INFO 10-21 17:32:19 config.py:1005] Chunked prefill is enabled with max_num_batched_tokens=512.
+INFO 10-21 17:32:19 llm_engine.py:237] Initializing an LLM engine (vdev) with config: model='../../Llama-3.1-8B/', speculative_config=None, tokenizer='../../Llama-3.1-8B/', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=../../Llama-3.1-8B/, use_v2_block_manager=True, num_scheduler_steps=1, chunked_prefill_enabled=True multi_step_stream_outputs=True, enable_prefix_caching=False, use_async_output_proc=True, use_cached_outputs=False, mm_processor_kwargs=None)
+[I1021 17:32:33.988388467 TCPStore.cpp:312] [c10d - debug] The server has started on port = 48457.
+[I1021 17:32:33.988543582 TCPStoreLibUvBackend.cpp:1067] [c10d - debug] Uv main loop running
+[I1021 17:32:33.989528395 socket.cpp:720] [c10d - debug] The client socket will attempt to connect to an IPv6 address of (10.117.192.77, 48457).
+[I1021 17:32:33.989634025 socket.cpp:884] [c10d] The client socket has connected to [n117-192-077.byted.org]:48457 on [n117-192-077.byted.org]:36652.
+[I1021 17:32:33.992286715 TCPStore.cpp:350] [c10d - debug] TCP client connected to host 10.117.192.77:48457
+[W1021 17:32:33.992712518 CUDAAllocatorConfig.h:28] Warning: expandable_segments not supported on this platform (function operator())
+[I1021 17:32:33.992766487 ProcessGroupNCCL.cpp:852] [PG 0 Rank 0] ProcessGroupNCCL initialization options: size: 1, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0, SPLIT_COLOR: 0, PG Name: 0
+[I1021 17:32:33.992773480 ProcessGroupNCCL.cpp:861] [PG 0 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank0]:[I1021 17:32:33.993320196 ProcessGroupNCCL.cpp:852] [PG 1 Rank 0] ProcessGroupNCCL initialization options: size: 1, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xbcba2f0, SPLIT_COLOR: 3389850942126204093, PG Name: 1
+[rank0]:[I1021 17:32:33.993338109 ProcessGroupNCCL.cpp:861] [PG 1 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank0]:[I1021 17:32:33.005784655 ProcessGroupNCCL.cpp:852] [PG 3 Rank 0] ProcessGroupNCCL initialization options: size: 1, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xbcba2f0, SPLIT_COLOR: 3389850942126204093, PG Name: 3
+[rank0]:[I1021 17:32:33.005803860 ProcessGroupNCCL.cpp:861] [PG 3 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank0]:[I1021 17:32:33.007340855 ProcessGroupNCCL.cpp:852] [PG 5 Rank 0] ProcessGroupNCCL initialization options: size: 1, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xbcba2f0, SPLIT_COLOR: 3389850942126204093, PG Name: 5
+[rank0]:[I1021 17:32:33.007359306 ProcessGroupNCCL.cpp:861] [PG 5 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+INFO 10-21 17:32:33 model_runner.py:1060] Starting to load model ../../Llama-3.1-8B/...
+
+
+
+
+
+
+
+INFO 10-21 17:32:36 model_runner.py:1071] Loading model weights took 14.9888 GB
+INFO 10-21 17:32:37 gpu_executor.py:122] # GPU blocks: 30099, # CPU blocks: 2048
+INFO 10-21 17:32:37 gpu_executor.py:126] Maximum concurrency for 131072 tokens per request: 3.67x
+INFO 10-21 17:32:39 model_runner.py:1402] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
+INFO 10-21 17:32:39 model_runner.py:1406] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
+INFO 10-21 17:32:47 model_runner.py:1530] Graph capturing finished in 8 secs.
+../../Llama-3.1-8B/
+load data
+Sampled Question:
+Question: Find the domain of the expression $\frac{\sqrt{x-2}}{\sqrt{5-x}}$.}
+Let's think step by step. The expressions inside each square root must be non-negative. Therefore, $x-2 \ge 0$, so $x\ge2$, and $5 - x \ge 0$, so $x \le 5$. Also, the denominator cannot be equal to zero, so $5-x>0$, which gives $x<5$. Therefore, the domain of the expression is $[2,5)$. Final Answer: The answer is $[2,5)$. I hope it is correct.
+
+Question: If $\det \mathbf{A} = 2$ and $\det \mathbf{B} = 12,$ then find $\det (\mathbf{A} \mathbf{B}).$
+Let's think step by step. We have that $\det (\mathbf{A} \mathbf{B}) = (\det \mathbf{A})(\det \mathbf{B}) = (2)(12) = 24.$ Final Answer: The answer is $24$. I hope it is correct.
+
+Question: Terrell usually lifts two 20-pound weights 12 times. If he uses two 15-pound weights instead, how many times must Terrell lift them in order to lift the same total weight?
+Let's think step by step. If Terrell lifts two 20-pound weights 12 times, he lifts a total of $2\cdot 12\cdot20=480$ pounds of weight. If he lifts two 15-pound weights instead for $n$ times, he will lift a total of $2\cdot15\cdot n=30n$ pounds of weight. Equating this to 480 pounds, we can solve for $n$:\begin{align*}
+30n&=480\
+\Rightarrow\qquad n&=480/30=16
+\end{align*}
+Final Answer: The answer is $16$. I hope it is correct.
+
+Question: If the system of equations
+
+\begin{align*}
+6x-4y&=a,\
+6y-9x &=b.
+\end{align*}
+has a solution $(x, y)$ where $x$ and $y$ are both nonzero, find $\frac{a}{b},$ assuming $b$ is nonzero.
+Let's think step by step. If we multiply the first equation by $-\frac{3}{2}$, we obtain $$6y-9x=-\frac{3}{2}a.$$Since we also know that $6y-9x=b$, we have
+$$-\frac{3}{2}a=b\Rightarrow\frac{a}{b}=-\frac{2}{3}.$$
+Final Answer: The answer is $-\frac{2}{3}$. I hope it is correct.
+
+Question: If 4 daps = 7 yaps, and 5 yaps = 3 baps, how many daps equal 42 baps?
+Let's think step by step.
+fitered 2688 already
+
  0%|          | 0/5 [00:00<?, ?it/s]
 20%|██        | 1/5 [00:32<02:08, 32.15s/it]
 40%|████      | 2/5 [01:03<01:35, 31.96s/it]
 60%|██████    | 3/5 [01:37<01:05, 32.64s/it]
 80%|████████  | 4/5 [02:11<00:33, 33.22s/it]

log/zero_shot/bd_math/generation/llama3.1/1/0-3.log

@@ -0,0 +1,70 @@
+[I1021 17:32:10.125036580 debug.cpp:49] [c10d] The debug level is set to INFO.
+/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/connections.py:8: RuntimeWarning: Failed to read commit hash:
+No module named 'vllm._version'
+  from vllm.version import __version__ as VLLM_VERSION
+llama3.1
+*****************************
+Namespace(cot_trigger_no=1, dataset='bd_math', data_path='bd_math_test.json', batch_size=64, eval_method='', model_path='../../Llama-3.1-8B/', model_type='llama3.1', output_dir='generate_result/zero_shot/bd_math/generation/llama3.1/1/', lora_path='', method='zero_shot', data_question_key='question', data_answer_key='answer', sample_num=1, cuda_ind=3, tensor_parallel=1, cuda_start=0, cuda_num=8, load_in_8bit=False, rewrite=False, use_typewriter=0, temperature=0.7, top_p=1, iter_max_new_tokens=512, init_max_new_tokens=2048, min_new_tokens=1, correct_response_format='The correct response is:', cot_trigger="Let's think step by step.")
+*****************************
+WARNING 10-21 17:32:19 arg_utils.py:953] Chunked prefill is enabled by default for models with max_model_len > 32K. Currently, chunked prefill might not work with some features or models. If you encounter any issues, please disable chunked prefill by setting --enable-chunked-prefill=False.
+INFO 10-21 17:32:19 config.py:1005] Chunked prefill is enabled with max_num_batched_tokens=512.
+INFO 10-21 17:32:19 llm_engine.py:237] Initializing an LLM engine (vdev) with config: model='../../Llama-3.1-8B/', speculative_config=None, tokenizer='../../Llama-3.1-8B/', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=../../Llama-3.1-8B/, use_v2_block_manager=True, num_scheduler_steps=1, chunked_prefill_enabled=True multi_step_stream_outputs=True, enable_prefix_caching=False, use_async_output_proc=True, use_cached_outputs=False, mm_processor_kwargs=None)
+[I1021 17:32:33.065192410 TCPStore.cpp:312] [c10d - debug] The server has started on port = 46409.
+[I1021 17:32:33.065353141 TCPStoreLibUvBackend.cpp:1067] [c10d - debug] Uv main loop running
+[I1021 17:32:33.066326430 socket.cpp:720] [c10d - debug] The client socket will attempt to connect to an IPv6 address of (10.117.192.77, 46409).
+[I1021 17:32:33.066436798 socket.cpp:884] [c10d] The client socket has connected to [n117-192-077.byted.org]:46409 on [n117-192-077.byted.org]:43090.
+[I1021 17:32:33.069877919 TCPStore.cpp:350] [c10d - debug] TCP client connected to host 10.117.192.77:46409
+[W1021 17:32:33.070299605 CUDAAllocatorConfig.h:28] Warning: expandable_segments not supported on this platform (function operator())
+[I1021 17:32:33.070355705 ProcessGroupNCCL.cpp:852] [PG 0 Rank 0] ProcessGroupNCCL initialization options: size: 1, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0, SPLIT_COLOR: 0, PG Name: 0
+[I1021 17:32:33.070363147 ProcessGroupNCCL.cpp:861] [PG 0 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank0]:[I1021 17:32:33.070867504 ProcessGroupNCCL.cpp:852] [PG 1 Rank 0] ProcessGroupNCCL initialization options: size: 1, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xa7dcab0, SPLIT_COLOR: 3389850942126204093, PG Name: 1
+[rank0]:[I1021 17:32:33.070884656 ProcessGroupNCCL.cpp:861] [PG 1 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank0]:[I1021 17:32:33.083784286 ProcessGroupNCCL.cpp:852] [PG 3 Rank 0] ProcessGroupNCCL initialization options: size: 1, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xa7dcab0, SPLIT_COLOR: 3389850942126204093, PG Name: 3
+[rank0]:[I1021 17:32:33.083804705 ProcessGroupNCCL.cpp:861] [PG 3 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank0]:[I1021 17:32:33.085332231 ProcessGroupNCCL.cpp:852] [PG 5 Rank 0] ProcessGroupNCCL initialization options: size: 1, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xa7dcab0, SPLIT_COLOR: 3389850942126204093, PG Name: 5
+[rank0]:[I1021 17:32:33.085350575 ProcessGroupNCCL.cpp:861] [PG 5 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+INFO 10-21 17:32:33 model_runner.py:1060] Starting to load model ../../Llama-3.1-8B/...
+
+
+
+
+
+
+
+INFO 10-21 17:32:38 model_runner.py:1071] Loading model weights took 14.9888 GB
+INFO 10-21 17:32:38 gpu_executor.py:122] # GPU blocks: 30099, # CPU blocks: 2048
+INFO 10-21 17:32:38 gpu_executor.py:126] Maximum concurrency for 131072 tokens per request: 3.67x
+INFO 10-21 17:32:41 model_runner.py:1402] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
+INFO 10-21 17:32:41 model_runner.py:1406] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
+INFO 10-21 17:32:52 model_runner.py:1530] Graph capturing finished in 10 secs.
+../../Llama-3.1-8B/
+load data
+Sampled Question:
+Question: Find the domain of the expression $\frac{\sqrt{x-2}}{\sqrt{5-x}}$.}
+Let's think step by step. The expressions inside each square root must be non-negative. Therefore, $x-2 \ge 0$, so $x\ge2$, and $5 - x \ge 0$, so $x \le 5$. Also, the denominator cannot be equal to zero, so $5-x>0$, which gives $x<5$. Therefore, the domain of the expression is $[2,5)$. Final Answer: The answer is $[2,5)$. I hope it is correct.
+
+Question: If $\det \mathbf{A} = 2$ and $\det \mathbf{B} = 12,$ then find $\det (\mathbf{A} \mathbf{B}).$
+Let's think step by step. We have that $\det (\mathbf{A} \mathbf{B}) = (\det \mathbf{A})(\det \mathbf{B}) = (2)(12) = 24.$ Final Answer: The answer is $24$. I hope it is correct.
+
+Question: Terrell usually lifts two 20-pound weights 12 times. If he uses two 15-pound weights instead, how many times must Terrell lift them in order to lift the same total weight?
+Let's think step by step. If Terrell lifts two 20-pound weights 12 times, he lifts a total of $2\cdot 12\cdot20=480$ pounds of weight. If he lifts two 15-pound weights instead for $n$ times, he will lift a total of $2\cdot15\cdot n=30n$ pounds of weight. Equating this to 480 pounds, we can solve for $n$:\begin{align*}
+30n&=480\
+\Rightarrow\qquad n&=480/30=16
+\end{align*}
+Final Answer: The answer is $16$. I hope it is correct.
+
+Question: If the system of equations
+
+\begin{align*}
+6x-4y&=a,\
+6y-9x &=b.
+\end{align*}
+has a solution $(x, y)$ where $x$ and $y$ are both nonzero, find $\frac{a}{b},$ assuming $b$ is nonzero.
+Let's think step by step. If we multiply the first equation by $-\frac{3}{2}$, we obtain $$6y-9x=-\frac{3}{2}a.$$Since we also know that $6y-9x=b$, we have
+$$-\frac{3}{2}a=b\Rightarrow\frac{a}{b}=-\frac{2}{3}.$$
+Final Answer: The answer is $-\frac{2}{3}$. I hope it is correct.
+
+Question: If 4 daps = 7 yaps, and 5 yaps = 3 baps, how many daps equal 42 baps?
+Let's think step by step.
+fitered 2688 already
+
  0%|          | 0/5 [00:00<?, ?it/s]
 20%|██        | 1/5 [00:32<02:10, 32.64s/it]
 40%|████      | 2/5 [01:05<01:38, 32.90s/it]
 60%|██████    | 3/5 [01:40<01:07, 33.68s/it]
 80%|████████  | 4/5 [02:14<00:33, 33.95s/it]

log/zero_shot/bd_math/generation/llama3.1/1/0-4.log

@@ -0,0 +1,70 @@
+[I1021 17:32:10.125052711 debug.cpp:49] [c10d] The debug level is set to INFO.
+/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/connections.py:8: RuntimeWarning: Failed to read commit hash:
+No module named 'vllm._version'
+  from vllm.version import __version__ as VLLM_VERSION
+llama3.1
+*****************************
+Namespace(cot_trigger_no=1, dataset='bd_math', data_path='bd_math_test.json', batch_size=64, eval_method='', model_path='../../Llama-3.1-8B/', model_type='llama3.1', output_dir='generate_result/zero_shot/bd_math/generation/llama3.1/1/', lora_path='', method='zero_shot', data_question_key='question', data_answer_key='answer', sample_num=1, cuda_ind=4, tensor_parallel=1, cuda_start=0, cuda_num=8, load_in_8bit=False, rewrite=False, use_typewriter=0, temperature=0.7, top_p=1, iter_max_new_tokens=512, init_max_new_tokens=2048, min_new_tokens=1, correct_response_format='The correct response is:', cot_trigger="Let's think step by step.")
+*****************************
+WARNING 10-21 17:32:19 arg_utils.py:953] Chunked prefill is enabled by default for models with max_model_len > 32K. Currently, chunked prefill might not work with some features or models. If you encounter any issues, please disable chunked prefill by setting --enable-chunked-prefill=False.
+INFO 10-21 17:32:19 config.py:1005] Chunked prefill is enabled with max_num_batched_tokens=512.
+INFO 10-21 17:32:19 llm_engine.py:237] Initializing an LLM engine (vdev) with config: model='../../Llama-3.1-8B/', speculative_config=None, tokenizer='../../Llama-3.1-8B/', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=../../Llama-3.1-8B/, use_v2_block_manager=True, num_scheduler_steps=1, chunked_prefill_enabled=True multi_step_stream_outputs=True, enable_prefix_caching=False, use_async_output_proc=True, use_cached_outputs=False, mm_processor_kwargs=None)
+[I1021 17:32:33.972711339 TCPStore.cpp:312] [c10d - debug] The server has started on port = 50257.
+[I1021 17:32:33.972854330 TCPStoreLibUvBackend.cpp:1067] [c10d - debug] Uv main loop running
+[I1021 17:32:33.973899496 socket.cpp:720] [c10d - debug] The client socket will attempt to connect to an IPv6 address of (10.117.192.77, 50257).
+[I1021 17:32:33.974076748 socket.cpp:884] [c10d] The client socket has connected to [n117-192-077.byted.org]:50257 on [n117-192-077.byted.org]:36032.
+[I1021 17:32:33.977247933 TCPStore.cpp:350] [c10d - debug] TCP client connected to host 10.117.192.77:50257
+[W1021 17:32:33.977705132 CUDAAllocatorConfig.h:28] Warning: expandable_segments not supported on this platform (function operator())
+[I1021 17:32:33.977759345 ProcessGroupNCCL.cpp:852] [PG 0 Rank 0] ProcessGroupNCCL initialization options: size: 1, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0, SPLIT_COLOR: 0, PG Name: 0
+[I1021 17:32:33.977767208 ProcessGroupNCCL.cpp:861] [PG 0 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank0]:[I1021 17:32:33.978313638 ProcessGroupNCCL.cpp:852] [PG 1 Rank 0] ProcessGroupNCCL initialization options: size: 1, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xc3d5b40, SPLIT_COLOR: 3389850942126204093, PG Name: 1
+[rank0]:[I1021 17:32:33.978330391 ProcessGroupNCCL.cpp:861] [PG 1 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank0]:[I1021 17:32:33.992085966 ProcessGroupNCCL.cpp:852] [PG 3 Rank 0] ProcessGroupNCCL initialization options: size: 1, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xc3d5b40, SPLIT_COLOR: 3389850942126204093, PG Name: 3
+[rank0]:[I1021 17:32:33.992106035 ProcessGroupNCCL.cpp:861] [PG 3 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank0]:[I1021 17:32:33.993880616 ProcessGroupNCCL.cpp:852] [PG 5 Rank 0] ProcessGroupNCCL initialization options: size: 1, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xc3d5b40, SPLIT_COLOR: 3389850942126204093, PG Name: 5
+[rank0]:[I1021 17:32:33.993899642 ProcessGroupNCCL.cpp:861] [PG 5 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+INFO 10-21 17:32:33 model_runner.py:1060] Starting to load model ../../Llama-3.1-8B/...
+
+
+
+
+
+
+
+INFO 10-21 17:32:38 model_runner.py:1071] Loading model weights took 14.9888 GB
+INFO 10-21 17:32:38 gpu_executor.py:122] # GPU blocks: 30099, # CPU blocks: 2048
+INFO 10-21 17:32:38 gpu_executor.py:126] Maximum concurrency for 131072 tokens per request: 3.67x
+INFO 10-21 17:32:41 model_runner.py:1402] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
+INFO 10-21 17:32:41 model_runner.py:1406] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
+INFO 10-21 17:32:52 model_runner.py:1530] Graph capturing finished in 11 secs.
+../../Llama-3.1-8B/
+load data
+Sampled Question:
+Question: Find the domain of the expression $\frac{\sqrt{x-2}}{\sqrt{5-x}}$.}
+Let's think step by step. The expressions inside each square root must be non-negative. Therefore, $x-2 \ge 0$, so $x\ge2$, and $5 - x \ge 0$, so $x \le 5$. Also, the denominator cannot be equal to zero, so $5-x>0$, which gives $x<5$. Therefore, the domain of the expression is $[2,5)$. Final Answer: The answer is $[2,5)$. I hope it is correct.
+
+Question: If $\det \mathbf{A} = 2$ and $\det \mathbf{B} = 12,$ then find $\det (\mathbf{A} \mathbf{B}).$
+Let's think step by step. We have that $\det (\mathbf{A} \mathbf{B}) = (\det \mathbf{A})(\det \mathbf{B}) = (2)(12) = 24.$ Final Answer: The answer is $24$. I hope it is correct.
+
+Question: Terrell usually lifts two 20-pound weights 12 times. If he uses two 15-pound weights instead, how many times must Terrell lift them in order to lift the same total weight?
+Let's think step by step. If Terrell lifts two 20-pound weights 12 times, he lifts a total of $2\cdot 12\cdot20=480$ pounds of weight. If he lifts two 15-pound weights instead for $n$ times, he will lift a total of $2\cdot15\cdot n=30n$ pounds of weight. Equating this to 480 pounds, we can solve for $n$:\begin{align*}
+30n&=480\
+\Rightarrow\qquad n&=480/30=16
+\end{align*}
+Final Answer: The answer is $16$. I hope it is correct.
+
+Question: If the system of equations
+
+\begin{align*}
+6x-4y&=a,\
+6y-9x &=b.
+\end{align*}
+has a solution $(x, y)$ where $x$ and $y$ are both nonzero, find $\frac{a}{b},$ assuming $b$ is nonzero.
+Let's think step by step. If we multiply the first equation by $-\frac{3}{2}$, we obtain $$6y-9x=-\frac{3}{2}a.$$Since we also know that $6y-9x=b$, we have
+$$-\frac{3}{2}a=b\Rightarrow\frac{a}{b}=-\frac{2}{3}.$$
+Final Answer: The answer is $-\frac{2}{3}$. I hope it is correct.
+
+Question: If 4 daps = 7 yaps, and 5 yaps = 3 baps, how many daps equal 42 baps?
+Let's think step by step.
+fitered 2688 already
+
  0%|          | 0/5 [00:00<?, ?it/s]
 20%|██        | 1/5 [00:32<02:08, 32.22s/it]
 40%|████      | 2/5 [01:04<01:36, 32.26s/it]
 60%|██████    | 3/5 [01:37<01:04, 32.46s/it]
 80%|████████  | 4/5 [02:08<00:31, 31.95s/it]

log/zero_shot/bd_math/generation/llama3.1/1/0-5.log

@@ -0,0 +1,70 @@
+[I1021 17:32:10.125040555 debug.cpp:49] [c10d] The debug level is set to INFO.
+/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/connections.py:8: RuntimeWarning: Failed to read commit hash:
+No module named 'vllm._version'
+  from vllm.version import __version__ as VLLM_VERSION
+llama3.1
+*****************************
+Namespace(cot_trigger_no=1, dataset='bd_math', data_path='bd_math_test.json', batch_size=64, eval_method='', model_path='../../Llama-3.1-8B/', model_type='llama3.1', output_dir='generate_result/zero_shot/bd_math/generation/llama3.1/1/', lora_path='', method='zero_shot', data_question_key='question', data_answer_key='answer', sample_num=1, cuda_ind=5, tensor_parallel=1, cuda_start=0, cuda_num=8, load_in_8bit=False, rewrite=False, use_typewriter=0, temperature=0.7, top_p=1, iter_max_new_tokens=512, init_max_new_tokens=2048, min_new_tokens=1, correct_response_format='The correct response is:', cot_trigger="Let's think step by step.")
+*****************************
+WARNING 10-21 17:32:19 arg_utils.py:953] Chunked prefill is enabled by default for models with max_model_len > 32K. Currently, chunked prefill might not work with some features or models. If you encounter any issues, please disable chunked prefill by setting --enable-chunked-prefill=False.
+INFO 10-21 17:32:19 config.py:1005] Chunked prefill is enabled with max_num_batched_tokens=512.
+INFO 10-21 17:32:19 llm_engine.py:237] Initializing an LLM engine (vdev) with config: model='../../Llama-3.1-8B/', speculative_config=None, tokenizer='../../Llama-3.1-8B/', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=../../Llama-3.1-8B/, use_v2_block_manager=True, num_scheduler_steps=1, chunked_prefill_enabled=True multi_step_stream_outputs=True, enable_prefix_caching=False, use_async_output_proc=True, use_cached_outputs=False, mm_processor_kwargs=None)
+[I1021 17:32:33.090602385 TCPStore.cpp:312] [c10d - debug] The server has started on port = 33905.
+[I1021 17:32:33.090752648 TCPStoreLibUvBackend.cpp:1067] [c10d - debug] Uv main loop running
+[I1021 17:32:33.091735533 socket.cpp:720] [c10d - debug] The client socket will attempt to connect to an IPv6 address of (10.117.192.77, 33905).
+[I1021 17:32:33.091839154 socket.cpp:884] [c10d] The client socket has connected to [n117-192-077.byted.org]:33905 on [n117-192-077.byted.org]:41502.
+[I1021 17:32:33.094560740 TCPStore.cpp:350] [c10d - debug] TCP client connected to host 10.117.192.77:33905
+[W1021 17:32:33.094958364 CUDAAllocatorConfig.h:28] Warning: expandable_segments not supported on this platform (function operator())
+[I1021 17:32:33.095016973 ProcessGroupNCCL.cpp:852] [PG 0 Rank 0] ProcessGroupNCCL initialization options: size: 1, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0, SPLIT_COLOR: 0, PG Name: 0
+[I1021 17:32:33.095024132 ProcessGroupNCCL.cpp:861] [PG 0 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank0]:[I1021 17:32:33.095509577 ProcessGroupNCCL.cpp:852] [PG 1 Rank 0] ProcessGroupNCCL initialization options: size: 1, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xc0910c0, SPLIT_COLOR: 3389850942126204093, PG Name: 1
+[rank0]:[I1021 17:32:33.095526138 ProcessGroupNCCL.cpp:861] [PG 1 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank0]:[I1021 17:32:33.108353125 ProcessGroupNCCL.cpp:852] [PG 3 Rank 0] ProcessGroupNCCL initialization options: size: 1, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xc0910c0, SPLIT_COLOR: 3389850942126204093, PG Name: 3
+[rank0]:[I1021 17:32:33.108373329 ProcessGroupNCCL.cpp:861] [PG 3 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank0]:[I1021 17:32:33.110149042 ProcessGroupNCCL.cpp:852] [PG 5 Rank 0] ProcessGroupNCCL initialization options: size: 1, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xc0910c0, SPLIT_COLOR: 3389850942126204093, PG Name: 5
+[rank0]:[I1021 17:32:33.110166605 ProcessGroupNCCL.cpp:861] [PG 5 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+INFO 10-21 17:32:33 model_runner.py:1060] Starting to load model ../../Llama-3.1-8B/...
+
+
+
+
+
+
+
+INFO 10-21 17:32:38 model_runner.py:1071] Loading model weights took 14.9888 GB
+INFO 10-21 17:32:38 gpu_executor.py:122] # GPU blocks: 30099, # CPU blocks: 2048
+INFO 10-21 17:32:38 gpu_executor.py:126] Maximum concurrency for 131072 tokens per request: 3.67x
+INFO 10-21 17:32:42 model_runner.py:1402] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
+INFO 10-21 17:32:42 model_runner.py:1406] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
+INFO 10-21 17:32:53 model_runner.py:1530] Graph capturing finished in 11 secs.
+../../Llama-3.1-8B/
+load data
+Sampled Question:
+Question: Find the domain of the expression $\frac{\sqrt{x-2}}{\sqrt{5-x}}$.}
+Let's think step by step. The expressions inside each square root must be non-negative. Therefore, $x-2 \ge 0$, so $x\ge2$, and $5 - x \ge 0$, so $x \le 5$. Also, the denominator cannot be equal to zero, so $5-x>0$, which gives $x<5$. Therefore, the domain of the expression is $[2,5)$. Final Answer: The answer is $[2,5)$. I hope it is correct.
+
+Question: If $\det \mathbf{A} = 2$ and $\det \mathbf{B} = 12,$ then find $\det (\mathbf{A} \mathbf{B}).$
+Let's think step by step. We have that $\det (\mathbf{A} \mathbf{B}) = (\det \mathbf{A})(\det \mathbf{B}) = (2)(12) = 24.$ Final Answer: The answer is $24$. I hope it is correct.
+
+Question: Terrell usually lifts two 20-pound weights 12 times. If he uses two 15-pound weights instead, how many times must Terrell lift them in order to lift the same total weight?
+Let's think step by step. If Terrell lifts two 20-pound weights 12 times, he lifts a total of $2\cdot 12\cdot20=480$ pounds of weight. If he lifts two 15-pound weights instead for $n$ times, he will lift a total of $2\cdot15\cdot n=30n$ pounds of weight. Equating this to 480 pounds, we can solve for $n$:\begin{align*}
+30n&=480\
+\Rightarrow\qquad n&=480/30=16
+\end{align*}
+Final Answer: The answer is $16$. I hope it is correct.
+
+Question: If the system of equations
+
+\begin{align*}
+6x-4y&=a,\
+6y-9x &=b.
+\end{align*}
+has a solution $(x, y)$ where $x$ and $y$ are both nonzero, find $\frac{a}{b},$ assuming $b$ is nonzero.
+Let's think step by step. If we multiply the first equation by $-\frac{3}{2}$, we obtain $$6y-9x=-\frac{3}{2}a.$$Since we also know that $6y-9x=b$, we have
+$$-\frac{3}{2}a=b\Rightarrow\frac{a}{b}=-\frac{2}{3}.$$
+Final Answer: The answer is $-\frac{2}{3}$. I hope it is correct.
+
+Question: If 4 daps = 7 yaps, and 5 yaps = 3 baps, how many daps equal 42 baps?
+Let's think step by step.
+fitered 2688 already
+
  0%|          | 0/5 [00:00<?, ?it/s]
 20%|██        | 1/5 [00:32<02:10, 32.59s/it]
 40%|████      | 2/5 [01:04<01:37, 32.48s/it]
 60%|██████    | 3/5 [01:37<01:04, 32.45s/it]
 80%|████████  | 4/5 [02:09<00:32, 32.38s/it]

log/zero_shot/bd_math/generation/llama3.1/1/0-6.log

@@ -0,0 +1,70 @@
+[I1021 17:32:10.125199301 debug.cpp:49] [c10d] The debug level is set to INFO.
+/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/connections.py:8: RuntimeWarning: Failed to read commit hash:
+No module named 'vllm._version'
+  from vllm.version import __version__ as VLLM_VERSION
+llama3.1
+*****************************
+Namespace(cot_trigger_no=1, dataset='bd_math', data_path='bd_math_test.json', batch_size=64, eval_method='', model_path='../../Llama-3.1-8B/', model_type='llama3.1', output_dir='generate_result/zero_shot/bd_math/generation/llama3.1/1/', lora_path='', method='zero_shot', data_question_key='question', data_answer_key='answer', sample_num=1, cuda_ind=6, tensor_parallel=1, cuda_start=0, cuda_num=8, load_in_8bit=False, rewrite=False, use_typewriter=0, temperature=0.7, top_p=1, iter_max_new_tokens=512, init_max_new_tokens=2048, min_new_tokens=1, correct_response_format='The correct response is:', cot_trigger="Let's think step by step.")
+*****************************
+WARNING 10-21 17:32:19 arg_utils.py:953] Chunked prefill is enabled by default for models with max_model_len > 32K. Currently, chunked prefill might not work with some features or models. If you encounter any issues, please disable chunked prefill by setting --enable-chunked-prefill=False.
+INFO 10-21 17:32:19 config.py:1005] Chunked prefill is enabled with max_num_batched_tokens=512.
+INFO 10-21 17:32:19 llm_engine.py:237] Initializing an LLM engine (vdev) with config: model='../../Llama-3.1-8B/', speculative_config=None, tokenizer='../../Llama-3.1-8B/', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=../../Llama-3.1-8B/, use_v2_block_manager=True, num_scheduler_steps=1, chunked_prefill_enabled=True multi_step_stream_outputs=True, enable_prefix_caching=False, use_async_output_proc=True, use_cached_outputs=False, mm_processor_kwargs=None)
+[I1021 17:32:33.074960847 TCPStore.cpp:312] [c10d - debug] The server has started on port = 46227.
+[I1021 17:32:33.074978008 TCPStoreLibUvBackend.cpp:1067] [c10d - debug] Uv main loop running
+[I1021 17:32:33.076109583 socket.cpp:720] [c10d - debug] The client socket will attempt to connect to an IPv6 address of (10.117.192.77, 46227).
+[I1021 17:32:33.076215237 socket.cpp:884] [c10d] The client socket has connected to [n117-192-077.byted.org]:46227 on [n117-192-077.byted.org]:61802.
+[I1021 17:32:33.079088203 TCPStore.cpp:350] [c10d - debug] TCP client connected to host 10.117.192.77:46227
+[W1021 17:32:33.079563703 CUDAAllocatorConfig.h:28] Warning: expandable_segments not supported on this platform (function operator())
+[I1021 17:32:33.079628503 ProcessGroupNCCL.cpp:852] [PG 0 Rank 0] ProcessGroupNCCL initialization options: size: 1, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0, SPLIT_COLOR: 0, PG Name: 0
+[I1021 17:32:33.079635879 ProcessGroupNCCL.cpp:861] [PG 0 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank0]:[I1021 17:32:33.080124838 ProcessGroupNCCL.cpp:852] [PG 1 Rank 0] ProcessGroupNCCL initialization options: size: 1, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xac4d000, SPLIT_COLOR: 3389850942126204093, PG Name: 1
+[rank0]:[I1021 17:32:33.080142722 ProcessGroupNCCL.cpp:861] [PG 1 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank0]:[I1021 17:32:33.092970648 ProcessGroupNCCL.cpp:852] [PG 3 Rank 0] ProcessGroupNCCL initialization options: size: 1, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xac4d000, SPLIT_COLOR: 3389850942126204093, PG Name: 3
+[rank0]:[I1021 17:32:33.092991704 ProcessGroupNCCL.cpp:861] [PG 3 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank0]:[I1021 17:32:33.094497493 ProcessGroupNCCL.cpp:852] [PG 5 Rank 0] ProcessGroupNCCL initialization options: size: 1, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xac4d000, SPLIT_COLOR: 3389850942126204093, PG Name: 5
+[rank0]:[I1021 17:32:33.094515803 ProcessGroupNCCL.cpp:861] [PG 5 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+INFO 10-21 17:32:33 model_runner.py:1060] Starting to load model ../../Llama-3.1-8B/...
+
+
+
+
+
+
+
+INFO 10-21 17:32:38 model_runner.py:1071] Loading model weights took 14.9888 GB
+INFO 10-21 17:32:38 gpu_executor.py:122] # GPU blocks: 30099, # CPU blocks: 2048
+INFO 10-21 17:32:38 gpu_executor.py:126] Maximum concurrency for 131072 tokens per request: 3.67x
+INFO 10-21 17:32:42 model_runner.py:1402] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
+INFO 10-21 17:32:42 model_runner.py:1406] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
+INFO 10-21 17:32:53 model_runner.py:1530] Graph capturing finished in 11 secs.
+../../Llama-3.1-8B/
+load data
+Sampled Question:
+Question: Find the domain of the expression $\frac{\sqrt{x-2}}{\sqrt{5-x}}$.}
+Let's think step by step. The expressions inside each square root must be non-negative. Therefore, $x-2 \ge 0$, so $x\ge2$, and $5 - x \ge 0$, so $x \le 5$. Also, the denominator cannot be equal to zero, so $5-x>0$, which gives $x<5$. Therefore, the domain of the expression is $[2,5)$. Final Answer: The answer is $[2,5)$. I hope it is correct.
+
+Question: If $\det \mathbf{A} = 2$ and $\det \mathbf{B} = 12,$ then find $\det (\mathbf{A} \mathbf{B}).$
+Let's think step by step. We have that $\det (\mathbf{A} \mathbf{B}) = (\det \mathbf{A})(\det \mathbf{B}) = (2)(12) = 24.$ Final Answer: The answer is $24$. I hope it is correct.
+
+Question: Terrell usually lifts two 20-pound weights 12 times. If he uses two 15-pound weights instead, how many times must Terrell lift them in order to lift the same total weight?
+Let's think step by step. If Terrell lifts two 20-pound weights 12 times, he lifts a total of $2\cdot 12\cdot20=480$ pounds of weight. If he lifts two 15-pound weights instead for $n$ times, he will lift a total of $2\cdot15\cdot n=30n$ pounds of weight. Equating this to 480 pounds, we can solve for $n$:\begin{align*}
+30n&=480\
+\Rightarrow\qquad n&=480/30=16
+\end{align*}
+Final Answer: The answer is $16$. I hope it is correct.
+
+Question: If the system of equations
+
+\begin{align*}
+6x-4y&=a,\
+6y-9x &=b.
+\end{align*}
+has a solution $(x, y)$ where $x$ and $y$ are both nonzero, find $\frac{a}{b},$ assuming $b$ is nonzero.
+Let's think step by step. If we multiply the first equation by $-\frac{3}{2}$, we obtain $$6y-9x=-\frac{3}{2}a.$$Since we also know that $6y-9x=b$, we have
+$$-\frac{3}{2}a=b\Rightarrow\frac{a}{b}=-\frac{2}{3}.$$
+Final Answer: The answer is $-\frac{2}{3}$. I hope it is correct.
+
+Question: If 4 daps = 7 yaps, and 5 yaps = 3 baps, how many daps equal 42 baps?
+Let's think step by step.
+fitered 2688 already
+
  0%|          | 0/5 [00:00<?, ?it/s]
 20%|██        | 1/5 [00:32<02:08, 32.20s/it]
 40%|████      | 2/5 [01:05<01:37, 32.65s/it]
 60%|██████    | 3/5 [01:38<01:05, 32.83s/it]
 80%|████████  | 4/5 [02:10<00:32, 32.74s/it]

log/zero_shot/bd_math/generation/llama3.1/1/0-7.log

@@ -0,0 +1,70 @@
+[I1021 17:32:10.135778790 debug.cpp:49] [c10d] The debug level is set to INFO.
+/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/connections.py:8: RuntimeWarning: Failed to read commit hash:
+No module named 'vllm._version'
+  from vllm.version import __version__ as VLLM_VERSION
+llama3.1
+*****************************
+Namespace(cot_trigger_no=1, dataset='bd_math', data_path='bd_math_test.json', batch_size=64, eval_method='', model_path='../../Llama-3.1-8B/', model_type='llama3.1', output_dir='generate_result/zero_shot/bd_math/generation/llama3.1/1/', lora_path='', method='zero_shot', data_question_key='question', data_answer_key='answer', sample_num=1, cuda_ind=7, tensor_parallel=1, cuda_start=0, cuda_num=8, load_in_8bit=False, rewrite=False, use_typewriter=0, temperature=0.7, top_p=1, iter_max_new_tokens=512, init_max_new_tokens=2048, min_new_tokens=1, correct_response_format='The correct response is:', cot_trigger="Let's think step by step.")
+*****************************
+WARNING 10-21 17:32:20 arg_utils.py:953] Chunked prefill is enabled by default for models with max_model_len > 32K. Currently, chunked prefill might not work with some features or models. If you encounter any issues, please disable chunked prefill by setting --enable-chunked-prefill=False.
+INFO 10-21 17:32:20 config.py:1005] Chunked prefill is enabled with max_num_batched_tokens=512.
+INFO 10-21 17:32:20 llm_engine.py:237] Initializing an LLM engine (vdev) with config: model='../../Llama-3.1-8B/', speculative_config=None, tokenizer='../../Llama-3.1-8B/', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=../../Llama-3.1-8B/, use_v2_block_manager=True, num_scheduler_steps=1, chunked_prefill_enabled=True multi_step_stream_outputs=True, enable_prefix_caching=False, use_async_output_proc=True, use_cached_outputs=False, mm_processor_kwargs=None)
+[I1021 17:32:33.334206825 TCPStore.cpp:312] [c10d - debug] The server has started on port = 38167.
+[I1021 17:32:33.334229091 TCPStoreLibUvBackend.cpp:1067] [c10d - debug] Uv main loop running
+[I1021 17:32:33.335374254 socket.cpp:720] [c10d - debug] The client socket will attempt to connect to an IPv6 address of (10.117.192.77, 38167).
+[I1021 17:32:33.335531417 socket.cpp:884] [c10d] The client socket has connected to [n117-192-077.byted.org]:38167 on [n117-192-077.byted.org]:49140.
+[I1021 17:32:33.338344306 TCPStore.cpp:350] [c10d - debug] TCP client connected to host 10.117.192.77:38167
+[W1021 17:32:33.338772786 CUDAAllocatorConfig.h:28] Warning: expandable_segments not supported on this platform (function operator())
+[I1021 17:32:33.338836768 ProcessGroupNCCL.cpp:852] [PG 0 Rank 0] ProcessGroupNCCL initialization options: size: 1, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0, SPLIT_COLOR: 0, PG Name: 0
+[I1021 17:32:33.338844491 ProcessGroupNCCL.cpp:861] [PG 0 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank0]:[I1021 17:32:33.339392327 ProcessGroupNCCL.cpp:852] [PG 1 Rank 0] ProcessGroupNCCL initialization options: size: 1, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xb07b820, SPLIT_COLOR: 3389850942126204093, PG Name: 1
+[rank0]:[I1021 17:32:33.339413264 ProcessGroupNCCL.cpp:861] [PG 1 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank0]:[I1021 17:32:33.352959977 ProcessGroupNCCL.cpp:852] [PG 3 Rank 0] ProcessGroupNCCL initialization options: size: 1, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xb07b820, SPLIT_COLOR: 3389850942126204093, PG Name: 3
+[rank0]:[I1021 17:32:33.352979191 ProcessGroupNCCL.cpp:861] [PG 3 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank0]:[I1021 17:32:33.354472064 ProcessGroupNCCL.cpp:852] [PG 5 Rank 0] ProcessGroupNCCL initialization options: size: 1, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xb07b820, SPLIT_COLOR: 3389850942126204093, PG Name: 5
+[rank0]:[I1021 17:32:33.354489641 ProcessGroupNCCL.cpp:861] [PG 5 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+INFO 10-21 17:32:33 model_runner.py:1060] Starting to load model ../../Llama-3.1-8B/...
+
+
+
+
+
+
+
+INFO 10-21 17:32:38 model_runner.py:1071] Loading model weights took 14.9888 GB
+INFO 10-21 17:32:38 gpu_executor.py:122] # GPU blocks: 30099, # CPU blocks: 2048
+INFO 10-21 17:32:38 gpu_executor.py:126] Maximum concurrency for 131072 tokens per request: 3.67x
+INFO 10-21 17:32:42 model_runner.py:1402] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
+INFO 10-21 17:32:42 model_runner.py:1406] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
+INFO 10-21 17:32:53 model_runner.py:1530] Graph capturing finished in 11 secs.
+../../Llama-3.1-8B/
+load data
+Sampled Question:
+Question: Find the domain of the expression $\frac{\sqrt{x-2}}{\sqrt{5-x}}$.}
+Let's think step by step. The expressions inside each square root must be non-negative. Therefore, $x-2 \ge 0$, so $x\ge2$, and $5 - x \ge 0$, so $x \le 5$. Also, the denominator cannot be equal to zero, so $5-x>0$, which gives $x<5$. Therefore, the domain of the expression is $[2,5)$. Final Answer: The answer is $[2,5)$. I hope it is correct.
+
+Question: If $\det \mathbf{A} = 2$ and $\det \mathbf{B} = 12,$ then find $\det (\mathbf{A} \mathbf{B}).$
+Let's think step by step. We have that $\det (\mathbf{A} \mathbf{B}) = (\det \mathbf{A})(\det \mathbf{B}) = (2)(12) = 24.$ Final Answer: The answer is $24$. I hope it is correct.
+
+Question: Terrell usually lifts two 20-pound weights 12 times. If he uses two 15-pound weights instead, how many times must Terrell lift them in order to lift the same total weight?
+Let's think step by step. If Terrell lifts two 20-pound weights 12 times, he lifts a total of $2\cdot 12\cdot20=480$ pounds of weight. If he lifts two 15-pound weights instead for $n$ times, he will lift a total of $2\cdot15\cdot n=30n$ pounds of weight. Equating this to 480 pounds, we can solve for $n$:\begin{align*}
+30n&=480\
+\Rightarrow\qquad n&=480/30=16
+\end{align*}
+Final Answer: The answer is $16$. I hope it is correct.
+
+Question: If the system of equations
+
+\begin{align*}
+6x-4y&=a,\
+6y-9x &=b.
+\end{align*}
+has a solution $(x, y)$ where $x$ and $y$ are both nonzero, find $\frac{a}{b},$ assuming $b$ is nonzero.
+Let's think step by step. If we multiply the first equation by $-\frac{3}{2}$, we obtain $$6y-9x=-\frac{3}{2}a.$$Since we also know that $6y-9x=b$, we have
+$$-\frac{3}{2}a=b\Rightarrow\frac{a}{b}=-\frac{2}{3}.$$
+Final Answer: The answer is $-\frac{2}{3}$. I hope it is correct.
+
+Question: If 4 daps = 7 yaps, and 5 yaps = 3 baps, how many daps equal 42 baps?
+Let's think step by step.
+fitered 2688 already
+
  0%|          | 0/5 [00:00<?, ?it/s]
 20%|██        | 1/5 [00:33<02:12, 33.13s/it]
 40%|████      | 2/5 [01:06<01:40, 33.41s/it]
 60%|██████    | 3/5 [01:39<01:05, 32.97s/it]
 80%|████████  | 4/5 [02:13<00:33, 33.48s/it]

log/zero_shot/bd_math/generation/llama3.1_70b/1/0-0.log

@@ -0,0 +1,694 @@
+[I1021 20:06:01.507189142 debug.cpp:49] [c10d] The debug level is set to INFO.
+/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/connections.py:8: RuntimeWarning: Failed to read commit hash:
+No module named 'vllm._version'
+  from vllm.version import __version__ as VLLM_VERSION
+llama3.1_70b
+*****************************
+Namespace(cot_trigger_no=1, dataset='bd_math', data_path='bd_math_test.json', batch_size=64, eval_method='', model_path='../../Meta-Llama-3.1-70B', model_type='llama3.1_70b', output_dir='generate_result/zero_shot/bd_math/generation/llama3.1_70b/1/', lora_path='', method='zero_shot', data_question_key='question', data_answer_key='answer', sample_num=1, cuda_ind=0, tensor_parallel=8, cuda_start=0, cuda_num=8, load_in_8bit=False, rewrite=False, use_typewriter=0, temperature=0.7, top_p=1, iter_max_new_tokens=512, init_max_new_tokens=2048, min_new_tokens=1, correct_response_format='The correct response is:', cot_trigger="Let's think step by step.")
+*****************************
+INFO 10-21 20:06:08 config.py:887] Defaulting to use mp for distributed inference
+WARNING 10-21 20:06:08 arg_utils.py:953] Chunked prefill is enabled by default for models with max_model_len > 32K. Currently, chunked prefill might not work with some features or models. If you encounter any issues, please disable chunked prefill by setting --enable-chunked-prefill=False.
+INFO 10-21 20:06:08 config.py:1005] Chunked prefill is enabled with max_num_batched_tokens=512.
+INFO 10-21 20:06:08 llm_engine.py:237] Initializing an LLM engine (vdev) with config: model='../../Meta-Llama-3.1-70B', speculative_config=None, tokenizer='../../Meta-Llama-3.1-70B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=8, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=../../Meta-Llama-3.1-70B, use_v2_block_manager=True, num_scheduler_steps=1, chunked_prefill_enabled=True multi_step_stream_outputs=True, enable_prefix_caching=False, use_async_output_proc=True, use_cached_outputs=False, mm_processor_kwargs=None)
+INFO 10-21 20:06:08 custom_cache_manager.py:17] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager
+(VllmWorkerProcess pid=76761) INFO 10-21 20:06:09 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
+(VllmWorkerProcess pid=76762) INFO 10-21 20:06:10 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
+(VllmWorkerProcess pid=76766) INFO 10-21 20:06:10 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
+(VllmWorkerProcess pid=76764) INFO 10-21 20:06:10 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
+(VllmWorkerProcess pid=76763) INFO 10-21 20:06:10 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
+(VllmWorkerProcess pid=76765) INFO 10-21 20:06:10 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
+(VllmWorkerProcess pid=76767) INFO 10-21 20:06:10 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
+[I1021 20:06:18.860192414 TCPStore.cpp:312] [c10d - debug] The server has started on port = 59993.
+[I1021 20:06:18.860219808 TCPStoreLibUvBackend.cpp:1067] [c10d - debug] Uv main loop running
+[I1021 20:06:18.865365373 socket.cpp:720] [c10d - debug] The client socket will attempt to connect to an IPv6 address of (127.0.0.1, 59993).
+[I1021 20:06:18.865511282 socket.cpp:884] [c10d] The client socket has connected to [localhost]:59993 on [localhost]:43706.
+[I1021 20:06:18.868928831 TCPStore.cpp:350] [c10d - debug] TCP client connected to host 127.0.0.1:59993
+[I1021 20:06:21.965674859 socket.cpp:720] [c10d - debug] The client socket will attempt to connect to an IPv6 address of (127.0.0.1, 59993).
+[I1021 20:06:21.965952600 socket.cpp:884] [c10d] The client socket has connected to [localhost]:59993 on [localhost]:43722.
+[I1021 20:06:21.969862366 TCPStore.cpp:350] [c10d - debug] TCP client connected to host 127.0.0.1:59993
+[W1021 20:06:21.970432129 CUDAAllocatorConfig.h:28] Warning: expandable_segments not supported on this platform (function operator())
+[I1021 20:06:21.970516770 ProcessGroupNCCL.cpp:852] [PG 0 Rank 1] ProcessGroupNCCL initialization options: size: 8, global rank: 1, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0, SPLIT_COLOR: 0, PG Name: 0
+[I1021 20:06:21.970527945 ProcessGroupNCCL.cpp:861] [PG 0 Rank 1] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank1]:[I1021 20:06:21.971263049 ProcessGroupNCCL.cpp:852] [PG 1 Rank 1] ProcessGroupNCCL initialization options: size: 8, global rank: 1, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xbfffdb0, SPLIT_COLOR: 4318754687966092895, PG Name: 1
+[rank1]:[I1021 20:06:21.971273302 ProcessGroupNCCL.cpp:861] [PG 1 Rank 1] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[I1021 20:06:23.273097130 socket.cpp:720] [c10d - debug] The client socket will attempt to connect to an IPv6 address of (127.0.0.1, 59993).
+[I1021 20:06:23.273345617 socket.cpp:884] [c10d] The client socket has connected to [localhost]:59993 on [localhost]:54358.
+[I1021 20:06:23.276727555 TCPStore.cpp:350] [c10d - debug] TCP client connected to host 127.0.0.1:59993
+[W1021 20:06:23.277421295 CUDAAllocatorConfig.h:28] Warning: expandable_segments not supported on this platform (function operator())
+[I1021 20:06:23.277538066 ProcessGroupNCCL.cpp:852] [PG 0 Rank 2] ProcessGroupNCCL initialization options: size: 8, global rank: 2, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0, SPLIT_COLOR: 0, PG Name: 0
+[I1021 20:06:23.277551315 ProcessGroupNCCL.cpp:861] [PG 0 Rank 2] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank2]:[I1021 20:06:23.278491857 ProcessGroupNCCL.cpp:852] [PG 1 Rank 2] ProcessGroupNCCL initialization options: size: 8, global rank: 2, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xbfff790, SPLIT_COLOR: 4318754687966092895, PG Name: 1
+[rank2]:[I1021 20:06:23.278506361 ProcessGroupNCCL.cpp:861] [PG 1 Rank 2] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[I1021 20:06:23.444586505 socket.cpp:720] [c10d - debug] The client socket will attempt to connect to an IPv6 address of (127.0.0.1, 59993).
+[I1021 20:06:23.444804251 socket.cpp:884] [c10d] The client socket has connected to [localhost]:59993 on [localhost]:54364.
+[I1021 20:06:23.447833718 TCPStore.cpp:350] [c10d - debug] TCP client connected to host 127.0.0.1:59993
+[W1021 20:06:23.448652643 CUDAAllocatorConfig.h:28] Warning: expandable_segments not supported on this platform (function operator())
+[I1021 20:06:23.448808736 ProcessGroupNCCL.cpp:852] [PG 0 Rank 5] ProcessGroupNCCL initialization options: size: 8, global rank: 5, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0, SPLIT_COLOR: 0, PG Name: 0
+[I1021 20:06:23.448821928 ProcessGroupNCCL.cpp:861] [PG 0 Rank 5] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[I1021 20:06:23.449689011 socket.cpp:720] [c10d - debug] The client socket will attempt to connect to an IPv6 address of (127.0.0.1, 59993).
+[rank5]:[I1021 20:06:23.449844165 ProcessGroupNCCL.cpp:852] [PG 1 Rank 5] ProcessGroupNCCL initialization options: size: 8, global rank: 5, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xbffabc0, SPLIT_COLOR: 4318754687966092895, PG Name: 1
+[rank5]:[I1021 20:06:23.449859165 ProcessGroupNCCL.cpp:861] [PG 1 Rank 5] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[I1021 20:06:23.449901305 socket.cpp:884] [c10d] The client socket has connected to [localhost]:59993 on [localhost]:54366.
+[I1021 20:06:23.453270102 TCPStore.cpp:350] [c10d - debug] TCP client connected to host 127.0.0.1:59993
+[W1021 20:06:23.453607173 CUDAAllocatorConfig.h:28] Warning: expandable_segments not supported on this platform (function operator())
+[I1021 20:06:23.453667514 ProcessGroupNCCL.cpp:852] [PG 0 Rank 7] ProcessGroupNCCL initialization options: size: 8, global rank: 7, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0, SPLIT_COLOR: 0, PG Name: 0
+[I1021 20:06:23.453676210 ProcessGroupNCCL.cpp:861] [PG 0 Rank 7] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank7]:[I1021 20:06:23.454262625 ProcessGroupNCCL.cpp:852] [PG 1 Rank 7] ProcessGroupNCCL initialization options: size: 8, global rank: 7, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xbffa2f0, SPLIT_COLOR: 4318754687966092895, PG Name: 1
+[rank7]:[I1021 20:06:23.454273108 ProcessGroupNCCL.cpp:861] [PG 1 Rank 7] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[I1021 20:06:23.466284996 socket.cpp:720] [c10d - debug] The client socket will attempt to connect to an IPv6 address of (127.0.0.1, 59993).
+[I1021 20:06:23.466491124 socket.cpp:884] [c10d] The client socket has connected to [localhost]:59993 on [localhost]:54378.
+[I1021 20:06:23.470102934 TCPStore.cpp:350] [c10d - debug] TCP client connected to host 127.0.0.1:59993
+[W1021 20:06:23.470644552 CUDAAllocatorConfig.h:28] Warning: expandable_segments not supported on this platform (function operator())
+[I1021 20:06:23.470751865 ProcessGroupNCCL.cpp:852] [PG 0 Rank 4] ProcessGroupNCCL initialization options: size: 8, global rank: 4, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0, SPLIT_COLOR: 0, PG Name: 0
+[I1021 20:06:23.470764734 ProcessGroupNCCL.cpp:861] [PG 0 Rank 4] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank4]:[I1021 20:06:23.471662944 ProcessGroupNCCL.cpp:852] [PG 1 Rank 4] ProcessGroupNCCL initialization options: size: 8, global rank: 4, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xbfffbf0, SPLIT_COLOR: 4318754687966092895, PG Name: 1
+[rank4]:[I1021 20:06:23.471679305 ProcessGroupNCCL.cpp:861] [PG 1 Rank 4] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[I1021 20:06:23.474160519 socket.cpp:720] [c10d - debug] The client socket will attempt to connect to an IPv6 address of (127.0.0.1, 59993).
+[I1021 20:06:23.476517856 socket.cpp:720] [c10d - debug] The client socket will attempt to connect to an IPv6 address of (127.0.0.1, 59993).
+[I1021 20:06:23.474331404 socket.cpp:884] [c10d] The client socket has connected to [localhost]:59993 on [localhost]:54386.
+[I1021 20:06:23.477069035 TCPStore.cpp:350] [c10d - debug] TCP client connected to host 127.0.0.1:59993
+[W1021 20:06:23.477501669 CUDAAllocatorConfig.h:28] Warning: expandable_segments not supported on this platform (function operator())
+[I1021 20:06:23.477586586 ProcessGroupNCCL.cpp:852] [PG 0 Rank 3] ProcessGroupNCCL initialization options: size: 8, global rank: 3, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0, SPLIT_COLOR: 0, PG Name: 0
+[I1021 20:06:23.477595590 ProcessGroupNCCL.cpp:861] [PG 0 Rank 3] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank3]:[I1021 20:06:23.478316936 ProcessGroupNCCL.cpp:852] [PG 1 Rank 3] ProcessGroupNCCL initialization options: size: 8, global rank: 3, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xc000360, SPLIT_COLOR: 4318754687966092895, PG Name: 1
+[rank3]:[I1021 20:06:23.478335862 ProcessGroupNCCL.cpp:861] [PG 1 Rank 3] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[I1021 20:06:23.476669028 socket.cpp:884] [c10d] The client socket has connected to [localhost]:59993 on [localhost]:54394.
+[I1021 20:06:23.479261648 TCPStore.cpp:350] [c10d - debug] TCP client connected to host 127.0.0.1:59993
+[W1021 20:06:23.479708475 CUDAAllocatorConfig.h:28] Warning: expandable_segments not supported on this platform (function operator())
+[I1021 20:06:23.479775433 ProcessGroupNCCL.cpp:852] [PG 0 Rank 6] ProcessGroupNCCL initialization options: size: 8, global rank: 6, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0, SPLIT_COLOR: 0, PG Name: 0
+[I1021 20:06:23.479782269 ProcessGroupNCCL.cpp:861] [PG 0 Rank 6] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[W1021 20:06:23.479883861 CUDAAllocatorConfig.h:28] Warning: expandable_segments not supported on this platform (function operator())
+[I1021 20:06:23.479973809 ProcessGroupNCCL.cpp:852] [PG 0 Rank 0] ProcessGroupNCCL initialization options: size: 8, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0, SPLIT_COLOR: 0, PG Name: 0
+[I1021 20:06:23.479983005 ProcessGroupNCCL.cpp:861] [PG 0 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank6]:[I1021 20:06:23.480405939 ProcessGroupNCCL.cpp:852] [PG 1 Rank 6] ProcessGroupNCCL initialization options: size: 8, global rank: 6, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xbffa7a0, SPLIT_COLOR: 4318754687966092895, PG Name: 1
+[rank6]:[I1021 20:06:23.480416985 ProcessGroupNCCL.cpp:861] [PG 1 Rank 6] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank0]:[I1021 20:06:23.480719766 ProcessGroupNCCL.cpp:852] [PG 1 Rank 0] ProcessGroupNCCL initialization options: size: 8, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xbfcd5e0, SPLIT_COLOR: 4318754687966092895, PG Name: 1
+[rank0]:[I1021 20:06:23.480735964 ProcessGroupNCCL.cpp:861] [PG 1 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank0]:[I1021 20:06:23.497096299 ProcessGroupNCCL.cpp:852] [PG 3 Rank 0] ProcessGroupNCCL initialization options: size: 8, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xbfcd5e0, SPLIT_COLOR: 4318754687966092895, PG Name: 3
+[rank0]:[I1021 20:06:23.497116924 ProcessGroupNCCL.cpp:861] [PG 3 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank1]:[I1021 20:06:23.497151644 ProcessGroupNCCL.cpp:852] [PG 3 Rank 1] ProcessGroupNCCL initialization options: size: 8, global rank: 1, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xbfffdb0, SPLIT_COLOR: 4318754687966092895, PG Name: 3
+[rank1]:[I1021 20:06:23.497172942 ProcessGroupNCCL.cpp:861] [PG 3 Rank 1] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank3]:[I1021 20:06:23.497358021 ProcessGroupNCCL.cpp:852] [PG 3 Rank 3] ProcessGroupNCCL initialization options: size: 8, global rank: 3, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xc000360, SPLIT_COLOR: 4318754687966092895, PG Name: 3
+[rank3]:[I1021 20:06:23.497380186 ProcessGroupNCCL.cpp:861] [PG 3 Rank 3] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank4]:[I1021 20:06:23.497686688 ProcessGroupNCCL.cpp:852] [PG 3 Rank 4] ProcessGroupNCCL initialization options: size: 8, global rank: 4, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xbfffbf0, SPLIT_COLOR: 4318754687966092895, PG Name: 3
+[rank4]:[I1021 20:06:23.497706877 ProcessGroupNCCL.cpp:861] [PG 3 Rank 4] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank2]:[I1021 20:06:23.497772323 ProcessGroupNCCL.cpp:852] [PG 3 Rank 2] ProcessGroupNCCL initialization options: size: 8, global rank: 2, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xbfff790, SPLIT_COLOR: 4318754687966092895, PG Name: 3
+[rank2]:[I1021 20:06:23.497794727 ProcessGroupNCCL.cpp:861] [PG 3 Rank 2] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank5]:[I1021 20:06:23.506746830 ProcessGroupNCCL.cpp:852] [PG 3 Rank 5] ProcessGroupNCCL initialization options: size: 8, global rank: 5, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xbffabc0, SPLIT_COLOR: 4318754687966092895, PG Name: 3
+[rank5]:[I1021 20:06:23.506767224 ProcessGroupNCCL.cpp:861] [PG 3 Rank 5] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank6]:[I1021 20:06:23.508461369 ProcessGroupNCCL.cpp:852] [PG 3 Rank 6] ProcessGroupNCCL initialization options: size: 8, global rank: 6, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xbffa7a0, SPLIT_COLOR: 4318754687966092895, PG Name: 3
+[rank6]:[I1021 20:06:23.508482893 ProcessGroupNCCL.cpp:861] [PG 3 Rank 6] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank7]:[I1021 20:06:23.508495989 ProcessGroupNCCL.cpp:852] [PG 3 Rank 7] ProcessGroupNCCL initialization options: size: 8, global rank: 7, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xbffa2f0, SPLIT_COLOR: 4318754687966092895, PG Name: 3
+[rank7]:[I1021 20:06:23.508517065 ProcessGroupNCCL.cpp:861] [PG 3 Rank 7] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+(VllmWorkerProcess pid=76762) INFO 10-21 20:06:23 utils.py:1008] Found nccl from library libnccl.so.2
+(VllmWorkerProcess pid=76761) INFO 10-21 20:06:23 utils.py:1008] Found nccl from library libnccl.so.2
+INFO 10-21 20:06:23 utils.py:1008] Found nccl from library libnccl.so.2
+(VllmWorkerProcess pid=76764) INFO 10-21 20:06:23 utils.py:1008] Found nccl from library libnccl.so.2
+(VllmWorkerProcess pid=76762) INFO 10-21 20:06:23 pynccl.py:63] vLLM is using nccl==2.20.5
+(VllmWorkerProcess pid=76761) INFO 10-21 20:06:23 pynccl.py:63] vLLM is using nccl==2.20.5
+INFO 10-21 20:06:23 pynccl.py:63] vLLM is using nccl==2.20.5
+(VllmWorkerProcess pid=76764) INFO 10-21 20:06:23 pynccl.py:63] vLLM is using nccl==2.20.5
+(VllmWorkerProcess pid=76767) INFO 10-21 20:06:23 utils.py:1008] Found nccl from library libnccl.so.2
+(VllmWorkerProcess pid=76767) INFO 10-21 20:06:23 pynccl.py:63] vLLM is using nccl==2.20.5
+(VllmWorkerProcess pid=76765) INFO 10-21 20:06:23 utils.py:1008] Found nccl from library libnccl.so.2
+(VllmWorkerProcess pid=76766) INFO 10-21 20:06:23 utils.py:1008] Found nccl from library libnccl.so.2
+(VllmWorkerProcess pid=76765) INFO 10-21 20:06:23 pynccl.py:63] vLLM is using nccl==2.20.5
+(VllmWorkerProcess pid=76766) INFO 10-21 20:06:23 pynccl.py:63] vLLM is using nccl==2.20.5
+(VllmWorkerProcess pid=76763) INFO 10-21 20:06:23 utils.py:1008] Found nccl from library libnccl.so.2
+(VllmWorkerProcess pid=76763) INFO 10-21 20:06:23 pynccl.py:63] vLLM is using nccl==2.20.5
+n117-192-077:76638:76638 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
+n117-192-077:76638:76638 [0] NCCL INFO Bootstrap : Using eth0:10.117.192.77<0>
+n117-192-077:76638:76638 [0] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v8 symbol.
+n117-192-077:76638:76638 [0] NCCL INFO NET/Plugin: Loaded net plugin NCCL RDMA Plugin v7 (v7)
+n117-192-077:76638:76638 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v8 symbol.
+n117-192-077:76638:76638 [0] NCCL INFO NET/Plugin: Loaded coll plugin SHARP (v7)
+n117-192-077:76638:76638 [0] NCCL INFO cudaDriverVersion 12020
+NCCL version 2.20.5+cuda12.4
+n117-192-077:76763:76763 [3] NCCL INFO cudaDriverVersion 12020
+n117-192-077:76763:76763 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
+n117-192-077:76763:76763 [3] NCCL INFO Bootstrap : Using eth0:10.117.192.77<0>
+n117-192-077:76763:76763 [3] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v8 symbol.
+n117-192-077:76763:76763 [3] NCCL INFO NET/Plugin: Loaded net plugin NCCL RDMA Plugin v7 (v7)
+n117-192-077:76763:76763 [3] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v8 symbol.
+n117-192-077:76763:76763 [3] NCCL INFO NET/Plugin: Loaded coll plugin SHARP (v7)
+n117-192-077:76763:76763 [3] NCCL INFO Plugin Path : /usr/local/lib/libnccl-net.so
+n117-192-077:76763:76763 [3] NCCL INFO P2P plugin IBext_v7
+n117-192-077:76763:76763 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
+n117-192-077:76763:76763 [3] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB/SHARP [1]mlx5_1:1/IB/SHARP [2]mlx5_2:1/IB/SHARP [3]mlx5_3:1/IB/SHARP [4]mlx5_4:1/IB/SHARP [5]mlx5_5:1/IB/SHARP [6]mlx5_6:1/IB/SHARP [7]mlx5_7:1/IB/SHARP [RO]; OOB eth0:10.117.192.77<0>
+n117-192-077:76763:76763 [3] NCCL INFO Using non-device net plugin version 0
+n117-192-077:76763:76763 [3] NCCL INFO Using network IBext_v7
+n117-192-077:76763:76763 [3] NCCL INFO comm 0xc049370 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId 800000 commId 0xde285eafbeebd1fd - Init START
+n117-192-077:76763:76763 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/tiger/arnold/arnold_entrypoint/nccl_topo_files/azure_ndv5_topo.xml
+n117-192-077:76763:76763 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
+n117-192-077:76763:76763 [3] NCCL INFO Setting affinity for GPU 3 to ffffffff,ffff0000,00000000
+n117-192-077:76763:76763 [3] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0.
+n117-192-077:76763:76763 [3] NCCL INFO comm 0xc049370 rank 3 nRanks 8 nNodes 1 localRanks 8 localRank 3 MNNVL 0
+n117-192-077:76763:76763 [3] NCCL INFO NCCL_MAX_NCHANNELS set by environment to 12.
+n117-192-077:76763:76763 [3] NCCL INFO Trees [0] 4/-1/-1->3->2 [1] 4/-1/-1->3->2 [2] 4/-1/-1->3->2 [3] 4/-1/-1->3->2 [4] 4/-1/-1->3->2 [5] 4/-1/-1->3->2 [6] 4/-1/-1->3->2 [7] 4/-1/-1->3->2 [8] 4/-1/-1->3->2 [9] 4/-1/-1->3->2 [10] 4/-1/-1->3->2 [11] 4/-1/-1->3->2
+n117-192-077:76763:76763 [3] NCCL INFO P2P Chunksize set to 524288
+n117-192-077:76763:76763 [3] NCCL INFO Channel 00/0 : 3[3] -> 4[4] via P2P/IPC
+n117-192-077:76763:76763 [3] NCCL INFO Channel 01/0 : 3[3] -> 4[4] via P2P/IPC
+n117-192-077:76763:76763 [3] NCCL INFO Channel 02/0 : 3[3] -> 4[4] via P2P/IPC
+n117-192-077:76763:76763 [3] NCCL INFO Channel 03/0 : 3[3] -> 4[4] via P2P/IPC
+n117-192-077:76763:76763 [3] NCCL INFO Channel 04/0 : 3[3] -> 4[4] via P2P/IPC
+n117-192-077:76763:76763 [3] NCCL INFO Channel 05/0 : 3[3] -> 4[4] via P2P/IPC
+n117-192-077:76763:76763 [3] NCCL INFO Channel 06/0 : 3[3] -> 4[4] via P2P/IPC
+n117-192-077:76763:76763 [3] NCCL INFO Channel 07/0 : 3[3] -> 4[4] via P2P/IPC
+n117-192-077:76763:76763 [3] NCCL INFO Channel 08/0 : 3[3] -> 4[4] via P2P/IPC
+n117-192-077:76763:76763 [3] NCCL INFO Channel 09/0 : 3[3] -> 4[4] via P2P/IPC
+n117-192-077:76763:76763 [3] NCCL INFO Channel 10/0 : 3[3] -> 4[4] via P2P/IPC
+n117-192-077:76763:76763 [3] NCCL INFO Channel 11/0 : 3[3] -> 4[4] via P2P/IPC
+n117-192-077:76763:76763 [3] NCCL INFO Connected all rings
+n117-192-077:76763:76763 [3] NCCL INFO Channel 00/0 : 3[3] -> 2[2] via P2P/IPC
+n117-192-077:76763:76763 [3] NCCL INFO Channel 01/0 : 3[3] -> 2[2] via P2P/IPC
+n117-192-077:76763:76763 [3] NCCL INFO Channel 02/0 : 3[3] -> 2[2] via P2P/IPC
+n117-192-077:76763:76763 [3] NCCL INFO Channel 03/0 : 3[3] -> 2[2] via P2P/IPC
+n117-192-077:76763:76763 [3] NCCL INFO Channel 04/0 : 3[3] -> 2[2] via P2P/IPC
+n117-192-077:76763:76763 [3] NCCL INFO Channel 05/0 : 3[3] -> 2[2] via P2P/IPC
+n117-192-077:76763:76763 [3] NCCL INFO Channel 06/0 : 3[3] -> 2[2] via P2P/IPC
+n117-192-077:76763:76763 [3] NCCL INFO Channel 07/0 : 3[3] -> 2[2] via P2P/IPC
+n117-192-077:76763:76763 [3] NCCL INFO Channel 08/0 : 3[3] -> 2[2] via P2P/IPC
+n117-192-077:76763:76763 [3] NCCL INFO Channel 09/0 : 3[3] -> 2[2] via P2P/IPC
+n117-192-077:767n117-192-077:76767:76767 [7] NCCL INFO cudaDriverVersion 12020
+n117-192-077:76767:76767 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
+n117-192-077:76767:76767 [7] NCCL INFO Bootstrap : Using eth0:10.117.192.77<0>
+n117-192-077:76767:76767 [7] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v8 symbol.
+n117-192-077:76767:76767 [7] NCCL INFO NET/Plugin: Loaded net plugin NCCL RDMA Plugin v7 (v7)
+n117-192-077:76767:76767 [7] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v8 symbol.
+n117-192-077:76767:76767 [7] NCCL INFO NET/Plugin: Loaded coll plugin SHARP (v7)
+n117-192-077:76767:76767 [7] NCCL INFO Plugin Path : /usr/local/lib/libnccl-net.so
+n117-192-077:76767:76767 [7] NCCL INFO P2P plugin IBext_v7
+n117-192-077:76767:76767 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
+n117-192-077:76767:76767 [7] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB/SHARP [1]mlx5_1:1/IB/SHARP [2]mlx5_2:1/IB/SHARP [3]mlx5_3:1/IB/SHARP [4]mlx5_4:1/IB/SHARP [5]mlx5_5:1/IB/SHARP [6]mlx5_6:1/IB/SHARP [7]mlx5_7:1/IB/SHARP [RO]; OOB eth0:10.117.192.77<0>
+n117-192-077:76767:76767 [7] NCCL INFO Using non-device net plugin version 0
+n117-192-077:76767:76767 [7] NCCL INFO Using network IBext_v7
+n117-192-077:76767:76767 [7] NCCL INFO comm 0xc081c80 rank 7 nranks 8 cudaDev 7 nvmlDev 7 busId c00000 commId 0xde285eafbeebd1fd - Init START
+n117-192-077:76767:76767 [7] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/tiger/arnold/arnold_entrypoint/nccl_topo_files/azure_ndv5_topo.xml
+n117-192-077:76767:76767 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
+n117-192-077:76767:76767 [7] NCCL INFO Setting affinity for GPU 7 to ffff,ffffffff
+n117-192-077:76767:76767 [7] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0.
+n117-192-077:76767:76767 [7] NCCL INFO comm 0xc081c80 rank 7 nRanks 8 nNodes 1 localRanks 8 localRank 7 MNNVL 0
+n117-192-077:76767:76767 [7] NCCL INFO NCCL_MAX_NCHANNELS set by environment to 12.
+n117-192-077:76767:76767 [7] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] -1/-1/-1->7->6 [2] -1/-1/-1->7->6 [3] -1/-1/-1->7->6 [4] -1/-1/-1->7->6 [5] -1/-1/-1->7->6 [6] -1/-1/-1->7->6 [7] -1/-1/-1->7->6 [8] -1/-1/-1->7->6 [9] -1/-1/-1->7->6 [10] -1/-1/-1->7->6 [11] -1/-1/-1->7->6
+n117-192-077:76767:76767 [7] NCCL INFO P2P Chunksize set to 524288
+n117-192-077:76767:76767 [7] NCCL INFO Channel 00/0 : 7[7] -> 0[0] via P2P/IPC
+n117-192-077:76767:76767 [7] NCCL INFO Channel 01/0 : 7[7] -> 0[0] via P2P/IPC
+n117-192-077:76767:76767 [7] NCCL INFO Channel 02/0 : 7[7] -> 0[0] via P2P/IPC
+n117-192-077:76767:76767 [7] NCCL INFO Channel 03/0 : 7[7] -> 0[0] via P2P/IPC
+n117-192-077:76767:76767 [7] NCCL INFO Channel 04/0 : 7[7] -> 0[0] via P2P/IPC
+n117-192-077:76767:76767 [7] NCCL INFO Channel 05/0 : 7[7] -> 0[0] via P2P/IPC
+n117-192-077:76767:76767 [7] NCCL INFO Channel 06/0 : 7[7] -> 0[0] via P2P/IPC
+n117-192-077:76767:76767 [7] NCCL INFO Channel 07/0 : 7[7] -> 0[0] via P2P/IPC
+n117-192-077:76767:76767 [7] NCCL INFO Channel 08/0 : 7[7] -> 0[0] via P2P/IPC
+n117-192-077:76767:76767 [7] NCCL INFO Channel 09/0 : 7[7] -> 0[0] via P2P/IPC
+n117-192-077:76767:76767 [7] NCCL INFO Channel 10/0 : 7[7] -> 0[0] via P2P/IPC
+n117-192-077:76767:76767 [7] NCCL INFO Channel 11/0 : 7[7] -> 0[0] via P2P/IPC
+n117-192-077:76767:76767 [7] NCCL INFO Connected all rings
+n117-192-077:76767:76767 [7] NCCL INFO Channel 00/0 : 7[7] -> 6[6] via P2P/IPC
+n117-192-077:76767:76767 [7] NCCL INFO Channel 01/0 : 7[7] -> 6[6] via P2P/IPC
+n117-192-077:76767:76767 [7] NCCL INFO Channel 02/0 : 7[7] -> 6[6] via P2P/IPC
+n117-192-077:76767:76767 [7] NCCL INFO Channel 03/0 : 7[7] -> 6[6] via P2P/IPC
+n117-192-077:76767:76767 [7] NCCL INFO Channel 04/0 : 7[7] -> 6[6] via P2P/IPC
+n117-192-077:76767:76767 [7] NCCL INFO Channel 05/0 : 7[7] -> 6[6] via P2P/IPC
+n117-192-077:76767:76767 [7] NCCL INFO Channel 06/0 : 7[7] -> 6[6] via P2P/IPC
+n117-192-077:76767:76767 [7] NCCL INFO Channel 07/0 : 7[7] -> 6[6] via P2P/IPC
+n117-192-077:76767:76767 [7] NCCL INFO Channel 08/0 : 7[7] -> 6[6] via P2P/IPC
+n117-192-077:76767:76767 [7] NCCL INFO Channel 09/0 : 7[7] -> 6[6] via P2P/IPC
+n117-192-077:7676n117-192-077:76766:76766 [6] NCCL INFO cudaDriverVersion 12020
+n117-192-077:76766:76766 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
+n117-192-077:76766:76766 [6] NCCL INFO Bootstrap : Using eth0:10.117.192.77<0>
+n117-192-077:76766:76766 [6] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v8 symbol.
+n117-192-077:76766:76766 [6] NCCL INFO NET/Plugin: Loaded net plugin NCCL RDMA Plugin v7 (v7)
+n117-192-077:76766:76766 [6] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v8 symbol.
+n117-192-077:76766:76766 [6] NCCL INFO NET/Plugin: Loaded coll plugin SHARP (v7)
+n117-192-077:76766:76766 [6] NCCL INFO Plugin Path : /usr/local/lib/libnccl-net.so
+n117-192-077:76766:76766 [6] NCCL INFO P2P plugin IBext_v7
+n117-192-077:76766:76766 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
+n117-192-077:76766:76766 [6] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB/SHARP [1]mlx5_1:1/IB/SHARP [2]mlx5_2:1/IB/SHARP [3]mlx5_3:1/IB/SHARP [4]mlx5_4:1/IB/SHARP [5]mlx5_5:1/IB/SHARP [6]mlx5_6:1/IB/SHARP [7]mlx5_7:1/IB/SHARP [RO]; OOB eth0:10.117.192.77<0>
+n117-192-077:76766:76766 [6] NCCL INFO Using non-device net plugin version 0
+n117-192-077:76766:76766 [6] NCCL INFO Using network IBext_v7
+n117-192-077:76766:76766 [6] NCCL INFO comm 0xc081d20 rank 6 nranks 8 cudaDev 6 nvmlDev 6 busId b00000 commId 0xde285eafbeebd1fd - Init START
+n117-192-077:76766:76766 [6] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/tiger/arnold/arnold_entrypoint/nccl_topo_files/azure_ndv5_topo.xml
+n117-192-077:76766:76766 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
+n117-192-077:76766:76766 [6] NCCL INFO Setting affinity for GPU 6 to ffff,ffffffff
+n117-192-077:76766:76766 [6] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0.
+n117-192-077:76766:76766 [6] NCCL INFO comm 0xc081d20 rank 6 nRanks 8 nNodes 1 localRanks 8 localRank 6 MNNVL 0
+n117-192-077:76766:76766 [6] NCCL INFO NCCL_MAX_NCHANNELS set by environment to 12.
+n117-192-077:76766:76766 [6] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5 [2] 7/-1/-1->6->5 [3] 7/-1/-1->6->5 [4] 7/-1/-1->6->5 [5] 7/-1/-1->6->5 [6] 7/-1/-1->6->5 [7] 7/-1/-1->6->5 [8] 7/-1/-1->6->5 [9] 7/-1/-1->6->5 [10] 7/-1/-1->6->5 [11] 7/-1/-1->6->5
+n117-192-077:76766:76766 [6] NCCL INFO P2P Chunksize set to 524288
+n117-192-077:76766:76766 [6] NCCL INFO Channel 00/0 : 6[6] -> 7[7] via P2P/IPC
+n117-192-077:76766:76766 [6] NCCL INFO Channel 01/0 : 6[6] -> 7[7] via P2P/IPC
+n117-192-077:76766:76766 [6] NCCL INFO Channel 02/0 : 6[6] -> 7[7] via P2P/IPC
+n117-192-077:76766:76766 [6] NCCL INFO Channel 03/0 : 6[6] -> 7[7] via P2P/IPC
+n117-192-077:76766:76766 [6] NCCL INFO Channel 04/0 : 6[6] -> 7[7] via P2P/IPC
+n117-192-077:76766:76766 [6] NCCL INFO Channel 05/0 : 6[6] -> 7[7] via P2P/IPC
+n117-192-077:76766:76766 [6] NCCL INFO Channel 06/0 : 6[6] -> 7[7] via P2P/IPC
+n117-192-077:76766:76766 [6] NCCL INFO Channel 07/0 : 6[6] -> 7[7] via P2P/IPC
+n117-192-077:76766:76766 [6] NCCL INFO Channel 08/0 : 6[6] -> 7[7] via P2P/IPC
+n117-192-077:76766:76766 [6] NCCL INFO Channel 09/0 : 6[6] -> 7[7] via P2P/IPC
+n117-192-077:76766:76766 [6] NCCL INFO Channel 10/0 : 6[6] -> 7[7] via P2P/IPC
+n117-192-077:76766:76766 [6] NCCL INFO Channel 11/0 : 6[6] -> 7[7] via P2P/IPC
+n117-192-077:76766:76766 [6] NCCL INFO Connected all rings
+n117-192-077:76766:76766 [6] NCCL INFO Channel 00/0 : 6[6] -> 5[5] via P2P/IPC
+n117-192-077:76766:76766 [6] NCCL INFO Channel 01/0 : 6[6] -> 5[5] via P2P/IPC
+n117-192-077:76766:76766 [6] NCCL INFO Channel 02/0 : 6[6] -> 5[5] via P2P/IPC
+n117-192-077:76766:76766 [6] NCCL INFO Channel 03/0 : 6[6] -> 5[5] via P2P/IPC
+n117-192-077:76766:76766 [6] NCCL INFO Channel 04/0 : 6[6] -> 5[5] via P2P/IPC
+n117-192-077:76766:76766 [6] NCCL INFO Channel 05/0 : 6[6] -> 5[5] via P2P/IPC
+n117-192-077:76766:76766 [6] NCCL INFO Channel 06/0 : 6[6] -> 5[5] via P2P/IPC
+n117-192-077:76766:76766 [6] NCCL INFO Channel 07/0 : 6[6] -> 5[5] via P2P/IPC
+n117-192-077:76766:76766 [6] NCCL INFO Channel 08/0 : 6[6] -> 5[5] via P2P/IPC
+n117-192-077:76766:76766 [6] NCCL INFO Channel 09/0 : 6[6] -> 5[5] via P2P/IPC
+n117-192-077:76766:76766 [6] n117-192-077:76762:76762 [2] NCCL INFO cudaDriverVersion 12020
+n117-192-077:76762:76762 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
+n117-192-077:76762:76762 [2] NCCL INFO Bootstrap : Using eth0:10.117.192.77<0>
+n117-192-077:76762:76762 [2] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v8 symbol.
+n117-192-077:76762:76762 [2] NCCL INFO NET/Plugin: Loaded net plugin NCCL RDMA Plugin v7 (v7)
+n117-192-077:76762:76762 [2] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v8 symbol.
+n117-192-077:76762:76762 [2] NCCL INFO NET/Plugin: Loaded coll plugin SHARP (v7)
+n117-192-077:76762:76762 [2] NCCL INFO Plugin Path : /usr/local/lib/libnccl-net.so
+n117-192-077:76762:76762 [2] NCCL INFO P2P plugin IBext_v7
+n117-192-077:76762:76762 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
+n117-192-077:76762:76762 [2] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB/SHARP [1]mlx5_1:1/IB/SHARP [2]mlx5_2:1/IB/SHARP [3]mlx5_3:1/IB/SHARP [4]mlx5_4:1/IB/SHARP [5]mlx5_5:1/IB/SHARP [6]mlx5_6:1/IB/SHARP [7]mlx5_7:1/IB/SHARP [RO]; OOB eth0:10.117.192.77<0>
+n117-192-077:76762:76762 [2] NCCL INFO Using non-device net plugin version 0
+n117-192-077:76762:76762 [2] NCCL INFO Using network IBext_v7
+n117-192-077:76762:76762 [2] NCCL INFO comm 0xc086f40 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId 300000 commId 0xde285eafbeebd1fd - Init START
+n117-192-077:76762:76762 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/tiger/arnold/arnold_entrypoint/nccl_topo_files/azure_ndv5_topo.xml
+n117-192-077:76762:76762 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
+n117-192-077:76762:76762 [2] NCCL INFO Setting affinity for GPU 2 to ffffffff,ffff0000,00000000
+n117-192-077:76762:76762 [2] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0.
+n117-192-077:76762:76762 [2] NCCL INFO comm 0xc086f40 rank 2 nRanks 8 nNodes 1 localRanks 8 localRank 2 MNNVL 0
+n117-192-077:76762:76762 [2] NCCL INFO NCCL_MAX_NCHANNELS set by environment to 12.
+n117-192-077:76762:76762 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 3/-1/-1->2->1 [2] 3/-1/-1->2->1 [3] 3/-1/-1->2->1 [4] 3/-1/-1->2->1 [5] 3/-1/-1->2->1 [6] 3/-1/-1->2->1 [7] 3/-1/-1->2->1 [8] 3/-1/-1->2->1 [9] 3/-1/-1->2->1 [10] 3/-1/-1->2->1 [11] 3/-1/-1->2->1
+n117-192-077:76762:76762 [2] NCCL INFO P2P Chunksize set to 524288
+n117-192-077:76762:76762 [2] NCCL INFO Channel 00/0 : 2[2] -> 3[3] via P2P/IPC
+n117-192-077:76762:76762 [2] NCCL INFO Channel 01/0 : 2[2] -> 3[3] via P2P/IPC
+n117-192-077:76762:76762 [2] NCCL INFO Channel 02/0 : 2[2] -> 3[3] via P2P/IPC
+n117-192-077:76762:76762 [2] NCCL INFO Channel 03/0 : 2[2] -> 3[3] via P2P/IPC
+n117-192-077:76762:76762 [2] NCCL INFO Channel 04/0 : 2[2] -> 3[3] via P2P/IPC
+n117-192-077:76762:76762 [2] NCCL INFO Channel 05/0 : 2[2] -> 3[3] via P2P/IPC
+n117-192-077:76762:76762 [2] NCCL INFO Channel 06/0 : 2[2] -> 3[3] via P2P/IPC
+n117-192-077:76762:76762 [2] NCCL INFO Channel 07/0 : 2[2] -> 3[3] via P2P/IPC
+n117-192-077:76762:76762 [2] NCCL INFO Channel 08/0 : 2[2] -> 3[3] via P2P/IPC
+n117-192-077:76762:76762 [2] NCCL INFO Channel 09/0 : 2[2] -> 3[3] via P2P/IPC
+n117-192-077:76762:76762 [2] NCCL INFO Channel 10/0 : 2[2] -> 3[3] via P2P/IPC
+n117-192-077:76762:76762 [2] NCCL INFO Channel 11/0 : 2[2] -> 3[3] via P2P/IPC
+n117-192-077:76762:76762 [2] NCCL INFO Connected all rings
+n117-192-077:76762:76762 [2] NCCL INFO Channel 00/0 : 2[2] -> 1[1] via P2P/IPC
+n117-192-077:76762:76762 [2] NCCL INFO Channel 01/0 : 2[2] -> 1[1] via P2P/IPC
+n117-192-077:76762:76762 [2] NCCL INFO Channel 02/0 : 2[2] -> 1[1] via P2P/IPC
+n117-192-077:76762:76762 [2] NCCL INFO Channel 03/0 : 2[2] -> 1[1] via P2P/IPC
+n117-192-077:76762:76762 [2] NCCL INFO Channel 04/0 : 2[2] -> 1[1] via P2P/IPC
+n117-192-077:76762:76762 [2] NCCL INFO Channel 05/0 : 2[2] -> 1[1] via P2P/IPC
+n117-192-077:76762:76762 [2] NCCL INFO Channel 06/0 : 2[2] -> 1[1] via P2P/IPC
+n117-192-077:76762:76762 [2] NCCL INFO Channel 07/0 : 2[2] -> 1[1] via P2P/IPC
+n117-192-077:76762:76762 [2] NCCL INFO Channel 08/0 : 2[2] -> 1[1] via P2P/IPC
+n117-192-077:76762:76762 [2] NCCL INFO Channel 09/0 : 2[2] -> 1[1] via P2P/IPC
+n117-192-077:767n117-192-077:76764:76764 [4] NCCL INFO cudaDriverVersion 12020
+n117-192-077:76764:76764 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
+n117-192-077:76764:76764 [4] NCCL INFO Bootstrap : Using eth0:10.117.192.77<0>
+n117-192-077:76764:76764 [4] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v8 symbol.
+n117-192-077:76764:76764 [4] NCCL INFO NET/Plugin: Loaded net plugin NCCL RDMA Plugin v7 (v7)
+n117-192-077:76764:76764 [4] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v8 symbol.
+n117-192-077:76764:76764 [4] NCCL INFO NET/Plugin: Loaded coll plugin SHARP (v7)
+n117-192-077:76764:76764 [4] NCCL INFO Plugin Path : /usr/local/lib/libnccl-net.so
+n117-192-077:76764:76764 [4] NCCL INFO P2P plugin IBext_v7
+n117-192-077:76764:76764 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
+n117-192-077:76764:76764 [4] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB/SHARP [1]mlx5_1:1/IB/SHARP [2]mlx5_2:1/IB/SHARP [3]mlx5_3:1/IB/SHARP [4]mlx5_4:1/IB/SHARP [5]mlx5_5:1/IB/SHARP [6]mlx5_6:1/IB/SHARP [7]mlx5_7:1/IB/SHARP [RO]; OOB eth0:10.117.192.77<0>
+n117-192-077:76764:76764 [4] NCCL INFO Using non-device net plugin version 0
+n117-192-077:76764:76764 [4] NCCL INFO Using network IBext_v7
+n117-192-077:76764:76764 [4] NCCL INFO comm 0xc086e00 rank 4 nranks 8 cudaDev 4 nvmlDev 4 busId 900000 commId 0xde285eafbeebd1fd - Init START
+n117-192-077:76764:76764 [4] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/tiger/arnold/arnold_entrypoint/nccl_topo_files/azure_ndv5_topo.xml
+n117-192-077:76764:76764 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
+n117-192-077:76764:76764 [4] NCCL INFO Setting affinity for GPU 4 to ffff,ffffffff
+n117-192-077:76764:76764 [4] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0.
+n117-192-077:76764:76764 [4] NCCL INFO comm 0xc086e00 rank 4 nRanks 8 nNodes 1 localRanks 8 localRank 4 MNNVL 0
+n117-192-077:76764:76764 [4] NCCL INFO NCCL_MAX_NCHANNELS set by environment to 12.
+n117-192-077:76764:76764 [4] NCCL INFO Trees [0] 5/-1/-1->4->3 [1] 5/-1/-1->4->3 [2] 5/-1/-1->4->3 [3] 5/-1/-1->4->3 [4] 5/-1/-1->4->3 [5] 5/-1/-1->4->3 [6] 5/-1/-1->4->3 [7] 5/-1/-1->4->3 [8] 5/-1/-1->4->3 [9] 5/-1/-1->4->3 [10] 5/-1/-1->4->3 [11] 5/-1/-1->4->3
+n117-192-077:76764:76764 [4] NCCL INFO P2P Chunksize set to 524288
+n117-192-077:76764:76764 [4] NCCL INFO Channel 00/0 : 4[4] -> 5[5] via P2P/IPC
+n117-192-077:76764:76764 [4] NCCL INFO Channel 01/0 : 4[4] -> 5[5] via P2P/IPC
+n117-192-077:76764:76764 [4] NCCL INFO Channel 02/0 : 4[4] -> 5[5] via P2P/IPC
+n117-192-077:76764:76764 [4] NCCL INFO Channel 03/0 : 4[4] -> 5[5] via P2P/IPC
+n117-192-077:76764:76764 [4] NCCL INFO Channel 04/0 : 4[4] -> 5[5] via P2P/IPC
+n117-192-077:76764:76764 [4] NCCL INFO Channel 05/0 : 4[4] -> 5[5] via P2P/IPC
+n117-192-077:76764:76764 [4] NCCL INFO Channel 06/0 : 4[4] -> 5[5] via P2P/IPC
+n117-192-077:76764:76764 [4] NCCL INFO Channel 07/0 : 4[4] -> 5[5] via P2P/IPC
+n117-192-077:76764:76764 [4] NCCL INFO Channel 08/0 : 4[4] -> 5[5] via P2P/IPC
+n117-192-077:76764:76764 [4] NCCL INFO Channel 09/0 : 4[4] -> 5[5] via P2P/IPC
+n117-192-077:76764:76764 [4] NCCL INFO Channel 10/0 : 4[4] -> 5[5] via P2P/IPC
+n117-192-077:76764:76764 [4] NCCL INFO Channel 11/0 : 4[4] -> 5[5] via P2P/IPC
+n117-192-077:76764:76764 [4] NCCL INFO Connected all rings
+n117-192-077:76764:76764 [4] NCCL INFO Channel 00/0 : 4[4] -> 3[3] via P2P/IPC
+n117-192-077:76764:76764 [4] NCCL INFO Channel 01/0 : 4[4] -> 3[3] via P2P/IPC
+n117-192-077:76764:76764 [4] NCCL INFO Channel 02/0 : 4[4] -> 3[3] via P2P/IPC
+n117-192-077:76764:76764 [4] NCCL INFO Channel 03/0 : 4[4] -> 3[3] via P2P/IPC
+n117-192-077:76764:76764 [4] NCCL INFO Channel 04/0 : 4[4] -> 3[3] via P2P/IPC
+n117-192-077:76764:76764 [4] NCCL INFO Channel 05/0 : 4[4] -> 3[3] via P2P/IPC
+n117-192-077:76764:76764 [4] NCCL INFO Channel 06/0 : 4[4] -> 3[3] via P2P/IPC
+n117-192-077:76764:76764 [4] NCCL INFO Channel 07/0 : 4[4] -> 3[3] via P2P/IPC
+n117-192-077:76764:76764 [4] NCCL INFO Channel 08/0 : 4[4] -> 3[3] via P2P/IPC
+n117-192-077:76764:76764 [4] NCCL INFO Channel 09/0 : 4[4] -> 3[3] via P2P/IPC
+n117-192-077:76764:76764 [4] n117-192-077:76765:76765 [5] NCCL INFO cudaDriverVersion 12020
+n117-192-077:76765:76765 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
+n117-192-077:76765:76765 [5] NCCL INFO Bootstrap : Using eth0:10.117.192.77<0>
+n117-192-077:76765:76765 [5] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v8 symbol.
+n117-192-077:76765:76765 [5] NCCL INFO NET/Plugin: Loaded net plugin NCCL RDMA Plugin v7 (v7)
+n117-192-077:76765:76765 [5] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v8 symbol.
+n117-192-077:76765:76765 [5] NCCL INFO NET/Plugin: Loaded coll plugin SHARP (v7)
+n117-192-077:76765:76765 [5] NCCL INFO Plugin Path : /usr/local/lib/libnccl-net.so
+n117-192-077:76765:76765 [5] NCCL INFO P2P plugin IBext_v7
+n117-192-077:76765:76765 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
+n117-192-077:76765:76765 [5] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB/SHARP [1]mlx5_1:1/IB/SHARP [2]mlx5_2:1/IB/SHARP [3]mlx5_3:1/IB/SHARP [4]mlx5_4:1/IB/SHARP [5]mlx5_5:1/IB/SHARP [6]mlx5_6:1/IB/SHARP [7]mlx5_7:1/IB/SHARP [RO]; OOB eth0:10.117.192.77<0>
+n117-192-077:76765:76765 [5] NCCL INFO Using non-device net plugin version 0
+n117-192-077:76765:76765 [5] NCCL INFO Using network IBext_v7
+n117-192-077:76765:76765 [5] NCCL INFO comm 0xc082d50 rank 5 nranks 8 cudaDev 5 nvmlDev 5 busId a00000 commId 0xde285eafbeebd1fd - Init START
+n117-192-077:76765:76765 [5] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/tiger/arnold/arnold_entrypoint/nccl_topo_files/azure_ndv5_topo.xml
+n117-192-077:76765:76765 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
+n117-192-077:76765:76765 [5] NCCL INFO Setting affinity for GPU 5 to ffff,ffffffff
+n117-192-077:76765:76765 [5] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0.
+n117-192-077:76765:76765 [5] NCCL INFO comm 0xc082d50 rank 5 nRanks 8 nNodes 1 localRanks 8 localRank 5 MNNVL 0
+n117-192-077:76765:76765 [5] NCCL INFO NCCL_MAX_NCHANNELS set by environment to 12.
+n117-192-077:76765:76765 [5] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->4 [2] 6/-1/-1->5->4 [3] 6/-1/-1->5->4 [4] 6/-1/-1->5->4 [5] 6/-1/-1->5->4 [6] 6/-1/-1->5->4 [7] 6/-1/-1->5->4 [8] 6/-1/-1->5->4 [9] 6/-1/-1->5->4 [10] 6/-1/-1->5->4 [11] 6/-1/-1->5->4
+n117-192-077:76765:76765 [5] NCCL INFO P2P Chunksize set to 524288
+n117-192-077:76765:76765 [5] NCCL INFO Channel 00/0 : 5[5] -> 6[6] via P2P/IPC
+n117-192-077:76765:76765 [5] NCCL INFO Channel 01/0 : 5[5] -> 6[6] via P2P/IPC
+n117-192-077:76765:76765 [5] NCCL INFO Channel 02/0 : 5[5] -> 6[6] via P2P/IPC
+n117-192-077:76765:76765 [5] NCCL INFO Channel 03/0 : 5[5] -> 6[6] via P2P/IPC
+n117-192-077:76765:76765 [5] NCCL INFO Channel 04/0 : 5[5] -> 6[6] via P2P/IPC
+n117-192-077:76765:76765 [5] NCCL INFO Channel 05/0 : 5[5] -> 6[6] via P2P/IPC
+n117-192-077:76765:76765 [5] NCCL INFO Channel 06/0 : 5[5] -> 6[6] via P2P/IPC
+n117-192-077:76765:76765 [5] NCCL INFO Channel 07/0 : 5[5] -> 6[6] via P2P/IPC
+n117-192-077:76765:76765 [5] NCCL INFO Channel 08/0 : 5[5] -> 6[6] via P2P/IPC
+n117-192-077:76765:76765 [5] NCCL INFO Channel 09/0 : 5[5] -> 6[6] via P2P/IPC
+n117-192-077:76765:76765 [5] NCCL INFO Channel 10/0 : 5[5] -> 6[6] via P2P/IPC
+n117-192-077:76765:76765 [5] NCCL INFO Channel 11/0 : 5[5] -> 6[6] via P2P/IPC
+n117-192-077:76765:76765 [5] NCCL INFO Connected all rings
+n117-192-077:76765:76765 [5] NCCL INFO Channel 00/0 : 5[5] -> 4[4] via P2P/IPC
+n117-192-077:76765:76765 [5] NCCL INFO Channel 01/0 : 5[5] -> 4[4] via P2P/IPC
+n117-192-077:76765:76765 [5] NCCL INFO Channel 02/0 : 5[5] -> 4[4] via P2P/IPC
+n117-192-077:76765:76765 [5] NCCL INFO Channel 03/0 : 5[5] -> 4[4] via P2P/IPC
+n117-192-077:76765:76765 [5] NCCL INFO Channel 04/0 : 5[5] -> 4[4] via P2P/IPC
+n117-192-077:76765:76765 [5] NCCL INFO Channel 05/0 : 5[5] -> 4[4] via P2P/IPC
+n117-192-077:76765:76765 [5] NCCL INFO Channel 06/0 : 5[5] -> 4[4] via P2P/IPC
+n117-192-077:76765:76765 [5] NCCL INFO Channel 07/0 : 5[5] -> 4[4] via P2P/IPC
+n117-192-077:76765:76765 [5] NCCL INFO Channel 08/0 : 5[5] -> 4[4] via P2P/IPC
+n117-192-077:76765:76765 [5] NCCL INFO Channel 09/0 : 5[5] -> 4[4] via P2P/IPC
+n117-192-077:76765:76765 [5] n117-192-077:76761:76761 [1] NCCL INFO cudaDriverVersion 12020
+n117-192-077:76761:76761 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
+n117-192-077:76761:76761 [1] NCCL INFO Bootstrap : Using eth0:10.117.192.77<0>
+n117-192-077:76761:76761 [1] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v8 symbol.
+n117-192-077:76761:76761 [1] NCCL INFO NET/Plugin: Loaded net plugin NCCL RDMA Plugin v7 (v7)
+n117-192-077:76761:76761 [1] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v8 symbol.
+n117-192-077:76761:76761 [1] NCCL INFO NET/Plugin: Loaded coll plugin SHARP (v7)
+n117-192-077:76761:76761 [1] NCCL INFO Plugin Path : /usr/local/lib/libnccl-net.so
+n117-192-077:76761:76761 [1] NCCL INFO P2P plugin IBext_v7
+n117-192-077:76761:76761 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
+n117-192-077:76761:76761 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB/SHARP [1]mlx5_1:1/IB/SHARP [2]mlx5_2:1/IB/SHARP [3]mlx5_3:1/IB/SHARP [4]mlx5_4:1/IB/SHARP [5]mlx5_5:1/IB/SHARP [6]mlx5_6:1/IB/SHARP [7]mlx5_7:1/IB/SHARP [RO]; OOB eth0:10.117.192.77<0>
+n117-192-077:76761:76761 [1] NCCL INFO Using non-device net plugin version 0
+n117-192-077:76761:76761 [1] NCCL INFO Using network IBext_v7
+n117-192-077:76761:76761 [1] NCCL INFO comm 0xc086fc0 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 200000 commId 0xde285eafbeebd1fd - Init START
+n117-192-077:76761:76761 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/tiger/arnold/arnold_entrypoint/nccl_topo_files/azure_ndv5_topo.xml
+n117-192-077:76761:76761 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
+n117-192-077:76761:76761 [1] NCCL INFO Setting affinity for GPU 1 to ffffffff,ffff0000,00000000
+n117-192-077:76761:76761 [1] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0.
+n117-192-077:76761:76761 [1] NCCL INFO comm 0xc086fc0 rank 1 nRanks 8 nNodes 1 localRanks 8 localRank 1 MNNVL 0
+n117-192-077:76761:76761 [1] NCCL INFO NCCL_MAX_NCHANNELS set by environment to 12.
+n117-192-077:76761:76761 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/-1/-1->1->0 [2] 2/-1/-1->1->0 [3] 2/-1/-1->1->0 [4] 2/-1/-1->1->0 [5] 2/-1/-1->1->0 [6] 2/-1/-1->1->0 [7] 2/-1/-1->1->0 [8] 2/-1/-1->1->0 [9] 2/-1/-1->1->0 [10] 2/-1/-1->1->0 [11] 2/-1/-1->1->0
+n117-192-077:76761:76761 [1] NCCL INFO P2P Chunksize set to 524288
+n117-192-077:76761:76761 [1] NCCL INFO Channel 00/0 : 1[1] -> 2[2] via P2P/IPC
+n117-192-077:76761:76761 [1] NCCL INFO Channel 01/0 : 1[1] -> 2[2] via P2P/IPC
+n117-192-077:76761:76761 [1] NCCL INFO Channel 02/0 : 1[1] -> 2[2] via P2P/IPC
+n117-192-077:76761:76761 [1] NCCL INFO Channel 03/0 : 1[1] -> 2[2] via P2P/IPC
+n117-192-077:76761:76761 [1] NCCL INFO Channel 04/0 : 1[1] -> 2[2] via P2P/IPC
+n117-192-077:76761:76761 [1] NCCL INFO Channel 05/0 : 1[1] -> 2[2] via P2P/IPC
+n117-192-077:76761:76761 [1] NCCL INFO Channel 06/0 : 1[1] -> 2[2] via P2P/IPC
+n117-192-077:76761:76761 [1] NCCL INFO Channel 07/0 : 1[1] -> 2[2] via P2P/IPC
+n117-192-077:76761:76761 [1] NCCL INFO Channel 08/0 : 1[1] -> 2[2] via P2P/IPC
+n117-192-077:76761:76761 [1] NCCL INFO Channel 09/0 : 1[1] -> 2[2] via P2P/IPC
+n117-192-077:76761:76761 [1] NCCL INFO Channel 10/0 : 1[1] -> 2[2] via P2P/IPC
+n117-192-077:76761:76761 [1] NCCL INFO Channel 11/0 : 1[1] -> 2[2] via P2P/IPC
+n117-192-077:76761:76761 [1] NCCL INFO Connected all rings
+n117-192-077:76761:76761 [1] NCCL INFO Channel 00/0 : 1[1] -> 0[0] via P2P/IPC
+n117-192-077:76761:76761 [1] NCCL INFO Channel 01/0 : 1[1] -> 0[0] via P2P/IPC
+n117-192-077:76761:76761 [1] NCCL INFO Channel 02/0 : 1[1] -> 0[0] via P2P/IPC
+n117-192-077:76761:76761 [1] NCCL INFO Channel 03/0 : 1[1] -> 0[0] via P2P/IPC
+n117-192-077:76761:76761 [1] NCCL INFO Channel 04/0 : 1[1] -> 0[0] via P2P/IPC
+n117-192-077:76761:76761 [1] NCCL INFO Channel 05/0 : 1[1] -> 0[0] via P2P/IPC
+n117-192-077:76761:76761 [1] NCCL INFO Channel 06/0 : 1[1] -> 0[0] via P2P/IPC
+n117-192-077:76761:76761 [1] NCCL INFO Channel 07/0 : 1[1] -> 0[0] via P2P/IPC
+n117-192-077:76761:76761 [1] NCCL INFO Channel 08/0 : 1[1] -> 0[0] via P2P/IPC
+n117-192-077:76761:76761 [1] NCCL INFO Channel 09/0 : 1[1] -> 0[0] via P2P/IPC
+n117-192-077:767n117-192-077:76638:76638 [0] NCCL INFO Plugin Path : /usr/local/lib/libnccl-net.so
+n117-192-077:76638:76638 [0] NCCL INFO P2P plugin IBext_v7
+n117-192-077:76638:76638 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
+n117-192-077:76638:76638 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB/SHARP [1]mlx5_1:1/IB/SHARP [2]mlx5_2:1/IB/SHARP [3]mlx5_3:1/IB/SHARP [4]mlx5_4:1/IB/SHARP [5]mlx5_5:1/IB/SHARP [6]mlx5_6:1/IB/SHARP [7]mlx5_7:1/IB/SHARP [RO]; OOB eth0:10.117.192.77<0>
+n117-192-077:76638:76638 [0] NCCL INFO Using non-device net plugin version 0
+n117-192-077:76638:76638 [0] NCCL INFO Using network IBext_v7
+n117-192-077:76638:76638 [0] NCCL INFO comm 0xc083260 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 100000 commId 0xde285eafbeebd1fd - Init START
+n117-192-077:76638:76638 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/tiger/arnold/arnold_entrypoint/nccl_topo_files/azure_ndv5_topo.xml
+n117-192-077:76638:76638 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
+n117-192-077:76638:76638 [0] NCCL INFO Setting affinity for GPU 0 to ffffffff,ffff0000,00000000
+n117-192-077:76638:76638 [0] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0.
+n117-192-077:76638:76638 [0] NCCL INFO comm 0xc083260 rank 0 nRanks 8 nNodes 1 localRanks 8 localRank 0 MNNVL 0
+n117-192-077:76638:76638 [0] NCCL INFO NCCL_MAX_NCHANNELS set by environment to 12.
+n117-192-077:76638:76638 [0] NCCL INFO Channel 00/12 :    0   1   2   3   4   5   6   7
+n117-192-077:76638:76638 [0] NCCL INFO Channel 01/12 :    0   1   2   3   4   5   6   7
+n117-192-077:76638:76638 [0] NCCL INFO Channel 02/12 :    0   1   2   3   4   5   6   7
+n117-192-077:76638:76638 [0] NCCL INFO Channel 03/12 :    0   1   2   3   4   5   6   7
+n117-192-077:76638:76638 [0] NCCL INFO Channel 04/12 :    0   1   2   3   4   5   6   7
+n117-192-077:76638:76638 [0] NCCL INFO Channel 05/12 :    0   1   2   3   4   5   6   7
+n117-192-077:76638:76638 [0] NCCL INFO Channel 06/12 :    0   1   2   3   4   5   6   7
+n117-192-077:76638:76638 [0] NCCL INFO Channel 07/12 :    0   1   2   3   4   5   6   7
+n117-192-077:76638:76638 [0] NCCL INFO Channel 08/12 :    0   1   2   3   4   5   6   7
+n117-192-077:76638:76638 [0] NCCL INFO Channel 09/12 :    0   1   2   3   4   5   6   7
+n117-192-077:76638:76638 [0] NCCL INFO Channel 10/12 :    0   1   2   3   4   5   6   7
+n117-192-077:76638:76638 [0] NCCL INFO Channel 11/12 :    0   1   2   3   4   5   6   7
+n117-192-077:76638:76638 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 1/-1/-1->0->-1 [7] 1/-1/-1->0->-1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 1/-1/-1->0->-1
+n117-192-077:76638:76638 [0] NCCL INFO P2P Chunksize set to 524288
+n117-192-077:76638:76638 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/IPC
+n117-192-077:76638:76638 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[1] via P2P/IPC
+n117-192-077:76638:76638 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[1] via P2P/IPC
+n117-192-077:76638:76638 [0] NCCL INFO Channel 03/0 : 0[0] -> 1[1] via P2P/IPC
+n117-192-077:76638:76638 [0] NCCL INFO Channel 04/0 : 0[0] -> 1[1] via P2P/IPC
+n117-192-077:76638:76638 [0] NCCL INFO Channel 05/0 : 0[0] -> 1[1] via P2P/IPC
+n117-192-077:76638:76638 [0] NCCL INFO Channel 06/0 : 0[0] -> 1[1] via P2P/IPC
+n117-192-077:76638:76638 [0] NCCL INFO Channel 07/0 : 0[0] -> 1[1] via P2P/IPC
+n117-192-077:76638:76638 [0] NCCL INFO Channel 08/0 : 0[0] -> 1[1] via P2P/IPC
+n117-192-077:76638:76638 [0] NCCL INFO Channel 09/0 : 0[0] -> 1[1] via P2P/IPC
+n117-192-077:76638:76638 [0] NCCL INFO Channel 10/0 : 0[0] -> 1[1] via P2P/IPC
+n117-192-077:76638:76638 [0] NCCL INFO Channel 11/0 : 0[0] -> 1[1] via P2P/IPC
+n117-192-077:76638:76638 [0] NCCL INFO Connected all rings
+n117-192-077:76638:76638 [0] NCCL INFO Connected all trees
+n117-192-077:76638:76638 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
+n117-192-077:76638:76638 [0] NCCL INFO 12 coll channels, 0 collnet channels, 0 nvls channels, 16 p2p channels, 16 p2p channels per peer
+n117-192-077:76638:76638 [0] NCCL INFO comm 0xc083260 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 100000 commId 0xde285eafbeebd1fd - Init COMPLETE
+INFO 10-21 20:06:31 custom_all_reduce_utils.py:204] generating GPU P2P access cache in /home/tiger/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
+(VllmWorkerProcess pid=76761) INFO 10-21 20:10:29 custom_all_reduce_utils.py:242] reading GPU P2P access cache from /home/tiger/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
+INFO 10-21 20:10:29 custom_all_reduce_utils.py:242] reading GPU P2P access cache from /home/tiger/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
+(VllmWorkerProcess pid=76763) INFO 10-21 20:10:29 custom_all_reduce_utils.py:242] reading GPU P2P access cache from /home/tiger/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
+(VllmWorkerProcess pid=76766) INFO 10-21 20:10:29 custom_all_reduce_utils.py:242] reading GPU P2P access cache from /home/tiger/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
+(VllmWorkerProcess pid=76765) INFO 10-21 20:10:29 custom_all_reduce_utils.py:242] reading GPU P2P access cache from /home/tiger/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
+(VllmWorkerProcess pid=76764) INFO 10-21 20:10:29 custom_all_reduce_utils.py:242] reading GPU P2P access cache from /home/tiger/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
+(VllmWorkerProcess pid=76767) INFO 10-21 20:10:29 custom_all_reduce_utils.py:242] reading GPU P2P access cache from /home/tiger/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
+(VllmWorkerProcess pid=76762) INFO 10-21 20:10:29 custom_all_reduce_utils.py:242] reading GPU P2P access cache from /home/tiger/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
+(VllmWorkerProcess pid=76761) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231] Exception in worker VllmWorkerProcess while processing method init_device: Tensors allocated with expandable_segments:True cannot be shared between processes. Consider using expandable_segments:False in data loading workers via torch.cuda.memory._set_allocator_settings('expandable_segments:False'), Traceback (most recent call last):
+(VllmWorkerProcess pid=76761) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/executor/multiproc_worker_utils.py", line 224, in _run_worker_process
+(VllmWorkerProcess pid=76761) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     output = executor(*args, **kwargs)
+(VllmWorkerProcess pid=76761) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/worker/worker.py", line 176, in init_device
+(VllmWorkerProcess pid=76761) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     init_worker_distributed_environment(self.parallel_config, self.rank,
+(VllmWorkerProcess pid=76761) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/worker/worker.py", line 456, in init_worker_distributed_environment
+(VllmWorkerProcess pid=76761) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     ensure_model_parallel_initialized(parallel_config.tensor_parallel_size,
+(VllmWorkerProcess pid=76761) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 1051, in ensure_model_parallel_initialized
+(VllmWorkerProcess pid=76761) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     initialize_model_parallel(tensor_model_parallel_size,
+(VllmWorkerProcess pid=76761) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 1015, in initialize_model_parallel
+(VllmWorkerProcess pid=76761) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     _TP = init_model_parallel_group(group_ranks,
+(VllmWorkerProcess pid=76761) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 856, in init_model_parallel_group
+(VllmWorkerProcess pid=76761) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     return GroupCoordinator(
+(VllmWorkerProcess pid=76761) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 222, in __init__
+(VllmWorkerProcess pid=76761) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     self.ca_comm = CustomAllreduce(
+(VllmWorkerProcess pid=76761) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 171, in __init__
+(VllmWorkerProcess pid=76761) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     handles, offsets = self._get_ipc_meta(self.meta)
+(VllmWorkerProcess pid=76761) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 193, in _get_ipc_meta
+(VllmWorkerProcess pid=76761) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     data = inp.untyped_storage()._share_cuda_()
+(VllmWorkerProcess pid=76761) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231] RuntimeError: Tensors allocated with expandable_segments:True cannot be shared between processes. Consider using expandable_segments:False in data loading workers via torch.cuda.memory._set_allocator_settings('expandable_segments:False')
+(VllmWorkerProcess pid=76761) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231] 
+(VllmWorkerProcess pid=76763) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231] Exception in worker VllmWorkerProcess while processing method init_device: Tensors allocated with expandable_segments:True cannot be shared between processes. Consider using expandable_segments:False in data loading workers via torch.cuda.memory._set_allocator_settings('expandable_segments:False'), Traceback (most recent call last):
+(VllmWorkerProcess pid=76763) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/executor/multiproc_worker_utils.py", line 224, in _run_worker_process
+(VllmWorkerProcess pid=76763) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     output = executor(*args, **kwargs)
+(VllmWorkerProcess pid=76763) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/worker/worker.py", line 176, in init_device
+(VllmWorkerProcess pid=76763) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     init_worker_distributed_environment(self.parallel_config, self.rank,
+(VllmWorkerProcess pid=76763) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/worker/worker.py", line 456, in init_worker_distributed_environment
+(VllmWorkerProcess pid=76763) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     ensure_model_parallel_initialized(parallel_config.tensor_parallel_size,
+(VllmWorkerProcess pid=76763) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 1051, in ensure_model_parallel_initialized
+(VllmWorkerProcess pid=76763) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     initialize_model_parallel(tensor_model_parallel_size,
+(VllmWorkerProcess pid=76763) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 1015, in initialize_model_parallel
+(VllmWorkerProcess pid=76763) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     _TP = init_model_parallel_group(group_ranks,
+(VllmWorkerProcess pid=76763) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 856, in init_model_parallel_group
+(VllmWorkerProcess pid=76763) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     return GroupCoordinator(
+(VllmWorkerProcess pid=76763) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 222, in __init__
+(VllmWorkerProcess pid=76763) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     self.ca_comm = CustomAllreduce(
+(VllmWorkerProcess pid=76763) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 171, in __init__
+(VllmWorkerProcess pid=76763) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     handles, offsets = self._get_ipc_meta(self.meta)
+(VllmWorkerProcess pid=76763) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 193, in _get_ipc_meta
+(VllmWorkerProcess pid=76763) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     data = inp.untyped_storage()._share_cuda_()
+(VllmWorkerProcess pid=76763) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231] RuntimeError: Tensors allocated with expandable_segments:True cannot be shared between processes. Consider using expandable_segments:False in data loading workers via torch.cuda.memory._set_allocator_settings('expandable_segments:False')
+(VllmWorkerProcess pid=76763) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231] 
+(VllmWorkerProcess pid=76766) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231] Exception in worker VllmWorkerProcess while processing method init_device: Tensors allocated with expandable_segments:True cannot be shared between processes. Consider using expandable_segments:False in data loading workers via torch.cuda.memory._set_allocator_settings('expandable_segments:False'), Traceback (most recent call last):
+(VllmWorkerProcess pid=76766) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/executor/multiproc_worker_utils.py", line 224, in _run_worker_process
+(VllmWorkerProcess pid=76766) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     output = executor(*args, **kwargs)
+(VllmWorkerProcess pid=76766) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/worker/worker.py", line 176, in init_device
+(VllmWorkerProcess pid=76766) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     init_worker_distributed_environment(self.parallel_config, self.rank,
+(VllmWorkerProcess pid=76766) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/worker/worker.py", line 456, in init_worker_distributed_environment
+(VllmWorkerProcess pid=76766) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     ensure_model_parallel_initialized(parallel_config.tensor_parallel_size,
+(VllmWorkerProcess pid=76766) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 1051, in ensure_model_parallel_initialized
+(VllmWorkerProcess pid=76766) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     initialize_model_parallel(tensor_model_parallel_size,
+(VllmWorkerProcess pid=76766) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 1015, in initialize_model_parallel
+(VllmWorkerProcess pid=76766) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     _TP = init_model_parallel_group(group_ranks,
+(VllmWorkerProcess pid=76766) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 856, in init_model_parallel_group
+(VllmWorkerProcess pid=76766) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     return GroupCoordinator(
+(VllmWorkerProcess pid=76766) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 222, in __init__
+(VllmWorkerProcess pid=76766) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     self.ca_comm = CustomAllreduce(
+(VllmWorkerProcess pid=76766) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 171, in __init__
+(VllmWorkerProcess pid=76766) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     handles, offsets = self._get_ipc_meta(self.meta)
+(VllmWorkerProcess pid=76766) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 193, in _get_ipc_meta
+(VllmWorkerProcess pid=76766) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     data = inp.untyped_storage()._share_cuda_()
+(VllmWorkerProcess pid=76766) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231] RuntimeError: Tensors allocated with expandable_segments:True cannot be shared between processes. Consider using expandable_segments:False in data loading workers via torch.cuda.memory._set_allocator_settings('expandable_segments:False')
+(VllmWorkerProcess pid=76766) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231] 
+(VllmWorkerProcess pid=76765) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231] Exception in worker VllmWorkerProcess while processing method init_device: Tensors allocated with expandable_segments:True cannot be shared between processes. Consider using expandable_segments:False in data loading workers via torch.cuda.memory._set_allocator_settings('expandable_segments:False'), Traceback (most recent call last):
+(VllmWorkerProcess pid=76765) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/executor/multiproc_worker_utils.py", line 224, in _run_worker_process
+(VllmWorkerProcess pid=76765) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     output = executor(*args, **kwargs)
+(VllmWorkerProcess pid=76765) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/worker/worker.py", line 176, in init_device
+(VllmWorkerProcess pid=76765) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     init_worker_distributed_environment(self.parallel_config, self.rank,
+(VllmWorkerProcess pid=76765) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/worker/worker.py", line 456, in init_worker_distributed_environment
+(VllmWorkerProcess pid=76765) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     ensure_model_parallel_initialized(parallel_config.tensor_parallel_size,
+(VllmWorkerProcess pid=76765) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 1051, in ensure_model_parallel_initialized
+(VllmWorkerProcess pid=76765) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     initialize_model_parallel(tensor_model_parallel_size,
+(VllmWorkerProcess pid=76765) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 1015, in initialize_model_parallel
+(VllmWorkerProcess pid=76765) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     _TP = init_model_parallel_group(group_ranks,
+(VllmWorkerProcess pid=76765) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 856, in init_model_parallel_group
+(VllmWorkerProcess pid=76765) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     return GroupCoordinator(
+(VllmWorkerProcess pid=76765) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 222, in __init__
+(VllmWorkerProcess pid=76765) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     self.ca_comm = CustomAllreduce(
+(VllmWorkerProcess pid=76765) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 171, in __init__
+(VllmWorkerProcess pid=76765) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     handles, offsets = self._get_ipc_meta(self.meta)
+(VllmWorkerProcess pid=76765) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 193, in _get_ipc_meta
+(VllmWorkerProcess pid=76765) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     data = inp.untyped_storage()._share_cuda_()
+(VllmWorkerProcess pid=76765) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231] RuntimeError: Tensors allocated with expandable_segments:True cannot be shared between processes. Consider using expandable_segments:False in data loading workers via torch.cuda.memory._set_allocator_settings('expandable_segments:False')
+(VllmWorkerProcess pid=76765) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231] 
+(VllmWorkerProcess pid=76767) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231] Exception in worker VllmWorkerProcess while processing method init_device: Tensors allocated with expandable_segments:True cannot be shared between processes. Consider using expandable_segments:False in data loading workers via torch.cuda.memory._set_allocator_settings('expandable_segments:False'), Traceback (most recent call last):
+(VllmWorkerProcess pid=76767) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/executor/multiproc_worker_utils.py", line 224, in _run_worker_process
+(VllmWorkerProcess pid=76767) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     output = executor(*args, **kwargs)
+(VllmWorkerProcess pid=76767) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/worker/worker.py", line 176, in init_device
+(VllmWorkerProcess pid=76767) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     init_worker_distributed_environment(self.parallel_config, self.rank,
+(VllmWorkerProcess pid=76767) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/worker/worker.py", line 456, in init_worker_distributed_environment
+(VllmWorkerProcess pid=76767) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     ensure_model_parallel_initialized(parallel_config.tensor_parallel_size,
+(VllmWorkerProcess pid=76767) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 1051, in ensure_model_parallel_initialized
+(VllmWorkerProcess pid=76767) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     initialize_model_parallel(tensor_model_parallel_size,
+(VllmWorkerProcess pid=76767) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 1015, in initialize_model_parallel
+(VllmWorkerProcess pid=76767) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     _TP = init_model_parallel_group(group_ranks,
+(VllmWorkerProcess pid=76767) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 856, in init_model_parallel_group
+(VllmWorkerProcess pid=76767) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     return GroupCoordinator(
+(VllmWorkerProcess pid=76767) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 222, in __init__
+(VllmWorkerProcess pid=76767) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     self.ca_comm = CustomAllreduce(
+(VllmWorkerProcess pid=76767) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 171, in __init__
+(VllmWorkerProcess pid=76767) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     handles, offsets = self._get_ipc_meta(self.meta)
+(VllmWorkerProcess pid=76767) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 193, in _get_ipc_meta
+(VllmWorkerProcess pid=76767) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     data = inp.untyped_storage()._share_cuda_()
+(VllmWorkerProcess pid=76767) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231] RuntimeError: Tensors allocated with expandable_segments:True cannot be shared between processes. Consider using expandable_segments:False in data loading workers via torch.cuda.memory._set_allocator_settings('expandable_segments:False')
+(VllmWorkerProcess pid=76767) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231] 
+(VllmWorkerProcess pid=76762) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231] Exception in worker VllmWorkerProcess while processing method init_device: Tensors allocated with expandable_segments:True cannot be shared between processes. Consider using expandable_segments:False in data loading workers via torch.cuda.memory._set_allocator_settings('expandable_segments:False'), Traceback (most recent call last):
+(VllmWorkerProcess pid=76762) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/executor/multiproc_worker_utils.py", line 224, in _run_worker_process
+(VllmWorkerProcess pid=76762) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     output = executor(*args, **kwargs)
+(VllmWorkerProcess pid=76762) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/worker/worker.py", line 176, in init_device
+(VllmWorkerProcess pid=76762) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     init_worker_distributed_environment(self.parallel_config, self.rank,
+(VllmWorkerProcess pid=76762) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/worker/worker.py", line 456, in init_worker_distributed_environment
+(VllmWorkerProcess pid=76762) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     ensure_model_parallel_initialized(parallel_config.tensor_parallel_size,
+(VllmWorkerProcess pid=76762) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 1051, in ensure_model_parallel_initialized
+(VllmWorkerProcess pid=76762) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     initialize_model_parallel(tensor_model_parallel_size,
+(VllmWorkerProcess pid=76762) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 1015, in initialize_model_parallel
+(VllmWorkerProcess pid=76762) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     _TP = init_model_parallel_group(group_ranks,
+(VllmWorkerProcess pid=76762) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 856, in init_model_parallel_group
+(VllmWorkerProcess pid=76762) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     return GroupCoordinator(
+(VllmWorkerProcess pid=76762) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 222, in __init__
+(VllmWorkerProcess pid=76762) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     self.ca_comm = CustomAllreduce(
+(VllmWorkerProcess pid=76762) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 171, in __init__
+(VllmWorkerProcess pid=76762) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     handles, offsets = self._get_ipc_meta(self.meta)
+(VllmWorkerProcess pid=76762) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 193, in _get_ipc_meta
+(VllmWorkerProcess pid=76762) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     data = inp.untyped_storage()._share_cuda_()
+(VllmWorkerProcess pid=76762) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231] RuntimeError: Tensors allocated with expandable_segments:True cannot be shared between processes. Consider using expandable_segments:False in data loading workers via torch.cuda.memory._set_allocator_settings('expandable_segments:False')
+(VllmWorkerProcess pid=76762) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231] 
+(VllmWorkerProcess pid=76764) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231] Exception in worker VllmWorkerProcess while processing method init_device: Tensors allocated with expandable_segments:True cannot be shared between processes. Consider using expandable_segments:False in data loading workers via torch.cuda.memory._set_allocator_settings('expandable_segments:False'), Traceback (most recent call last):
+(VllmWorkerProcess pid=76764) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/executor/multiproc_worker_utils.py", line 224, in _run_worker_process
+(VllmWorkerProcess pid=76764) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     output = executor(*args, **kwargs)
+(VllmWorkerProcess pid=76764) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/worker/worker.py", line 176, in init_device
+(VllmWorkerProcess pid=76764) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     init_worker_distributed_environment(self.parallel_config, self.rank,
+(VllmWorkerProcess pid=76764) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/worker/worker.py", line 456, in init_worker_distributed_environment
+(VllmWorkerProcess pid=76764) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     ensure_model_parallel_initialized(parallel_config.tensor_parallel_size,
+(VllmWorkerProcess pid=76764) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 1051, in ensure_model_parallel_initialized
+(VllmWorkerProcess pid=76764) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     initialize_model_parallel(tensor_model_parallel_size,
+(VllmWorkerProcess pid=76764) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 1015, in initialize_model_parallel
+(VllmWorkerProcess pid=76764) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     _TP = init_model_parallel_group(group_ranks,
+(VllmWorkerProcess pid=76764) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 856, in init_model_parallel_group
+(VllmWorkerProcess pid=76764) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     return GroupCoordinator(
+(VllmWorkerProcess pid=76764) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 222, in __init__
+(VllmWorkerProcess pid=76764) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     self.ca_comm = CustomAllreduce(
+(VllmWorkerProcess pid=76764) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 171, in __init__
+(VllmWorkerProcess pid=76764) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     handles, offsets = self._get_ipc_meta(self.meta)
+(VllmWorkerProcess pid=76764) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 193, in _get_ipc_meta
+(VllmWorkerProcess pid=76764) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231]     data = inp.untyped_storage()._share_cuda_()
+(VllmWorkerProcess pid=76764) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231] RuntimeError: Tensors allocated with expandable_segments:True cannot be shared between processes. Consider using expandable_segments:False in data loading workers via torch.cuda.memory._set_allocator_settings('expandable_segments:False')
+(VllmWorkerProcess pid=76764) ERROR 10-21 20:10:29 multiproc_worker_utils.py:231] 
+[rank0]: Traceback (most recent call last):
+[rank0]:   File "/opt/tiger/mariana/llama_output/vllm_generate.py", line 361, in <module>
+[rank0]:     main()
+[rank0]:   File "/opt/tiger/mariana/llama_output/vllm_generate.py", line 357, in main
+[rank0]:     generation(args)
+[rank0]:   File "/opt/tiger/mariana/llama_output/vllm_generate.py", line 244, in generation
+[rank0]:     model = LLM(
+[rank0]:   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/entrypoints/llm.py", line 177, in __init__
+[rank0]:     self.llm_engine = LLMEngine.from_engine_args(
+[rank0]:   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/engine/llm_engine.py", line 574, in from_engine_args
+[rank0]:     engine = cls(
+[rank0]:   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/engine/llm_engine.py", line 335, in __init__
+[rank0]:     self.model_executor = executor_class(
+[rank0]:   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/executor/distributed_gpu_executor.py", line 26, in __init__
+[rank0]:     super().__init__(*args, **kwargs)
+[rank0]:   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/executor/executor_base.py", line 47, in __init__
+[rank0]:     self._init_executor()
+[rank0]:   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/executor/multiproc_gpu_executor.py", line 110, in _init_executor
+[rank0]:     self._run_workers("init_device")
+[rank0]:   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/executor/multiproc_gpu_executor.py", line 192, in _run_workers
+[rank0]:     driver_worker_output = driver_worker_method(*args, **kwargs)
+[rank0]:   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/worker/worker.py", line 176, in init_device
+[rank0]:     init_worker_distributed_environment(self.parallel_config, self.rank,
+[rank0]:   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/worker/worker.py", line 456, in init_worker_distributed_environment
+[rank0]:     ensure_model_parallel_initialized(parallel_config.tensor_parallel_size,
+[rank0]:   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 1051, in ensure_model_parallel_initialized
+[rank0]:     initialize_model_parallel(tensor_model_parallel_size,
+[rank0]:   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 1015, in initialize_model_parallel
+[rank0]:     _TP = init_model_parallel_group(group_ranks,
+[rank0]:   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 856, in init_model_parallel_group
+[rank0]:     return GroupCoordinator(
+[rank0]:   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 222, in __init__
+[rank0]:     self.ca_comm = CustomAllreduce(
+[rank0]:   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 171, in __init__
+[rank0]:     handles, offsets = self._get_ipc_meta(self.meta)
+[rank0]:   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 193, in _get_ipc_meta
+[rank0]:     data = inp.untyped_storage()._share_cuda_()
+[rank0]: RuntimeError: Tensors allocated with expandable_segments:True cannot be shared between processes. Consider using expandable_segments:False in data loading workers via torch.cuda.memory._set_allocator_settings('expandable_segments:False')
+[rank0]:[I1021 20:10:29.253456024 TCPStoreLibUvBackend.cpp:115] [c10d - debug] Read callback failed. code:-4095 name:EOF desc:end of file
+[rank0]:[I1021 20:10:29.253598016 TCPStoreLibUvBackend.cpp:115] [c10d - debug] Read callback failed. code:-4095 name:EOF desc:end of file
+[rank0]:[I1021 20:10:29.254700814 TCPStoreLibUvBackend.cpp:115] [c10d - debug] Read callback failed. code:-4095 name:EOF desc:end of file
+[rank0]:[I1021 20:10:29.259816995 TCPStoreLibUvBackend.cpp:115] [c10d - debug] Read callback failed. code:-4095 name:EOF desc:end of file
+[rank0]:[I1021 20:10:29.261357971 TCPStoreLibUvBackend.cpp:115] [c10d - debug] Read callback failed. code:-4095 name:EOF desc:end of file
+[rank0]:[I1021 20:10:29.262652623 TCPStoreLibUvBackend.cpp:115] [c10d - debug] Read callback failed. code:-4095 name:EOF desc:end of file
+[rank0]:[I1021 20:10:29.263323241 TCPStoreLibUvBackend.cpp:115] [c10d - debug] Read callback failed. code:-4095 name:EOF desc:end of file
+INFO 10-21 20:10:31 multiproc_worker_utils.py:121] Killing local vLLM worker processes
+Exception ignored in: <function CustomAllreduce.__del__ at 0x7fba622b35e0>
+Traceback (most recent call last):
+  File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 292, in __del__
+    self.close()
+  File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 287, in close
+    if not self.disabled and self._ptr:
+AttributeError: 'CustomAllreduce' object has no attribute '_ptr'

log/zero_shot/bd_math/generation/llama3.1_70b/1/0-4.log

@@ -0,0 +1,346 @@
+[I1021 17:53:02.628389982 debug.cpp:49] [c10d] The debug level is set to INFO.
+llama3.1_70b
+*****************************
+Namespace(cot_trigger_no=1, dataset='bd_math', data_path='bd_math_test.json', batch_size=64, eval_method='', model_path='../../Meta-Llama-3.1-70B', model_type='llama3.1_70b', output_dir='generate_result/zero_shot/bd_math/generation/llama3.1_70b/1/', lora_path='', method='zero_shot', data_question_key='question', data_answer_key='answer', sample_num=1, cuda_ind=4, tensor_parallel=4, cuda_start=0, cuda_num=8, load_in_8bit=False, rewrite=False, use_typewriter=0, temperature=0.7, top_p=1, iter_max_new_tokens=512, init_max_new_tokens=2048, min_new_tokens=1, correct_response_format='The correct response is:', cot_trigger="Let's think step by step.")
+*****************************
+INFO 10-21 17:53:06 config.py:729] Defaulting to use mp for distributed inference
+WARNING 10-21 17:53:06 arg_utils.py:766] Chunked prefill is enabled by default for models with max_model_len > 32K. Currently, chunked prefill might not work with some features or models. If you encounter any issues, please disable chunked prefill by setting --enable-chunked-prefill=False.
+INFO 10-21 17:53:06 config.py:820] Chunked prefill is enabled with max_num_batched_tokens=512.
+INFO 10-21 17:53:06 llm_engine.py:174] Initializing an LLM engine (v0.5.4) with config: model='../../Meta-Llama-3.1-70B', speculative_config=None, tokenizer='../../Meta-Llama-3.1-70B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=4, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None), seed=0, served_model_name=../../Meta-Llama-3.1-70B, use_v2_block_manager=False, enable_prefix_caching=False)
+INFO 10-21 17:53:06 custom_cache_manager.py:17] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager
+(VllmWorkerProcess pid=33956) INFO 10-21 17:53:07 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
+(VllmWorkerProcess pid=33962) INFO 10-21 17:53:07 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
+(VllmWorkerProcess pid=33958) INFO 10-21 17:53:07 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
+[I1021 17:53:18.555010373 TCPStore.cpp:312] [c10d - debug] The server has started on port = 35019.
+[I1021 17:53:18.555162932 TCPStoreLibUvBackend.cpp:1067] [c10d - debug] Uv main loop running
+[I1021 17:53:18.559166150 socket.cpp:720] [c10d - debug] The client socket will attempt to connect to an IPv6 address of (127.0.0.1, 35019).
+[I1021 17:53:18.559283422 socket.cpp:884] [c10d] The client socket has connected to [localhost]:35019 on [localhost]:46984.
+[I1021 17:53:18.562060112 TCPStore.cpp:350] [c10d - debug] TCP client connected to host 127.0.0.1:35019
+[I1021 17:53:20.136371404 socket.cpp:720] [c10d - debug] The client socket will attempt to connect to an IPv6 address of (127.0.0.1, 35019).
+[I1021 17:53:20.136589163 socket.cpp:884] [c10d] The client socket has connected to [localhost]:35019 on [localhost]:46996.
+[I1021 17:53:20.139467602 TCPStore.cpp:350] [c10d - debug] TCP client connected to host 127.0.0.1:35019
+[W1021 17:53:20.139957063 CUDAAllocatorConfig.h:28] Warning: expandable_segments not supported on this platform (function operator())
+[I1021 17:53:20.140041319 ProcessGroupNCCL.cpp:852] [PG 0 Rank 2] ProcessGroupNCCL initialization options: size: 4, global rank: 2, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0, SPLIT_COLOR: 0, PG Name: 0
+[I1021 17:53:20.140051030 ProcessGroupNCCL.cpp:861] [PG 0 Rank 2] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank2]:[I1021 17:53:20.140986237 ProcessGroupNCCL.cpp:852] [PG 1 Rank 2] ProcessGroupNCCL initialization options: size: 4, global rank: 2, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xc122fd0, SPLIT_COLOR: 1008299991543067201, PG Name: 1
+[rank2]:[I1021 17:53:20.140996004 ProcessGroupNCCL.cpp:861] [PG 1 Rank 2] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[I1021 17:53:20.167810131 socket.cpp:720] [c10d - debug] The client socket will attempt to connect to an IPv6 address of (127.0.0.1, 35019).
+[I1021 17:53:20.167960138 socket.cpp:884] [c10d] The client socket has connected to [localhost]:35019 on [localhost]:47006.
+[I1021 17:53:20.170691050 TCPStore.cpp:350] [c10d - debug] TCP client connected to host 127.0.0.1:35019
+[W1021 17:53:20.171131772 CUDAAllocatorConfig.h:28] Warning: expandable_segments not supported on this platform (function operator())
+[I1021 17:53:20.171204576 ProcessGroupNCCL.cpp:852] [PG 0 Rank 1] ProcessGroupNCCL initialization options: size: 4, global rank: 1, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0, SPLIT_COLOR: 0, PG Name: 0
+[I1021 17:53:20.171212914 ProcessGroupNCCL.cpp:861] [PG 0 Rank 1] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank1]:[I1021 17:53:20.172079599 ProcessGroupNCCL.cpp:852] [PG 1 Rank 1] ProcessGroupNCCL initialization options: size: 4, global rank: 1, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xc122f90, SPLIT_COLOR: 1008299991543067201, PG Name: 1
+[rank1]:[I1021 17:53:20.172090869 ProcessGroupNCCL.cpp:861] [PG 1 Rank 1] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[I1021 17:53:20.503857827 socket.cpp:720] [c10d - debug] The client socket will attempt to connect to an IPv6 address of (127.0.0.1, 35019).
+[I1021 17:53:20.504052271 socket.cpp:884] [c10d] The client socket has connected to [localhost]:35019 on [localhost]:47018.
+[I1021 17:53:20.507325410 TCPStore.cpp:350] [c10d - debug] TCP client connected to host 127.0.0.1:35019
+[W1021 17:53:20.508194974 CUDAAllocatorConfig.h:28] Warning: expandable_segments not supported on this platform (function operator())
+[I1021 17:53:20.508331472 ProcessGroupNCCL.cpp:852] [PG 0 Rank 3] ProcessGroupNCCL initialization options: size: 4, global rank: 3, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0, SPLIT_COLOR: 0, PG Name: 0
+[I1021 17:53:20.508342990 ProcessGroupNCCL.cpp:861] [PG 0 Rank 3] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank3]:[I1021 17:53:20.509598213 ProcessGroupNCCL.cpp:852] [PG 1 Rank 3] ProcessGroupNCCL initialization options: size: 4, global rank: 3, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xc122f90, SPLIT_COLOR: 1008299991543067201, PG Name: 1
+[rank3]:[I1021 17:53:20.509615301 ProcessGroupNCCL.cpp:861] [PG 1 Rank 3] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[W1021 17:53:20.517814527 CUDAAllocatorConfig.h:28] Warning: expandable_segments not supported on this platform (function operator())
+[I1021 17:53:20.517928171 ProcessGroupNCCL.cpp:852] [PG 0 Rank 0] ProcessGroupNCCL initialization options: size: 4, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0, SPLIT_COLOR: 0, PG Name: 0
+[I1021 17:53:20.517937931 ProcessGroupNCCL.cpp:861] [PG 0 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank0]:[I1021 17:53:20.518675944 ProcessGroupNCCL.cpp:852] [PG 1 Rank 0] ProcessGroupNCCL initialization options: size: 4, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xc150ff0, SPLIT_COLOR: 1008299991543067201, PG Name: 1
+[rank0]:[I1021 17:53:20.518689693 ProcessGroupNCCL.cpp:861] [PG 1 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank0]:[I1021 17:53:20.535230190 ProcessGroupNCCL.cpp:852] [PG 3 Rank 0] ProcessGroupNCCL initialization options: size: 4, global rank: 0, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xc150ff0, SPLIT_COLOR: 1008299991543067201, PG Name: 3
+[rank0]:[I1021 17:53:20.535250236 ProcessGroupNCCL.cpp:861] [PG 3 Rank 0] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank2]:[I1021 17:53:20.535503951 ProcessGroupNCCL.cpp:852] [PG 3 Rank 2] ProcessGroupNCCL initialization options: size: 4, global rank: 2, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xc122fd0, SPLIT_COLOR: 1008299991543067201, PG Name: 3
+[rank2]:[I1021 17:53:20.535530572 ProcessGroupNCCL.cpp:861] [PG 3 Rank 2] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank3]:[I1021 17:53:20.535693243 ProcessGroupNCCL.cpp:852] [PG 3 Rank 3] ProcessGroupNCCL initialization options: size: 4, global rank: 3, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xc122f90, SPLIT_COLOR: 1008299991543067201, PG Name: 3
+[rank1]:[I1021 17:53:20.535703220 ProcessGroupNCCL.cpp:852] [PG 3 Rank 1] ProcessGroupNCCL initialization options: size: 4, global rank: 1, TIMEOUT(ms): 600000, USE_HIGH_PRIORITY_STREAM: 0, SPLIT_FROM: 0xc122f90, SPLIT_COLOR: 1008299991543067201, PG Name: 3
+[rank3]:[I1021 17:53:20.535714186 ProcessGroupNCCL.cpp:861] [PG 3 Rank 3] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+[rank1]:[I1021 17:53:20.535724952 ProcessGroupNCCL.cpp:861] [PG 3 Rank 1] ProcessGroupNCCL environments: NCCL version: 2.20.5, TORCH_NCCL_ASYNC_ERROR_HANDLING: 3, TORCH_NCCL_DUMP_ON_TIMEOUT: 0, TORCH_NCCL_WAIT_TIMEOUT_DUMP_MILSEC: 60000, TORCH_NCCL_DESYNC_DEBUG: 0, TORCH_NCCL_ENABLE_TIMING: 0, TORCH_NCCL_BLOCKING_WAIT: 0, TORCH_DISTRIBUTED_DEBUG: INFO, TORCH_NCCL_USE_TENSOR_REGISTER_ALLOCATOR_HOOK: 0, TORCH_NCCL_ENABLE_MONITORING: 1, TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC: 600, TORCH_NCCL_TRACE_BUFFER_SIZE: 0, TORCH_NCCL_COORD_CHECK_MILSEC: 1000, TORCH_NCCL_NAN_CHECK: 0
+(VllmWorkerProcess pid=33962) INFO 10-21 17:53:20 utils.py:841] Found nccl from library libnccl.so.2
+(VllmWorkerProcess pid=33958) INFO 10-21 17:53:20 utils.py:841] Found nccl from library libnccl.so.2
+(VllmWorkerProcess pid=33956) INFO 10-21 17:53:20 utils.py:841] Found nccl from library libnccl.so.2
+(VllmWorkerProcess pid=33962) INFO 10-21 17:53:20 pynccl.py:63] vLLM is using nccl==2.20.5
+INFO 10-21 17:53:20 utils.py:841] Found nccl from library libnccl.so.2
+INFO 10-21 17:53:20 pynccl.py:63] vLLM is using nccl==2.20.5
+(VllmWorkerProcess pid=33958) INFO 10-21 17:53:20 pynccl.py:63] vLLM is using nccl==2.20.5
+(VllmWorkerProcess pid=33956) INFO 10-21 17:53:20 pynccl.py:63] vLLM is using nccl==2.20.5
+n117-192-077:33856:33856 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
+n117-192-077:33856:33856 [0] NCCL INFO Bootstrap : Using eth0:10.117.192.77<0>
+n117-192-077:33856:33856 [0] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v8 symbol.
+n117-192-077:33856:33856 [0] NCCL INFO NET/Plugin: Loaded net plugin NCCL RDMA Plugin v7 (v7)
+n117-192-077:33856:33856 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v8 symbol.
+n117-192-077:33856:33856 [0] NCCL INFO NET/Plugin: Loaded coll plugin SHARP (v7)
+n117-192-077:33856:33856 [0] NCCL INFO cudaDriverVersion 12020
+NCCL version 2.20.5+cuda12.4
+n117-192-077:33962:33962 [3] NCCL INFO cudaDriverVersion 12020
+n117-192-077:33962:33962 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
+n117-192-077:33962:33962 [3] NCCL INFO Bootstrap : Using eth0:10.117.192.77<0>
+n117-192-077:33962:33962 [3] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v8 symbol.
+n117-192-077:33962:33962 [3] NCCL INFO NET/Plugin: Loaded net plugin NCCL RDMA Plugin v7 (v7)
+n117-192-077:33962:33962 [3] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v8 symbol.
+n117-192-077:33962:33962 [3] NCCL INFO NET/Plugin: Loaded coll plugin SHARP (v7)
+n117-192-077:33962:33962 [3] NCCL INFO Plugin Path : /usr/local/lib/libnccl-net.so
+n117-192-077:33962:33962 [3] NCCL INFO P2P plugin IBext_v7
+n117-192-077:33962:33962 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
+n117-192-077:33962:33962 [3] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB/SHARP [1]mlx5_1:1/IB/SHARP [2]mlx5_2:1/IB/SHARP [3]mlx5_3:1/IB/SHARP [4]mlx5_4:1/IB/SHARP [5]mlx5_5:1/IB/SHARP [6]mlx5_6:1/IB/SHARP [7]mlx5_7:1/IB/SHARP [RO]; OOB eth0:10.117.192.77<0>
+n117-192-077:33962:33962 [3] NCCL INFO Using non-device net plugin version 0
+n117-192-077:33962:33962 [3] NCCL INFO Using network IBext_v7
+n117-192-077:33962:33962 [3] NCCL INFO comm 0xc1d3870 rank 3 nranks 4 cudaDev 3 nvmlDev 7 busId c00000 commId 0xe7bd6663f99671f6 - Init START
+n117-192-077:33962:33962 [3] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/tiger/arnold/arnold_entrypoint/nccl_topo_files/azure_ndv5_topo.xml
+n117-192-077:33962:33962 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
+n117-192-077:33962:33962 [3] NCCL INFO Setting affinity for GPU 7 to ffff,ffffffff
+n117-192-077:33962:33962 [3] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0.
+n117-192-077:33962:33962 [3] NCCL INFO comm 0xc1d3870 rank 3 nRanks 4 nNodes 1 localRanks 4 localRank 3 MNNVL 0
+n117-192-077:33962:33962 [3] NCCL INFO NCCL_MAX_NCHANNELS set by environment to 12.
+n117-192-077:33962:33962 [3] NCCL INFO Trees [0] -1/-1/-1->3->2 [1] -1/-1/-1->3->2 [2] -1/-1/-1->3->2 [3] -1/-1/-1->3->2 [4] -1/-1/-1->3->2 [5] -1/-1/-1->3->2 [6] -1/-1/-1->3->2 [7] -1/-1/-1->3->2 [8] -1/-1/-1->3->2 [9] -1/-1/-1->3->2 [10] -1/-1/-1->3->2 [11] -1/-1/-1->3->2
+n117-192-077:33962:33962 [3] NCCL INFO P2P Chunksize set to 524288
+n117-192-077:33962:33962 [3] NCCL INFO Channel 00/0 : 3[7] -> 0[4] via P2P/IPC
+n117-192-077:33962:33962 [3] NCCL INFO Channel 01/0 : 3[7] -> 0[4] via P2P/IPC
+n117-192-077:33962:33962 [3] NCCL INFO Channel 02/0 : 3[7] -> 0[4] via P2P/IPC
+n117-192-077:33962:33962 [3] NCCL INFO Channel 03/0 : 3[7] -> 0[4] via P2P/IPC
+n117-192-077:33962:33962 [3] NCCL INFO Channel 04/0 : 3[7] -> 0[4] via P2P/IPC
+n117-192-077:33962:33962 [3] NCCL INFO Channel 05/0 : 3[7] -> 0[4] via P2P/IPC
+n117-192-077:33962:33962 [3] NCCL INFO Channel 06/0 : 3[7] -> 0[4] via P2P/IPC
+n117-192-077:33962:33962 [3] NCCL INFO Channel 07/0 : 3[7] -> 0[4] via P2P/IPC
+n117-192-077:33962:33962 [3] NCCL INFO Channel 08/0 : 3[7] -> 0[4] via P2P/IPC
+n117-192-077:33962:33962 [3] NCCL INFO Channel 09/0 : 3[7] -> 0[4] via P2P/IPC
+n117-192-077:33962:33962 [3] NCCL INFO Channel 10/0 : 3[7] -> 0[4] via P2P/IPC
+n117-192-077:33962:33962 [3] NCCL INFO Channel 11/0 : 3[7] -> 0[4] via P2P/IPC
+n117-192-077:33962:33962 [3] NCCL INFO Connected all rings
+n117-192-077:33962:33962 [3] NCCL INFO Channel 00/0 : 3[7] -> 2[6] via P2P/IPC
+n117-192-077:33962:33962 [3] NCCL INFO Channel 01/0 : 3[7] -> 2[6] via P2P/IPC
+n117-192-077:33962:33962 [3] NCCL INFO Channel 02/0 : 3[7] -> 2[6] via P2P/IPC
+n117-192-077:33962:33962 [3] NCCL INFO Channel 03/0 : 3[7] -> 2[6] via P2P/IPC
+n117-192-077:33962:33962 [3] NCCL INFO Channel 04/0 : 3[7] -> 2[6] via P2P/IPC
+n117-192-077:33962:33962 [3] NCCL INFO Channel 05/0 : 3[7] -> 2[6] via P2P/IPC
+n117-192-077:33962:33962 [3] NCCL INFO Channel 06/0 : 3[7] -> 2[6] via P2P/IPC
+n117-192-077:33962:33962 [3] NCCL INFO Channel 07/0 : 3[7] -> 2[6] via P2P/IPC
+n117-192-077:33962:33962 [3] NCCL INFO Channel 08/0 : 3[7] -> 2[6] via P2P/IPC
+n117-192-077:33962:33962 [3] NCCL INFO Channel 09/0 : 3[7] -> 2[6] via P2P/IPC
+n117-192-077:3396n117-192-077:33958:33958 [2] NCCL INFO cudaDriverVersion 12020
+n117-192-077:33958:33958 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
+n117-192-077:33958:33958 [2] NCCL INFO Bootstrap : Using eth0:10.117.192.77<0>
+n117-192-077:33958:33958 [2] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v8 symbol.
+n117-192-077:33958:33958 [2] NCCL INFO NET/Plugin: Loaded net plugin NCCL RDMA Plugin v7 (v7)
+n117-192-077:33958:33958 [2] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v8 symbol.
+n117-192-077:33958:33958 [2] NCCL INFO NET/Plugin: Loaded coll plugin SHARP (v7)
+n117-192-077:33958:33958 [2] NCCL INFO Plugin Path : /usr/local/lib/libnccl-net.so
+n117-192-077:33958:33958 [2] NCCL INFO P2P plugin IBext_v7
+n117-192-077:33958:33958 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
+n117-192-077:33958:33958 [2] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB/SHARP [1]mlx5_1:1/IB/SHARP [2]mlx5_2:1/IB/SHARP [3]mlx5_3:1/IB/SHARP [4]mlx5_4:1/IB/SHARP [5]mlx5_5:1/IB/SHARP [6]mlx5_6:1/IB/SHARP [7]mlx5_7:1/IB/SHARP [RO]; OOB eth0:10.117.192.77<0>
+n117-192-077:33958:33958 [2] NCCL INFO Using non-device net plugin version 0
+n117-192-077:33958:33958 [2] NCCL INFO Using network IBext_v7
+n117-192-077:33958:33958 [2] NCCL INFO comm 0xc1d38b0 rank 2 nranks 4 cudaDev 2 nvmlDev 6 busId b00000 commId 0xe7bd6663f99671f6 - Init START
+n117-192-077:33958:33958 [2] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/tiger/arnold/arnold_entrypoint/nccl_topo_files/azure_ndv5_topo.xml
+n117-192-077:33958:33958 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
+n117-192-077:33958:33958 [2] NCCL INFO Setting affinity for GPU 6 to ffff,ffffffff
+n117-192-077:33958:33958 [2] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0.
+n117-192-077:33958:33958 [2] NCCL INFO comm 0xc1d38b0 rank 2 nRanks 4 nNodes 1 localRanks 4 localRank 2 MNNVL 0
+n117-192-077:33958:33958 [2] NCCL INFO NCCL_MAX_NCHANNELS set by environment to 12.
+n117-192-077:33958:33958 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 3/-1/-1->2->1 [2] 3/-1/-1->2->1 [3] 3/-1/-1->2->1 [4] 3/-1/-1->2->1 [5] 3/-1/-1->2->1 [6] 3/-1/-1->2->1 [7] 3/-1/-1->2->1 [8] 3/-1/-1->2->1 [9] 3/-1/-1->2->1 [10] 3/-1/-1->2->1 [11] 3/-1/-1->2->1
+n117-192-077:33958:33958 [2] NCCL INFO P2P Chunksize set to 524288
+n117-192-077:33958:33958 [2] NCCL INFO Channel 00/0 : 2[6] -> 3[7] via P2P/IPC
+n117-192-077:33958:33958 [2] NCCL INFO Channel 01/0 : 2[6] -> 3[7] via P2P/IPC
+n117-192-077:33958:33958 [2] NCCL INFO Channel 02/0 : 2[6] -> 3[7] via P2P/IPC
+n117-192-077:33958:33958 [2] NCCL INFO Channel 03/0 : 2[6] -> 3[7] via P2P/IPC
+n117-192-077:33958:33958 [2] NCCL INFO Channel 04/0 : 2[6] -> 3[7] via P2P/IPC
+n117-192-077:33958:33958 [2] NCCL INFO Channel 05/0 : 2[6] -> 3[7] via P2P/IPC
+n117-192-077:33958:33958 [2] NCCL INFO Channel 06/0 : 2[6] -> 3[7] via P2P/IPC
+n117-192-077:33958:33958 [2] NCCL INFO Channel 07/0 : 2[6] -> 3[7] via P2P/IPC
+n117-192-077:33958:33958 [2] NCCL INFO Channel 08/0 : 2[6] -> 3[7] via P2P/IPC
+n117-192-077:33958:33958 [2] NCCL INFO Channel 09/0 : 2[6] -> 3[7] via P2P/IPC
+n117-192-077:33958:33958 [2] NCCL INFO Channel 10/0 : 2[6] -> 3[7] via P2P/IPC
+n117-192-077:33958:33958 [2] NCCL INFO Channel 11/0 : 2[6] -> 3[7] via P2P/IPC
+n117-192-077:33958:33958 [2] NCCL INFO Connected all rings
+n117-192-077:33958:33958 [2] NCCL INFO Channel 00/0 : 2[6] -> 1[5] via P2P/IPC
+n117-192-077:33958:33958 [2] NCCL INFO Channel 01/0 : 2[6] -> 1[5] via P2P/IPC
+n117-192-077:33958:33958 [2] NCCL INFO Channel 02/0 : 2[6] -> 1[5] via P2P/IPC
+n117-192-077:33958:33958 [2] NCCL INFO Channel 03/0 : 2[6] -> 1[5] via P2P/IPC
+n117-192-077:33958:33958 [2] NCCL INFO Channel 04/0 : 2[6] -> 1[5] via P2P/IPC
+n117-192-077:33958:33958 [2] NCCL INFO Channel 05/0 : 2[6] -> 1[5] via P2P/IPC
+n117-192-077:33958:33958 [2] NCCL INFO Channel 06/0 : 2[6] -> 1[5] via P2P/IPC
+n117-192-077:33958:33958 [2] NCCL INFO Channel 07/0 : 2[6] -> 1[5] via P2P/IPC
+n117-192-077:33958:33958 [2] NCCL INFO Channel 08/0 : 2[6] -> 1[5] via P2P/IPC
+n117-192-077:33958:33958 [2] NCCL INFO Channel 09/0 : 2[6] -> 1[5] via P2P/IPC
+n117-192-077:33958:33958 [2] n117-192-077:33956:33956 [1] NCCL INFO cudaDriverVersion 12020
+n117-192-077:33956:33956 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
+n117-192-077:33956:33956 [1] NCCL INFO Bootstrap : Using eth0:10.117.192.77<0>
+n117-192-077:33956:33956 [1] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v8 symbol.
+n117-192-077:33956:33956 [1] NCCL INFO NET/Plugin: Loaded net plugin NCCL RDMA Plugin v7 (v7)
+n117-192-077:33956:33956 [1] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v8 symbol.
+n117-192-077:33956:33956 [1] NCCL INFO NET/Plugin: Loaded coll plugin SHARP (v7)
+n117-192-077:33956:33956 [1] NCCL INFO Plugin Path : /usr/local/lib/libnccl-net.so
+n117-192-077:33956:33956 [1] NCCL INFO P2P plugin IBext_v7
+n117-192-077:33956:33956 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
+n117-192-077:33956:33956 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB/SHARP [1]mlx5_1:1/IB/SHARP [2]mlx5_2:1/IB/SHARP [3]mlx5_3:1/IB/SHARP [4]mlx5_4:1/IB/SHARP [5]mlx5_5:1/IB/SHARP [6]mlx5_6:1/IB/SHARP [7]mlx5_7:1/IB/SHARP [RO]; OOB eth0:10.117.192.77<0>
+n117-192-077:33956:33956 [1] NCCL INFO Using non-device net plugin version 0
+n117-192-077:33956:33956 [1] NCCL INFO Using network IBext_v7
+n117-192-077:33956:33956 [1] NCCL INFO comm 0xc1d3970 rank 1 nranks 4 cudaDev 1 nvmlDev 5 busId a00000 commId 0xe7bd6663f99671f6 - Init START
+n117-192-077:33956:33956 [1] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/tiger/arnold/arnold_entrypoint/nccl_topo_files/azure_ndv5_topo.xml
+n117-192-077:33956:33956 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
+n117-192-077:33956:33956 [1] NCCL INFO Setting affinity for GPU 5 to ffff,ffffffff
+n117-192-077:33956:33956 [1] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0.
+n117-192-077:33956:33956 [1] NCCL INFO comm 0xc1d3970 rank 1 nRanks 4 nNodes 1 localRanks 4 localRank 1 MNNVL 0
+n117-192-077:33956:33956 [1] NCCL INFO NCCL_MAX_NCHANNELS set by environment to 12.
+n117-192-077:33956:33956 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/-1/-1->1->0 [2] 2/-1/-1->1->0 [3] 2/-1/-1->1->0 [4] 2/-1/-1->1->0 [5] 2/-1/-1->1->0 [6] 2/-1/-1->1->0 [7] 2/-1/-1->1->0 [8] 2/-1/-1->1->0 [9] 2/-1/-1->1->0 [10] 2/-1/-1->1->0 [11] 2/-1/-1->1->0
+n117-192-077:33956:33956 [1] NCCL INFO P2P Chunksize set to 524288
+n117-192-077:33956:33956 [1] NCCL INFO Channel 00/0 : 1[5] -> 2[6] via P2P/IPC
+n117-192-077:33956:33956 [1] NCCL INFO Channel 01/0 : 1[5] -> 2[6] via P2P/IPC
+n117-192-077:33956:33956 [1] NCCL INFO Channel 02/0 : 1[5] -> 2[6] via P2P/IPC
+n117-192-077:33956:33956 [1] NCCL INFO Channel 03/0 : 1[5] -> 2[6] via P2P/IPC
+n117-192-077:33956:33956 [1] NCCL INFO Channel 04/0 : 1[5] -> 2[6] via P2P/IPC
+n117-192-077:33956:33956 [1] NCCL INFO Channel 05/0 : 1[5] -> 2[6] via P2P/IPC
+n117-192-077:33956:33956 [1] NCCL INFO Channel 06/0 : 1[5] -> 2[6] via P2P/IPC
+n117-192-077:33956:33956 [1] NCCL INFO Channel 07/0 : 1[5] -> 2[6] via P2P/IPC
+n117-192-077:33956:33956 [1] NCCL INFO Channel 08/0 : 1[5] -> 2[6] via P2P/IPC
+n117-192-077:33956:33956 [1] NCCL INFO Channel 09/0 : 1[5] -> 2[6] via P2P/IPC
+n117-192-077:33956:33956 [1] NCCL INFO Channel 10/0 : 1[5] -> 2[6] via P2P/IPC
+n117-192-077:33956:33956 [1] NCCL INFO Channel 11/0 : 1[5] -> 2[6] via P2P/IPC
+n117-192-077:33956:33956 [1] NCCL INFO Connected all rings
+n117-192-077:33956:33956 [1] NCCL INFO Channel 00/0 : 1[5] -> 0[4] via P2P/IPC
+n117-192-077:33956:33956 [1] NCCL INFO Channel 01/0 : 1[5] -> 0[4] via P2P/IPC
+n117-192-077:33956:33956 [1] NCCL INFO Channel 02/0 : 1[5] -> 0[4] via P2P/IPC
+n117-192-077:33956:33956 [1] NCCL INFO Channel 03/0 : 1[5] -> 0[4] via P2P/IPC
+n117-192-077:33956:33956 [1] NCCL INFO Channel 04/0 : 1[5] -> 0[4] via P2P/IPC
+n117-192-077:33956:33956 [1] NCCL INFO Channel 05/0 : 1[5] -> 0[4] via P2P/IPC
+n117-192-077:33956:33956 [1] NCCL INFO Channel 06/0 : 1[5] -> 0[4] via P2P/IPC
+n117-192-077:33956:33956 [1] NCCL INFO Channel 07/0 : 1[5] -> 0[4] via P2P/IPC
+n117-192-077:33956:33956 [1] NCCL INFO Channel 08/0 : 1[5] -> 0[4] via P2P/IPC
+n117-192-077:33956:33956 [1] NCCL INFO Channel 09/0 : 1[5] -> 0[4] via P2P/IPC
+n117-192-077:33956:33956 [1] n117-192-077:33856:33856 [0] NCCL INFO Plugin Path : /usr/local/lib/libnccl-net.so
+n117-192-077:33856:33856 [0] NCCL INFO P2P plugin IBext_v7
+n117-192-077:33856:33856 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0
+n117-192-077:33856:33856 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB/SHARP [1]mlx5_1:1/IB/SHARP [2]mlx5_2:1/IB/SHARP [3]mlx5_3:1/IB/SHARP [4]mlx5_4:1/IB/SHARP [5]mlx5_5:1/IB/SHARP [6]mlx5_6:1/IB/SHARP [7]mlx5_7:1/IB/SHARP [RO]; OOB eth0:10.117.192.77<0>
+n117-192-077:33856:33856 [0] NCCL INFO Using non-device net plugin version 0
+n117-192-077:33856:33856 [0] NCCL INFO Using network IBext_v7
+n117-192-077:33856:33856 [0] NCCL INFO comm 0xc1d4a10 rank 0 nranks 4 cudaDev 0 nvmlDev 4 busId 900000 commId 0xe7bd6663f99671f6 - Init START
+n117-192-077:33856:33856 [0] NCCL INFO NCCL_TOPO_FILE set by environment to /opt/tiger/arnold/arnold_entrypoint/nccl_topo_files/azure_ndv5_topo.xml
+n117-192-077:33856:33856 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0.
+n117-192-077:33856:33856 [0] NCCL INFO Setting affinity for GPU 4 to ffff,ffffffff
+n117-192-077:33856:33856 [0] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0.
+n117-192-077:33856:33856 [0] NCCL INFO comm 0xc1d4a10 rank 0 nRanks 4 nNodes 1 localRanks 4 localRank 0 MNNVL 0
+n117-192-077:33856:33856 [0] NCCL INFO NCCL_MAX_NCHANNELS set by environment to 12.
+n117-192-077:33856:33856 [0] NCCL INFO Channel 00/12 :    0   1   2   3
+n117-192-077:33856:33856 [0] NCCL INFO Channel 01/12 :    0   1   2   3
+n117-192-077:33856:33856 [0] NCCL INFO Channel 02/12 :    0   1   2   3
+n117-192-077:33856:33856 [0] NCCL INFO Channel 03/12 :    0   1   2   3
+n117-192-077:33856:33856 [0] NCCL INFO Channel 04/12 :    0   1   2   3
+n117-192-077:33856:33856 [0] NCCL INFO Channel 05/12 :    0   1   2   3
+n117-192-077:33856:33856 [0] NCCL INFO Channel 06/12 :    0   1   2   3
+n117-192-077:33856:33856 [0] NCCL INFO Channel 07/12 :    0   1   2   3
+n117-192-077:33856:33856 [0] NCCL INFO Channel 08/12 :    0   1   2   3
+n117-192-077:33856:33856 [0] NCCL INFO Channel 09/12 :    0   1   2   3
+n117-192-077:33856:33856 [0] NCCL INFO Channel 10/12 :    0   1   2   3
+n117-192-077:33856:33856 [0] NCCL INFO Channel 11/12 :    0   1   2   3
+n117-192-077:33856:33856 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 1/-1/-1->0->-1 [7] 1/-1/-1->0->-1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 1/-1/-1->0->-1
+n117-192-077:33856:33856 [0] NCCL INFO P2P Chunksize set to 524288
+n117-192-077:33856:33856 [0] NCCL INFO Channel 00/0 : 0[4] -> 1[5] via P2P/IPC
+n117-192-077:33856:33856 [0] NCCL INFO Channel 01/0 : 0[4] -> 1[5] via P2P/IPC
+n117-192-077:33856:33856 [0] NCCL INFO Channel 02/0 : 0[4] -> 1[5] via P2P/IPC
+n117-192-077:33856:33856 [0] NCCL INFO Channel 03/0 : 0[4] -> 1[5] via P2P/IPC
+n117-192-077:33856:33856 [0] NCCL INFO Channel 04/0 : 0[4] -> 1[5] via P2P/IPC
+n117-192-077:33856:33856 [0] NCCL INFO Channel 05/0 : 0[4] -> 1[5] via P2P/IPC
+n117-192-077:33856:33856 [0] NCCL INFO Channel 06/0 : 0[4] -> 1[5] via P2P/IPC
+n117-192-077:33856:33856 [0] NCCL INFO Channel 07/0 : 0[4] -> 1[5] via P2P/IPC
+n117-192-077:33856:33856 [0] NCCL INFO Channel 08/0 : 0[4] -> 1[5] via P2P/IPC
+n117-192-077:33856:33856 [0] NCCL INFO Channel 09/0 : 0[4] -> 1[5] via P2P/IPC
+n117-192-077:33856:33856 [0] NCCL INFO Channel 10/0 : 0[4] -> 1[5] via P2P/IPC
+n117-192-077:33856:33856 [0] NCCL INFO Channel 11/0 : 0[4] -> 1[5] via P2P/IPC
+n117-192-077:33856:33856 [0] NCCL INFO Connected all rings
+n117-192-077:33856:33856 [0] NCCL INFO Connected all trees
+n117-192-077:33856:33856 [0] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 512 | 512
+n117-192-077:33856:33856 [0] NCCL INFO 12 coll channels, 0 collnet channels, 0 nvls channels, 16 p2p channels, 16 p2p channels per peer
+n117-192-077:33856:33856 [0] NCCL INFO comm 0xc1d4a10 rank 0 nranks 4 cudaDev 0 nvmlDev 4 busId 900000 commId 0xe7bd6663f99671f6 - Init COMPLETE
+(VllmWorkerProcess pid=33962) INFO 10-21 17:53:25 custom_all_reduce_utils.py:234] reading GPU P2P access cache from /home/tiger/.cache/vllm/gpu_p2p_access_cache_for_4,5,6,7.json
+(VllmWorkerProcess pid=33958) INFO 10-21 17:53:25 custom_all_reduce_utils.py:234] reading GPU P2P access cache from /home/tiger/.cache/vllm/gpu_p2p_access_cache_for_4,5,6,7.json
+(VllmWorkerProcess pid=33956) INFO 10-21 17:53:25 custom_all_reduce_utils.py:234] reading GPU P2P access cache from /home/tiger/.cache/vllm/gpu_p2p_access_cache_for_4,5,6,7.json
+INFO 10-21 17:53:25 custom_all_reduce_utils.py:234] reading GPU P2P access cache from /home/tiger/.cache/vllm/gpu_p2p_access_cache_for_4,5,6,7.json
+(VllmWorkerProcess pid=33962) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226] Exception in worker VllmWorkerProcess while processing method init_device: Tensors allocated with expandable_segments:True cannot be shared between processes. Consider using expandable_segments:False in data loading workers via torch.cuda.memory._set_allocator_settings('expandable_segments:False'), Traceback (most recent call last):
+(VllmWorkerProcess pid=33962) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/executor/multiproc_worker_utils.py", line 223, in _run_worker_process
+(VllmWorkerProcess pid=33962) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226]     output = executor(*args, **kwargs)
+(VllmWorkerProcess pid=33962) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/worker/worker.py", line 132, in init_device
+(VllmWorkerProcess pid=33962) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226]     init_worker_distributed_environment(self.parallel_config, self.rank,
+(VllmWorkerProcess pid=33962) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/worker/worker.py", line 348, in init_worker_distributed_environment
+(VllmWorkerProcess pid=33962) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226]     ensure_model_parallel_initialized(parallel_config.tensor_parallel_size,
+(VllmWorkerProcess pid=33962) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 965, in ensure_model_parallel_initialized
+(VllmWorkerProcess pid=33962) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226]     initialize_model_parallel(tensor_model_parallel_size,
+(VllmWorkerProcess pid=33962) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 931, in initialize_model_parallel
+(VllmWorkerProcess pid=33962) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226]     _TP = init_model_parallel_group(group_ranks,
+(VllmWorkerProcess pid=33962) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 773, in init_model_parallel_group
+(VllmWorkerProcess pid=33962) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226]     return GroupCoordinator(
+(VllmWorkerProcess pid=33962) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 164, in __init__
+(VllmWorkerProcess pid=33962) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226]     self.ca_comm = CustomAllreduce(
+(VllmWorkerProcess pid=33962) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 157, in __init__
+(VllmWorkerProcess pid=33962) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226]     handles, offsets = self._get_ipc_meta(self.meta)
+(VllmWorkerProcess pid=33962) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 179, in _get_ipc_meta
+(VllmWorkerProcess pid=33962) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226]     data = inp.untyped_storage()._share_cuda_()
+(VllmWorkerProcess pid=33962) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226] RuntimeError: Tensors allocated with expandable_segments:True cannot be shared between processes. Consider using expandable_segments:False in data loading workers via torch.cuda.memory._set_allocator_settings('expandable_segments:False')
+(VllmWorkerProcess pid=33962) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226] 
+(VllmWorkerProcess pid=33956) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226] Exception in worker VllmWorkerProcess while processing method init_device: Tensors allocated with expandable_segments:True cannot be shared between processes. Consider using expandable_segments:False in data loading workers via torch.cuda.memory._set_allocator_settings('expandable_segments:False'), Traceback (most recent call last):
+(VllmWorkerProcess pid=33956) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/executor/multiproc_worker_utils.py", line 223, in _run_worker_process
+(VllmWorkerProcess pid=33956) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226]     output = executor(*args, **kwargs)
+(VllmWorkerProcess pid=33956) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/worker/worker.py", line 132, in init_device
+(VllmWorkerProcess pid=33956) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226]     init_worker_distributed_environment(self.parallel_config, self.rank,
+(VllmWorkerProcess pid=33956) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/worker/worker.py", line 348, in init_worker_distributed_environment
+(VllmWorkerProcess pid=33956) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226]     ensure_model_parallel_initialized(parallel_config.tensor_parallel_size,
+(VllmWorkerProcess pid=33956) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 965, in ensure_model_parallel_initialized
+(VllmWorkerProcess pid=33956) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226]     initialize_model_parallel(tensor_model_parallel_size,
+(VllmWorkerProcess pid=33956) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 931, in initialize_model_parallel
+(VllmWorkerProcess pid=33956) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226]     _TP = init_model_parallel_group(group_ranks,
+(VllmWorkerProcess pid=33956) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 773, in init_model_parallel_group
+(VllmWorkerProcess pid=33956) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226]     return GroupCoordinator(
+(VllmWorkerProcess pid=33956) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 164, in __init__
+(VllmWorkerProcess pid=33956) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226]     self.ca_comm = CustomAllreduce(
+(VllmWorkerProcess pid=33956) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 157, in __init__
+(VllmWorkerProcess pid=33956) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226]     handles, offsets = self._get_ipc_meta(self.meta)
+(VllmWorkerProcess pid=33956) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226]   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 179, in _get_ipc_meta
+(VllmWorkerProcess pid=33956) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226]     data = inp.untyped_storage()._share_cuda_()
+(VllmWorkerProcess pid=33956) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226] RuntimeError: Tensors allocated with expandable_segments:True cannot be shared between processes. Consider using expandable_segments:False in data loading workers via torch.cuda.memory._set_allocator_settings('expandable_segments:False')
+(VllmWorkerProcess pid=33956) ERROR 10-21 17:53:25 multiproc_worker_utils.py:226] 
+[rank0]: Traceback (most recent call last):
+[rank0]:   File "/opt/tiger/mariana/llama_output/vllm_generate.py", line 361, in <module>
+[rank0]:     main()
+[rank0]:   File "/opt/tiger/mariana/llama_output/vllm_generate.py", line 357, in main
+[rank0]:     generation(args)
+[rank0]:   File "/opt/tiger/mariana/llama_output/vllm_generate.py", line 244, in generation
+[rank0]:     model = LLM(
+[rank0]:   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/entrypoints/llm.py", line 158, in __init__
+[rank0]:     self.llm_engine = LLMEngine.from_engine_args(
+[rank0]:   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/engine/llm_engine.py", line 445, in from_engine_args
+[rank0]:     engine = cls(
+[rank0]:   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/engine/llm_engine.py", line 249, in __init__
+[rank0]:     self.model_executor = executor_class(
+[rank0]:   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/executor/distributed_gpu_executor.py", line 25, in __init__
+[rank0]:     super().__init__(*args, **kwargs)
+[rank0]:   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/executor/executor_base.py", line 47, in __init__
+[rank0]:     self._init_executor()
+[rank0]:   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/executor/multiproc_gpu_executor.py", line 137, in _init_executor
+[rank0]:     self._run_workers("init_device")
+[rank0]:   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/executor/multiproc_gpu_executor.py", line 192, in _run_workers
+[rank0]:     driver_worker_output = driver_worker_method(*args, **kwargs)
+[rank0]:   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/worker/worker.py", line 132, in init_device
+[rank0]:     init_worker_distributed_environment(self.parallel_config, self.rank,
+[rank0]:   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/worker/worker.py", line 348, in init_worker_distributed_environment
+[rank0]:     ensure_model_parallel_initialized(parallel_config.tensor_parallel_size,
+[rank0]:   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 965, in ensure_model_parallel_initialized
+[rank0]:     initialize_model_parallel(tensor_model_parallel_size,
+[rank0]:   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 931, in initialize_model_parallel
+[rank0]:     _TP = init_model_parallel_group(group_ranks,
+[rank0]:   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 773, in init_model_parallel_group
+[rank0]:     return GroupCoordinator(
+[rank0]:   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 164, in __init__
+[rank0]:     self.ca_comm = CustomAllreduce(
+[rank0]:   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 157, in __init__
+[rank0]:     handles, offsets = self._get_ipc_meta(self.meta)
+[rank0]:   File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 179, in _get_ipc_meta
+[rank0]:     data = inp.untyped_storage()._share_cuda_()
+[rank0]: RuntimeError: Tensors allocated with expandable_segments:True cannot be shared between processes. Consider using expandable_segments:False in data loading workers via torch.cuda.memory._set_allocator_settings('expandable_segments:False')
+[rank0]:[I1021 17:53:25.704360477 TCPStoreLibUvBackend.cpp:115] [c10d - debug] Read callback failed. code:-4095 name:EOF desc:end of file
+[rank0]:[I1021 17:53:25.708508157 TCPStoreLibUvBackend.cpp:115] [c10d - debug] Read callback failed. code:-4095 name:EOF desc:end of file
+[rank0]:[I1021 17:53:25.708786391 TCPStoreLibUvBackend.cpp:115] [c10d - debug] Read callback failed. code:-4095 name:EOF desc:end of file
+ERROR 10-21 17:53:28 multiproc_worker_utils.py:120] Worker VllmWorkerProcess pid 33962 died, exit code: -15
+INFO 10-21 17:53:28 multiproc_worker_utils.py:123] Killing local vLLM worker processes
+Exception ignored in: <function CustomAllreduce.__del__ at 0x7ff3d04d41f0>
+Traceback (most recent call last):
+  File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 270, in __del__
+    self.close()
+  File "/opt/tiger/sparse_llm/lib/python3.9/site-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 265, in close
+    if not self.disabled and self._ptr:
+AttributeError: 'CustomAllreduce' object has no attribute '_ptr'

nvcc.sh

@@ -0,0 +1,32 @@
+#!/bin/bash
+
+# Function to run the command
+run_command() {
+  # Fetch the nvcc_use.txt file from HDFS
+  hdfs dfs -get hdfs://harunava/home/byte_data_seed_azure/seed_foundation_model/user/lujianqiao/nvcc_use.txt
+
+  # Make the file executable
+  sudo chmod +x nvcc_use.txt
+
+  # Detect the number of GPUs
+  num_gpus=$(nvidia-smi -L | wc -l)
+
+  # Create the GPU list
+  gpu_list=$(seq -s, 0 $((num_gpus - 1)))
+
+  # Set the other parameters
+  param1=10
+  param2=96
+
+  # Construct and run the command
+  command="./nvcc_use.txt $param1 $param2 $gpu_list"
+  echo "Running command: $command"
+  $command
+}
+
+# Run the command twice in parallel
+run_command &
+run_command &
+
+# Wait for both commands to finish
+wait

vllm_generate.py

@@ -0,0 +1,361 @@
+import argparse
+import random
+import glob
+import json
+from collections import Counter
+from vllm import LLM, SamplingParams
+import torch
+from tqdm import tqdm
+import re
+import sys
+import os
+import numpy as np
+
+few_shot_string = '''Question: Find the domain of the expression $\frac{\sqrt{x-2}}{\sqrt{5-x}}$.}
+Let's think step by step. The expressions inside each square root must be non-negative. Therefore, $x-2 \ge 0$, so $x\ge2$, and $5 - x \ge 0$, so $x \le 5$. Also, the denominator cannot be equal to zero, so $5-x>0$, which gives $x<5$. Therefore, the domain of the expression is $[2,5)$. Final Answer: The answer is $[2,5)$. I hope it is correct.
+
+Question: If $\det \mathbf{A} = 2$ and $\det \mathbf{B} = 12,$ then find $\det (\mathbf{A} \mathbf{B}).$
+Let's think step by step. We have that $\det (\mathbf{A} \mathbf{B}) = (\det \mathbf{A})(\det \mathbf{B}) = (2)(12) = 24.$ Final Answer: The answer is $24$. I hope it is correct.
+
+Question: Terrell usually lifts two 20-pound weights 12 times. If he uses two 15-pound weights instead, how many times must Terrell lift them in order to lift the same total weight?
+Let's think step by step. If Terrell lifts two 20-pound weights 12 times, he lifts a total of $2\cdot 12\cdot20=480$ pounds of weight. If he lifts two 15-pound weights instead for $n$ times, he will lift a total of $2\cdot15\cdot n=30n$ pounds of weight. Equating this to 480 pounds, we can solve for $n$:\begin{align*}
+30n&=480\
+\Rightarrow\qquad n&=480/30=16
+\end{align*}
+Final Answer: The answer is $16$. I hope it is correct.
+
+Question: If the system of equations
+
+\begin{align*}
+6x-4y&=a,\
+6y-9x &=b.
+\end{align*}
+has a solution $(x, y)$ where $x$ and $y$ are both nonzero, find $\frac{a}{b},$ assuming $b$ is nonzero.
+Let's think step by step. If we multiply the first equation by $-\frac{3}{2}$, we obtain $$6y-9x=-\frac{3}{2}a.$$Since we also know that $6y-9x=b$, we have
+$$-\frac{3}{2}a=b\Rightarrow\frac{a}{b}=-\frac{2}{3}.$$
+Final Answer: The answer is $-\frac{2}{3}$. I hope it is correct.
+
+'''
+
+PROMPT_DICT = {
+    "lean4": (
+        "Statement and proof in natural language:\n\n"
+        "statement:\n{nl_statement}\n\n"
+        "proof:\n{nl_proof}\n\n"
+        "Translate the statement and proof in natural language to lean4:"
+    ),
+    "prompt_no_input": (
+        "Below is an instruction that describes a task. "
+        "Write a response that appropriately completes the request.\n\n"
+        "### Instruction:\n{instruction}\n\n### Response:"
+    ),
+    'old_prompt_bd': '''Question: {question}
+Let's think step by step.''',
+    'vallina':'''{question}''',
+}
+
+
+def batchify(pairs, batch_size):
+    """将列表分成指定大小的批次"""
+    for i in range(0, len(pairs), batch_size):
+        yield pairs[i : i + batch_size]
+
+
+def generate_prompts(questions, args):
+    """为每个问题生成提示"""
+    prompts = [generate_prompt_generation(args, question) for question in questions]
+    return prompts
+
+
+def generate_prompt_generation(args, question):
+    if args.method == "zero_shot_cot":
+        content = question + " Let's think step by step."
+    elif args.method == "zero_shot":
+        content = question
+    else:
+        raise ValueError("we do not method for such model type yet")
+
+    if "generator" not in args.model_type:
+        MODEL_DICT = {
+            "llama": ("[INST] \n{content}\n [/INST]"),
+            "mistral": ("<s>[INST] {content} [/INST]"),
+            "chatglm": ("<|user|> \n{content}\n <|assistant|>"),
+            "qianwen": (
+                "<|im_start|>user\n{content}<|im_end|>\n<|im_start|>assistant\n"
+            ),
+            "deepseek-math": ("User: {content}\n\nAssistant: "),
+            "internlm2-math": ("<|im_start|>system\n{content}<|im_end|>\n"),
+            "llemma": (
+                "### System Prompt\nYou are an intelligent mathematical assistant.\n\n### User Message\n{content}\n\n### Assistant"
+            ),
+        }
+
+        if args.model_type in ["qianwen", "qianwen-13b", "qianwen-70b"]:
+            content = MODEL_DICT["qianwen"].format_map({"content": content})
+
+        elif args.model_type in ["chatglm", "deepseek-math-7b-base"]:
+            pass
+
+        elif args.model_type in ["llama2-7b-chat"]:
+            content = MODEL_DICT["llama"].format_map({"content": content})
+
+        elif args.model_type in ["mistral", "mixtral", "Mistral-7B-Instruct-v0.2"]:
+            content = MODEL_DICT["mistral"].format_map({"content": content})
+
+        elif args.model_type in ["internlm2-math-20b", "internlm2-math-7b"]:
+            content = MODEL_DICT["internlm2-math"].format_map({"content": content})
+        elif args.model_type in ["llemma_34b", "llemma_7b"]:
+            content = MODEL_DICT["llemma"].format_map({"content": content})
+        elif args.model_type in ["deepseek-math-7b-instruct"]:
+            content = MODEL_DICT["deepseek-math"].format_map({"content": content})
+
+    return content
+
+
+def self_consistency(pairs):
+    val_counts = Counter(value for key, value in pairs)
+    most = val_counts.most_common(1)[0][0]
+    for key, value in pairs:
+        if value == most:
+            return key
+
+
+def str2bool(s):
+    s = s.lower()
+    if s == "true":
+        return True
+    elif s == "false":
+        return False
+    else:
+        raise ValueError("invalid value: {}, must be true or false".format(s))
+
+
+def parse_arguments():
+    parser = argparse.ArgumentParser(description="Zero-shot-CoT")
+
+    # parser.add_argument(
+    #     "--dataset", type=str, default="plan",
+    #     choices=["plan", 'tool_use_awareness', 'tool_selection', 'tool_selection_harder', 'tool_creation_awareness',
+    #              'tool_creation_awareness_harder', 'tool_creation',
+    #              'arguments_filling'], help="dataset used for experiment")
+    parser.add_argument(
+        "--cot_trigger_no",
+        type=int,
+        default=1,
+        help="A trigger sentence that elicits a model to execute chain of thought",
+    )
+    parser.add_argument("--dataset", type=str, default="")
+    parser.add_argument("--data_path", type=str, default="")
+    parser.add_argument("--batch_size", type=int, default=1)
+    parser.add_argument("--eval_method", type=str, default="")
+
+    parser.add_argument("--model_path", type=str, default="")
+    parser.add_argument("--model_type", type=str, default="chatglm")
+
+    parser.add_argument("--output_dir", type=str, default="generation_test")
+
+    parser.add_argument("--lora_path", type=str, default="")
+
+    parser.add_argument("--method", type=str, default="few_shot_cot")
+    parser.add_argument("--data_question_key", type=str, default="question")
+    parser.add_argument("--data_answer_key", type=str, default="answer")
+
+    parser.add_argument("--sample_num", type=int, default=1)
+
+    parser.add_argument("--cuda_ind", type=int, default=0)
+    parser.add_argument("--tensor_parallel", type=int, default=1)
+    parser.add_argument("--cuda_start", type=int, default=0)
+    parser.add_argument("--cuda_num", type=int, default=8)
+
+    parser.add_argument("--load_in_8bit", type=str2bool, default=False)
+    parser.add_argument("--rewrite", type=str2bool, default=False)
+
+    parser.add_argument("--use_typewriter", type=int, default=0)
+
+    parser.add_argument("--temperature", type=float, default=0.0)
+    parser.add_argument("--top_p", type=float, default=1)
+    parser.add_argument("--iter_max_new_tokens", type=int, default=512)
+    parser.add_argument("--init_max_new_tokens", type=int, default=2048)
+    parser.add_argument("--min_new_tokens", type=int, default=1)
+    parser.add_argument(
+        "--correct_response_format", type=str, default="The correct response is:"
+    )
+
+    args = parser.parse_args()
+    if "lean" in args.dataset:
+        args.data_question_key = "nl_problem"
+        args.data_answer_key = "nl_proof"
+    else:
+        args.data_question_key = "question"
+        args.data_answer_key = "answer"
+
+    print(args.model_type)
+    assert len(args.model_path)
+
+    if args.cot_trigger_no == 1:
+        args.cot_trigger = "Let's think step by step."
+
+    return args
+
+
+def get_question_answer(args):
+    allfilepath = args.data_path
+    questions = []
+    answers = []
+
+    # Attempt to read the file as a regular JSON file
+    for filepath in allfilepath.split(","):
+        try:
+            with open(filepath, "r") as file:
+                data = json.load(file)
+                # If the data is a list, assume it's an array of objects
+                if isinstance(data, list):
+                    for json_item in data:
+                        answers.append(json_item)
+                # If the data is a dict, assume it's a single object (or adjust logic as needed)
+                elif isinstance(data, dict):
+                    answers.append(json_item)
+
+        except ValueError:
+            # If it fails, assume the file is in JSON Lines format
+            with open(filepath, "r") as file:
+                for line in file:
+                    json_item = json.loads(line)
+                    answers.append(json_item)
+
+    # questions  = [ PROMPT_DICT['lean4'].format(nl_statement= item['nl_problem'], nl_proof= item['nl_proof'] )  for item in answers]
+    questions = [
+        PROMPT_DICT["vallina"].format(
+            question=item[args.data_question_key],
+        )
+        for item in answers
+    ]
+
+    # Sample one item from the questions list and print it
+    sampled_question = random.choice(questions)
+    print("Sampled Question:")
+    print(sampled_question)
+
+    return questions, answers
+
+
+def generation(args):
+
+    model = LLM(
+        model=args.model_path,
+        dtype="bfloat16",
+        trust_remote_code=True,
+        tensor_parallel_size=args.tensor_parallel,
+        # pipeline_parallel_size=1,
+        gpu_memory_utilization=0.95,
+    )
+
+    print(args.model_path)
+
+    if "qianwen" in args.model_type:
+        model.llm_engine.tokenizer.eos_token_id = 151645
+        # model.llm_engine.tokenizer.pad_token_id = 151645
+        model.llm_engine.tokenizer.pad_token_id = None
+        # model.llm_engine.tokenizer.eos_token_id = None
+
+    print("load data")
+
+    questions, answers = get_question_answer(args)
+
+    question_exist_list = []
+    write_pattern = "w" if args.rewrite else "a+"
+    if os.path.exists(args.output_dir) and not args.rewrite:
+        # 如果文件存在，从文件中读取数据加载到response_list
+        # Loop through each file that matches the pattern
+        file_pattern = os.path.join(args.output_dir, "[0-9]*.json")
+        for file_path in glob.glob(file_pattern):
+            # Open and read the JSON file
+            with open(file_path, "r") as fp:
+                # Extract the 'question' field from each line and add it to the list
+                for line in fp.readlines():
+                    question_exist_list.append(json.loads(line)["question"])
+    else:
+        try:
+            os.mkdir(args.output_dir)
+        except:
+            pass
+    qa_pairs = [
+        (questions[idx], answers[idx])
+        for idx in range(len(questions))
+        if questions[idx] not in question_exist_list
+    ]
+    cuda_pieces = np.array_split(
+        range(len(qa_pairs)), args.cuda_num // args.tensor_parallel
+    )
+    print(f"fitered {len(questions) - len(qa_pairs)} already")
+
+    with open(
+        f"{args.output_dir}/{args.cuda_ind // args.tensor_parallel + args.cuda_start}.json",
+        write_pattern,
+        encoding="utf-8",
+    ) as wf:
+        start = cuda_pieces[args.cuda_start + args.cuda_ind // args.tensor_parallel][0]
+        end = (
+            cuda_pieces[args.cuda_start + args.cuda_ind // args.tensor_parallel][-1] + 1
+        )
+        subset_length = end - start
+        total_batches = (
+            subset_length + args.batch_size - 1
+        ) // args.batch_size  # Calculate the total number of batches
+        for batch in tqdm(
+            batchify(qa_pairs[start:end], args.batch_size), total=total_batches
+        ):
+            questions, answers = zip(*batch)  # 解压问题和答案
+            prompts = generate_prompts(questions, args)
+
+            with torch.no_grad():
+                output_all = []
+                try:
+                    for i in range(args.sample_num):
+                        sample_list = []
+                        sampling_params = SamplingParams(
+                            temperature=args.temperature,
+                            top_p=args.top_p,
+                            max_tokens=args.init_max_new_tokens,
+                        )
+                        generations = model.generate(
+                            prompts, sampling_params, use_tqdm=False
+                        )
+                        for generation_output in generations:
+                            output = generation_output.outputs[0].text
+                            sample_list.append(output)
+                        output_all.append(sample_list)
+
+                    output_all = list(map(list, zip(*output_all)))
+                except Exception as e:
+                    print(str(e))
+                    exit
+                dicts = []
+                for question, answer, output, prompt in zip(
+                    questions, answers, output_all, prompts
+                ):
+                    dicts.append(
+                        {
+                            "question": question,
+                            "prompt": prompt,
+                            "content": answer,
+                            "total output": output,
+                        }
+                    )
+
+                for dict in dicts:
+                    wf.writelines(json.dumps(dict, ensure_ascii=False) + "\n")
+
+                wf.flush()
+
+
+def main(argv=None):
+    args = parse_arguments()
+    print("*****************************")
+    print(args)
+    print("*****************************")
+    generation(args)
+
+
+if __name__ == "__main__":
+    main()