full pipeline

app.py

@@ -1,4 +1,47 @@
 import streamlit as st
+import re
+from llmengine import Model, Completion
 
-x = st.slider("Select a value")
-st.write(x, "squared is", x*x)
+
+def get_model_output(model_num, model, prompt):
+	output = None
+	try:
+		resp = Completion.create(prompt = prompt, model = model, max_new_tokens = 300, temperature = 0.0, timeout = 120)
+		output = resp.output.text
+	except Exception as e:
+		st.write(f"Model {model_num} failed due to: {e}")
+	return output
+
+
+def parse_model_output(model_num, output):
+	if output is None:
+		model_score = None
+	else:
+		score_match = re.match(r"DIGNITY INDEX: (\d)", output)
+		if score_match:
+			model_score = int(score_match.group(1))
+		else:
+			model_score = None
+	if model_score:
+		st.write(f"Predicted score: {model_score}")
+	else:
+		st.write(f"Model {model_num} could not score this message. Full model output here:\n{output}")
+
+
+st.title("Dignity Index Scoring")
+st.write("Interface to score passages based on the Dignity Index")
+user_input = st.text_input("Passage you want scored here")
+
+with open("./coding_instructions_long.txt", "r") as f:
+	coding_inst_lines = f.readlines()
+	coding_inst = " ".join(line.strip() for line in coding_inst_lines)
+question = "\n\nWhere does the following passage fall on the aforementioned scale? Explain your reasoning first, then give your answer in the form DIGNITY INDEX: [SCORE], where SCORE is an integer between 1 and 8. Passage:\n"
+full_input = coding_inst + question + user_input
+
+model1 = "mistral-7b-instruct.gold-medium-n10-d00-lr25.231124-222719"
+model2 = "mistral-7b-instruct.mistral-gold-raw-simple.231030-070416"
+output1 = get_model_output(1, model1, full_input)
+output2 = get_model_output(2, model2, full_input)
+
+parse_model_output(1, output1)
+parse_model_output(2, output2)

coding_instructions_long.txt

@@ -0,0 +1,41 @@
+The Dignity Index is an eight-point scale that measures what we do when we disagree. It ranges from one — which sees no dignity at all in the other side — to eight, which sees the dignity in everyone. Each point on the scale reflects a particular mindset, and each mindset is associated with certain beliefs and behaviors. As a rule of thumb, if I treat you with dignity, it means that I can see myself in you; if I treat you with contempt, it means I see myself above you. The higher our actions are on the Dignity Index, the more our words and behaviors ease division, prevent violence and solve problems. The more we move down the index toward contempt, the more we increase divisions, encourage violence, and create problems. Dignity brings us together. Contempt tears us apart.
+
+The following, for each score on the Index, is a description of what the score represents and what someone in the corresponding mindset might think. In addition, there are comparisons between every successive pair of scores.
+
+LEVEL ONE: One escalates from violent words to violent actions based on an unquestioned belief in the irredeemable evil of the other side and the conviction that violence against them is heroic. One is a combination of believing that the other side is evil and in the midst of destroying us – and inciting or taking part in violence against the other side. Name-calling and negative labels might include: source of all evil, scum, subhuman, poison.
+Those at level one think: It’s our moral duty to destroy them. They’re not even human. They’re the source of all evil. They’re destroying everything we value. They don’t deserve to live. We have to kill them now before they kill us.
+
+LEVEL TWO: Two accuses the other side not just of doing bad or being bad, but promoting evil. Two obsesses over the evil of the other side and tells alarming stories about the things they do and the danger they pose. Name-calling and negative labels might include: evil, traitors.
+Those at level two think: Those people are evil and a danger to everything we value. They’re going to ruin us if we let them. It’s us or them. They’re an existential threat.
+
+What's the difference between a ONE and a TWO? At two, we believe that the other side is evil and they’re an existential threat to us, but at the two mindset, it’s still rhetoric. We’re just talking. We’re not yet committing acts of violence or calling for them or inciting them. At one, the language used to describe the other side hits the lowest point, and not only condones violence, but requires it.
+
+LEVEL THREE: Three attacks the other side’s moral character, not just their capabilities or competence. Name-calling and negative labels might include: liars, cheaters, thieves.
+Those at level three think: We’re the good people and they’re the bad people. We’re responsible for the good things and they’re responsible for the bad things. They hate us and want to hurt us. It’s us vs. them. We win only if they lose. Contempt for them becomes a key part of our personal identity and a requirement of group membership. We tell stories about the other side that make us angry, and then get addicted to the anger.
+
+What's the difference between a TWO and a THREE? When we’re in the three mindset, we believe that the other side is made up of bad people who want to hurt us. When we enter a two mindset, we’re saying the other side is filled with evil people who are trying to ruin us. We’re now facing an existential threat. Us vs. them becomes us or them. The tactics of the three and two mindsets are the same: telling stories about the other side claiming grievance, stoking resentment. But the theme of the story changes from they’re bad people trying to hurt us to they’re evil people trying to ruin us.
+
+LEVEL FOUR: Four mocks and attacks the other side’s commitment, competence, performance. Statements may include generally dignified language that lauds their own side in a way that suggests the other side is lesser or does not support their values. There can be statements of expressing caution, suspicion, or lack of confidence that the other side will do the right thing. A deliberate mischaracterization of opponent’s position to make it appear unappealing is at best a four, and could score lower on the scale depending on what’s implied. Name-calling and negative labels might include: "those people," reckless, careless, incompetent, failures.
+Those at level four think: We’re better than those people. They’re different. They don’t really belong. They don’t share our values. They’re not one of us. They’re just in the way. We shouldn’t trust them.
+
+What's the difference between a THREE and a FOUR? In the four mindset, we believe that we’re different and better than the other side. We don’t want to meet with them or cooperate with them. We will challenge their capabilities, their competence, their commitment. But we are not drawing the striking distinction that we do in the three mindset where we pit ourselves against the other side in an us vs. them confrontation, saying that they are the bad people, and we are the good people. When we criticize someone from a four mindset, it is a criticism of someone’s competence or capability or diligence, or perhaps their cultural values. When we criticize someone from a three mindset, it is a character attack, and we are criticizing their moral standing and declaring them bad people. Four tends to see the other as "us and them." Three is "us vs. them." Two is "us or them."
+
+LEVEL FIVE: Five listens to the other side and is willing to have a working relationship, will put forward ideas and proposals without contempt, explain their views, their goals, their reasoning, but is not actively engaging the other side to find common interests and values. From Five and up, criticism is based on decisions, actions, and outcomes, not on motives or moral character. At this level, there is no name-calling or negative labels. A simple statement of policies, with no contempt shown for the other side, is the first step of dignity.
+Those at level five think: I believe everyone has a right to be here and a right to be heard.
+
+What's the difference between a FOUR and a FIVE? Five is the mindset of speaking openly and respectfully and listening intently. In five, when we challenge the other side, we question policies and positions and outcomes — not capabilities or competence. The four mindset expresses contempt by dismissing the other side as not serious — questioning their competence, their capabilities, and using name-calling such as clowns, losers, bunglers, outsiders and questioning their priorities or who they belong to.
+
+LEVEL SIX: Six sees it as a welcome duty to work with the other side to find common ground and act on it. Finding common interests is key for people at six, but they are not quite ready to examine what the two sides don’t share or how the other side came to believe what they believe. The intent or openness to engaging with the other side to solve problems is explicitly stated or clearly implied in some way.
+Those at level six think: I have strong beliefs and a strong sense of belonging to my group, but I’m also curious about the other side, and I want to engage them in discussion, compare ideas and proposals, and talk about what we believe, because I think we have a duty to find common values and interests and use them as a basis for cooperation. It gives me a feeling of pride to accomplish something with the other side.
+
+What's the difference between a FIVE and a SIX? When we’re at six, we feel it’s our duty to engage the other side, identify common ground, and build on common interests to solve problems and seize opportunities. It’s a sense of duty, but it also brings pride when we’re able to get it done. We feel like we’re doing something elevating for the country. Six is different from five, because five will also avoid all marks and signs of contempt, and five will explain their positions clearly and openly to the other side, a clear mark of treating someone with dignity, and five will listen to the other side, certainly never refuse to listen, but at five we don’t feel the obligation to engage the other side for the sake of finding common interests and cooperating on projects. Five puts aside contempt, and gives the other side the dignity of being present for a reading of position papers, to put it that way, but at five we’re not particularly curious about the other side, and we don't want to sit down and engage in policy talks in an effort to solve big problems.
+
+LEVEL SEVEN: Seven will work with the other side to openly discuss their deepest disagreements to see what they can learn and what problems they can solve. Friendship and good will can flourish in the face of disagreement. A level seven statement is vulnerable and open to accepting some responsibility for difficulties related to a highly contentious topic between their group and the opposition.
+Those at level seven think: I have strong convictions and I’m loyal to my group, but I fully engage with people from other groups, discussing even the values and interests we don’t share, so we can learn from our disagreements. I want to find out how they came to believe what they believe. I’m not afraid of being criticized, losing an argument or being told I’m wrong. I see how we divide ourselves by believing we’re always right and the other side is always wrong, we’re good and they’re bad. That’s false. It’s polarizing, and I won’t be part of it. I’m willing to admit mistakes and change my mind if people can show me something I haven’t seen.
+
+What's the difference between a SIX and a SEVEN? When we’re in a seven mindset, we are genuinely excited to engage the other side in a discussion of our deepest disagreements, because we are not afraid to be wrong, to lose an argument, or to trigger anger in the other side. On the contrary, we are excited for what we might see that we wouldn’t see if we never talked to anyone who disagrees. Seven is different from six, because when we’re at a six, we find it painful to be wrong, to be disagreed with, to have our deep convictions criticized — so we will limit conversations with the other side to efforts to find agreement — and if we run into areas of significant disagreement, we will tighten up or shut down.
+
+LEVEL EIGHT: Eight is extraordinary, particularly in political speech, because it involves standing with the demonized and defending their dignity — in the face of the people who are demonizing them. It involves a higher degree of difficulty than other speech because the person is risking exclusion from their own group.
+Those at level eight think: I don’t hate anyone, and I refuse to condemn anyone. If I had lived their life, I might have done what they did. I love and cherish my own group, but I can connect with most anyone. I see myself in others. I will talk and work with anyone to find solutions. I don’t insist on my approach. I don’t need to be right. I don’t care who gets the credit. I just want decisions and solutions that protect the dignity of everyone. I don’t hate or condemn anyone, and it doesn’t hurt me if others hate me. I believe everyone has inherent worth so I treat everyone with dignity no matter what.
+
+What's the difference between a SEVEN and an EIGHT: People in the eight mindset love and cherish my own people, but I can connect with all people. So, they have no defensiveness toward others or their ideas. They don’t have any emotional attachment to being right. They will talk or work with anyone. They don’t insist on a particular approach, and they don’t care who gets credit. Their only non-negotiable is that they want to create solutions that defend the dignity of all people. Their most distinctive quality is their ability to absorb the pain of being hated without passing it on. They are different from seven because at seven we still principally identify with one group over others, we clearly prefer our own group, we instinctively think we are right and the other group is wrong, we assess other views by how much they differ from our own, and we want our group to win. When we’re at eight, we don’t really have an "other." Our tribe is humanity, or even creation. In short, seven wants credit and wants their side to win; eight doesn’t care who gets credit and hopes everyone wins. Seven is able to take the pain of being wrong; eight can handle the pain of being hated.

requirements.txt

@@ -0,0 +1,3 @@
+streamlit
+numpy
+scale-llm-engine