jansowa commited on
Commit
4aa0174
·
verified ·
1 Parent(s): 90ff805

Fix bug with unknown param number

Browse files
Files changed (1) hide show
  1. app.py +23 -10
app.py CHANGED
@@ -71,12 +71,10 @@ st.markdown("""
71
  .center-text {
72
  text-align: center;
73
  }
74
-
75
  a:link {color:#FDA428;} /* unvisited link */
76
  a:hover {color:#FDA428;} /* Mouse over link */
77
  a:visited {color:#FDA428;} /* visited link */
78
  a:active {color:#FDA428;} /* selected link */
79
-
80
  </style>
81
  """, unsafe_allow_html=True)
82
 
@@ -143,7 +141,10 @@ with tab1:
143
  # Prepare data
144
  data = load_data('data.json')
145
 
146
- data['Params'] = data['Params'].str.replace('B', '').astype(float)
 
 
 
147
  data = data.sort_values(by=AVERAGE_COLUMN_NAME, ascending=False)
148
 
149
  # Closing filters in a expander
@@ -152,8 +153,25 @@ with tab1:
152
  col_filter_params, col_filter_average, col_filter_sentiment, col_filter_understanding, col_filter_phraseology = st.columns(5, gap='medium')
153
 
154
  with col_filter_params:
155
- params_slider = st.slider("Models Size [B]", min_value=0.0, max_value=float(data['Params'].max()), value=(0.0, float(data['Params'].max())), step=0.1, format="%.1f")
156
- data = data[(data['Params'] >= params_slider[0]) & (data['Params'] <= params_slider[1])]
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
157
 
158
  with col_filter_average:
159
  average_slider = st.slider("Average score", step=0.1, min_value=0.0, max_value=5.0, value=(0.0, 5.0))
@@ -235,24 +253,20 @@ with tab2:
235
  st.markdown("""
236
  ### <span style='text-decoration: #FDA428 wavy underline;'>**Cause of Creation**</span>
237
  1. **Need**: Models face significant challenges when dealing with understanding complex, context-reliant texts that involve meanings implied beyond the literal content of a statement. Such cases include sarcasm, implicatures, and phraseological compounds.
238
-
239
  Traditional sentiment classifiers typically rely on word-based features (e.g., identifying positive or negative words) to assess sentiment. However, with sarcasm, the literal meaning of words often contradicts the intended sentiment, making it difficult for models to accurately gauge tone. Sarcasm's context-dependence further complicates matters, as these classifiers typically lack the ability to grasp nuanced cues in context, especially when sarcasm is subtle.
240
  Similarly, classifiers struggle with implicatures, where the underlying intent is implied rather than explicitly stated. Here, models fail to capture the full sentiment because they rely heavily on surface-level words, missing the non-literal meaning that often drives the sentiment.
241
  Phraseological compounds add another layer of difficulty. These are fixed or semi-fixed expressions whose meanings cannot be directly inferred from the individual words. Language models, trained on word-level patterns, often misinterpret these expressions because they fail to recognize the idiomatic or non-literal meaning, leading to inaccurate sentiment analysis.
242
  In addition to sentiment analysis, we decided to include the understanding of more complex texts in the benchmark, which was measured by the ability to uncover the intended meaning.
243
-
244
  ### <span style='text-decoration: #FDA428 wavy underline;'>**Dataset Information**</span>
245
  The dataset contains 200 examples, all written in Polish. Each example consists of the following:
246
  - **Main Text**: This is a statement (often an opinion) on any topic that includes a certain type of implicature, often several simultaneously, such as sarcasm or phraseological compounds.
247
  - **Reference Sentiment**: The sentiment associated with the main text. We use three categories: negative, neutral, and positive. Ambiguous examples were labeled as "neutral" to exclude them from sentiment classification testing.
248
  - **Reference phraseological compounds**: A list of phraseological compounds found in the main text.
249
  - **Reference Explanation**: An explanation of the underlying intentions that the author of the main text might have had.
250
-
251
  ### <span style='text-decoration: #FDA428 wavy underline;'>**Evaluation Procedure**</span>
252
  We distinguish between two models in the evaluation process:
253
  - **Evaluated Model**: The model that performs specific tasks, is then assessed based on its performance, and added to a ranking.
254
  - **Judge Metamodel**: One of the currently strongest, most versatile LLMs.
255
-
256
  ### <span style='text-decoration: #FDA428 wavy underline;'>**GENERATING RESPONSES FROM THE EVALUATED MODEL**</span>
257
  1. For each text in the dataset, the evaluated model was required to list the following in three points:
258
  - The sentiment (only positive/negative).
@@ -268,7 +282,6 @@ We distinguish between two models in the evaluation process:
268
  - **Assistant Prompt**: A human-written example answer for the second example text.
269
  - **User Prompt**: The target text, based on which the evaluated model will be assessed.
270
  3. The decision to split the examples into user prompts and assistant prompts was made due to the better results achieved by the vast majority of models. The two examples were selected based on diversity: one has a negative sentiment and several phraseological compounds, while the other is positive and lacks phraseological compounds.
271
-
272
  ### <span style='text-decoration: #FDA428 wavy underline;'>**GENERATING METAMODEL EVALUATIONS**</span>
273
  1. The purpose of the metamodel is to return the following evaluations:
274
  - **Understanding of the Text**: A comparison of the evaluated model's response description to the reference explanation.
 
71
  .center-text {
72
  text-align: center;
73
  }
 
74
  a:link {color:#FDA428;} /* unvisited link */
75
  a:hover {color:#FDA428;} /* Mouse over link */
76
  a:visited {color:#FDA428;} /* visited link */
77
  a:active {color:#FDA428;} /* selected link */
 
78
  </style>
79
  """, unsafe_allow_html=True)
80
 
 
141
  # Prepare data
142
  data = load_data('data.json')
143
 
144
+ data['Params'] = pd.to_numeric(
145
+ data['Params'].str.replace('B', ''),
146
+ errors='coerce'
147
+ )
148
  data = data.sort_values(by=AVERAGE_COLUMN_NAME, ascending=False)
149
 
150
  # Closing filters in a expander
 
153
  col_filter_params, col_filter_average, col_filter_sentiment, col_filter_understanding, col_filter_phraseology = st.columns(5, gap='medium')
154
 
155
  with col_filter_params:
156
+ max_params = data['Params'].max(skipna=True)
157
+ if pd.isna(max_params):
158
+ max_params = 0.0
159
+
160
+ params_slider = st.slider(
161
+ "Models Size [B]",
162
+ min_value=0.0,
163
+ max_value=float(max_params),
164
+ value=(0.0, float(max_params)),
165
+ step=0.1,
166
+ format="%.1f"
167
+ )
168
+ data = data[
169
+ data['Params'].isna() |
170
+ (
171
+ (data['Params'] >= params_slider[0]) &
172
+ (data['Params'] <= params_slider[1])
173
+ )
174
+ ]
175
 
176
  with col_filter_average:
177
  average_slider = st.slider("Average score", step=0.1, min_value=0.0, max_value=5.0, value=(0.0, 5.0))
 
253
  st.markdown("""
254
  ### <span style='text-decoration: #FDA428 wavy underline;'>**Cause of Creation**</span>
255
  1. **Need**: Models face significant challenges when dealing with understanding complex, context-reliant texts that involve meanings implied beyond the literal content of a statement. Such cases include sarcasm, implicatures, and phraseological compounds.
 
256
  Traditional sentiment classifiers typically rely on word-based features (e.g., identifying positive or negative words) to assess sentiment. However, with sarcasm, the literal meaning of words often contradicts the intended sentiment, making it difficult for models to accurately gauge tone. Sarcasm's context-dependence further complicates matters, as these classifiers typically lack the ability to grasp nuanced cues in context, especially when sarcasm is subtle.
257
  Similarly, classifiers struggle with implicatures, where the underlying intent is implied rather than explicitly stated. Here, models fail to capture the full sentiment because they rely heavily on surface-level words, missing the non-literal meaning that often drives the sentiment.
258
  Phraseological compounds add another layer of difficulty. These are fixed or semi-fixed expressions whose meanings cannot be directly inferred from the individual words. Language models, trained on word-level patterns, often misinterpret these expressions because they fail to recognize the idiomatic or non-literal meaning, leading to inaccurate sentiment analysis.
259
  In addition to sentiment analysis, we decided to include the understanding of more complex texts in the benchmark, which was measured by the ability to uncover the intended meaning.
 
260
  ### <span style='text-decoration: #FDA428 wavy underline;'>**Dataset Information**</span>
261
  The dataset contains 200 examples, all written in Polish. Each example consists of the following:
262
  - **Main Text**: This is a statement (often an opinion) on any topic that includes a certain type of implicature, often several simultaneously, such as sarcasm or phraseological compounds.
263
  - **Reference Sentiment**: The sentiment associated with the main text. We use three categories: negative, neutral, and positive. Ambiguous examples were labeled as "neutral" to exclude them from sentiment classification testing.
264
  - **Reference phraseological compounds**: A list of phraseological compounds found in the main text.
265
  - **Reference Explanation**: An explanation of the underlying intentions that the author of the main text might have had.
 
266
  ### <span style='text-decoration: #FDA428 wavy underline;'>**Evaluation Procedure**</span>
267
  We distinguish between two models in the evaluation process:
268
  - **Evaluated Model**: The model that performs specific tasks, is then assessed based on its performance, and added to a ranking.
269
  - **Judge Metamodel**: One of the currently strongest, most versatile LLMs.
 
270
  ### <span style='text-decoration: #FDA428 wavy underline;'>**GENERATING RESPONSES FROM THE EVALUATED MODEL**</span>
271
  1. For each text in the dataset, the evaluated model was required to list the following in three points:
272
  - The sentiment (only positive/negative).
 
282
  - **Assistant Prompt**: A human-written example answer for the second example text.
283
  - **User Prompt**: The target text, based on which the evaluated model will be assessed.
284
  3. The decision to split the examples into user prompts and assistant prompts was made due to the better results achieved by the vast majority of models. The two examples were selected based on diversity: one has a negative sentiment and several phraseological compounds, while the other is positive and lacks phraseological compounds.
 
285
  ### <span style='text-decoration: #FDA428 wavy underline;'>**GENERATING METAMODEL EVALUATIONS**</span>
286
  1. The purpose of the metamodel is to return the following evaluations:
287
  - **Understanding of the Text**: A comparison of the evaluated model's response description to the reference explanation.