This paper presents a novel prototype platform that uses the same LaTeX mark-up language, commonly used to typeset mathematical content, as an input language for modeling optimization problems of various classes. The platform converts the LaTeX model into a formal Algebraic Modeling Language (AML) representation based on Pyomo through a parsing engine written in Python and solves by either via NEOS server or locally installed solvers, using a friendly Graphical User Interface (GUI). The distinct advantages of our approach can be summarized in (i) simplification and speed-up of the model design and development process (ii) non-commercial character (iii) cross-platform support (iv) easier typo and logic error detection in the description of the models and (v) minimization of working knowledge of programming and AMLs to perform mathematical programming modeling. Overall, this is a presentation of a complete workable scheme on using LaTeX for mathematical programming modeling which assists in furthering our ability to reproduce and replicate scientific work. Mathematical modeling constitutes a rigorous way of inexpensively simulating complex systems’ behavior in order to gain further understanding about the underlying mechanisms and trade-offs. By exploiting mathematical modeling techniques, one may manipulate the system under analysis so as to guarantee its optimal and robust operation.http://www.Pyomo.org/) (https://www.python.org/), (The dominant computing tool to assist in modeling is the Algebraic Modeling Languages (AMLs) . AMLs hamo.org/)  for moden.org/),  and AMPLn.org/),  are the  •a strict and specific syntax for the mathematical notation to describe the models; •understand in terms of structural demands;solver interfaces, the bridge between mathematics and what the solver can  •a series of available optimization solvers for as many classes of problems as supported  with the associated functional interfaces implemented; •explicit data file formats and implementation of the respective import/export mechanisms.vertical abstraction) or extend the embedded functionality  . This limited declaration of model components elevates the amount of processing that the platform has to conduct in order to provide equivalent formulations of the input.AMLs provide a level of abstraction, which is higher than the direct approach of generating a model using a programming language. The different levels in the design process of a model are depicted in raction) . The layA systems approach, MOSAIC , has beeLiterate Programming and (ii) the notions of reproducible and replicable research, the fundamental basis of scientific analysis. Literate Programming focuses on generating programs based on logical flow and thinking rather than being limited by the imposing syntactical constraints of a programming language. In essence, we employ a simple mark-up language, LaTeX, to describe a problem  and then in turn produce compilable code (Pyomo abstract model) which can be used outside of the presented prototype platform’s framework. Reproducibility and the ability to replicate scientific analysis is crucial and challenging to achieve. As software tools become the vessel to unravel the computational complexity of decision-making, developing open-source software is not necessarily sufficient; the ability for the averagely versed developer to reproduce and replicate scientific work is very important to effectively deliver impact (https://www.coin-or.org/), Science evolves when previous results can be easily replicated.Our work expands upon two axes: (i) the programming paradigm introduced by Donald E. Knuth  on Literr impact . To quotvertical abstraction. It therefore strengthens the ability to reproduce and replicate optimization models across literature for further analysis by reducing the demands in working knowledge of AMLs or coding. The key capability is that it parses LaTeX formulations of mathematical programs (optimization problems) directly into Pyomo abstract models. The framework then combines the produced abstract model with data provided in the AMPL .dat format (containing parameters and sets) to produce a concrete model. This capability is provided through a graphical interface which accepts LaTeX input and AMPL data files, parses a Pyomo model, solves with a selected solver , and returns the optimal solution if feasible, as the output. The aim is not to substitute but to establish a link between those using a higher level of abstraction. Therefore, the platform does not eliminate the use of an AML or the advantages emanating from it.In the endeavor of simplifying the syntactical requirements imposed by AMLs we have developed a prototype platform. This new framework is materializing a level of modeling design that is higher than the AMLs in terms of This is a complete prototype workable scheme to address how LaTeX could be used as an input language to perform mathematical programming modeling, and currently supports Linear Programming (LP), Mixed-Integer Linear Programming (MILP) as well as Mixed-Integer Quadratic Programming (MIQP) formulations. Linear Optimization  has provThis paper is organized as follows: in ‘Functionality’, we describe the current functionality supported by the platform at this prototype stage. In ‘Parser - Execution Engine’, we present the implementation details of the parser. ‘An illustrative parsing example’ provides a description of an illustrative example. A discussion follows in ‘Discussion’. Some concluding remarks are drawn in ‘Conclusion’. Examples of optimization models that were reproduced from scientific papers as well as their corresponding LaTeX formulations and Pyomo models can be found in the The set of rules that are admissible to formulate models in this platform are formal LaTeX commands and they do not represent in-house modifications. We assume that the model will be in the typical format that optimization programs commonly appear in scientific journals. Therefore, the model must contain the following three main parts and with respect to the correct order as well: 1.the objective function to be optimized (either maximized or minimized); 2.the (sets of) constraints, or else the relationships between the decision variables and coefficients, right-hand side (RHS); 3.the decision variables and their domain space.reads LaTeX code and then writes Pyomo abstract models or the code generates code. The resulting .py file is usable outside of the platform’s frame, thus not making the binding and usage of these two necessary after conversion. The main components that we employed for this purpose are the following:We used the programming environment of Python coupled with its modeling library, namely Pyomo. Similar approaches in terms of software selection have been presented for Differential and Algebraic Equations (DAE) modeling and optimization in . By comb •https://www.mathjax.org/) and Google Polymer (https://www.polymer-project.org/);Front-end: HTML, JavaScript, MathJax ( •https://www.djangoproject.com/) and Pyomo.Back-end: Python with Django (https://www.polymer-project.org/). As the main feature of the platform is to allow modeling in LaTeX language, we used MathJax as the rendering engine. In this way, the user can see the compiled version of the input model. All of these components form a single suite that works across different computational environments. The front-end is plain but incorporates the necessary functionality for input and output, as well as some solver options. The role of the back-end is to establish the communication between the GUI and the parser with the functions therein. In this way the inputs are being processed inside Python in the background, and the user simply witnesses a seamless working environment without having to understand the black-box parser in detail.In order to increase the effectiveness and user-friendliness of the platform, a Graphical-User Interface (GUI) based on HTML, JavaScript (front-end) and Django as the web-framework (back-end) has been implemented, as shown in The main components of the GUI are: •Abstract model input: The input of the LaTeX model, either directly inside the Polymer input text-box or via file upload (a .tex containing the required source LaTeX code) •Data files: The input of the data set which follows the abstract definition of the model via uploading the AMPL-format (.dat) data file •Solver options: An array of solver - related options such as:  1.NEOS server job using CPLEX 2.Solve the relaxed LP (if MILP) 3.Select GPLK (built-in) as the optimization solver 4.Select CPLEX (if available) as the optimization solver (currently set to default)The following is an example of a LaTeX formulated optimization problem which is ready to use with the platform; the well-known Traveling Salesman Problem (TSP) : \documeand the raw LaTeX code used to generate this was:  __________________________________________________________________________________________________________ ∖ text { minimize } ∖ sum ∖ l i m i t s  { i , j : i ∖ neq j }ˆ{}  ∖∖ ∖ text { subject to : }∖∖ ∖ sum ∖ l i m i t s  { j : i ∖ neq j }ˆ{}  = 1 ∖ quad ∖ quad ∖ f o r a l l i ∖∖ ∖ sum ∖ l i m i t s  { i : i ∖ neq j }ˆ{}  = 1 ∖ quad ∖ quad ∖ f o r a l l j ∖∖ u { i } − u { j } + nx { i , j } ∖ leq n − 1 ∖ quad ∖ quad ∖ f o r a l l i ∖ geq 2 , j ∖ leq | j |−1, i      ∖neq j ∖∖ u ∖ in  ∖ mathbb Z , x ∖ in  ∖{0 ,1∖}∖∖ ____________________________________________________________________________________________________________    pre-made .tex file which can be uploaded in the corresponding field of the GUI. Either way, the MathJax Engine then renders LaTeX appropriately so the user can see the resulting compiled model live. Subject to syntax-errors, the MathJax engine might or might not render the model eventually, as naturally expected. Empty lines or spaces do not play a role, as well as commented-out lines using the standard notation (the percentage symbol %). The model file always begins with the objective function sense, the function itself, and then the sets of constraints follow, with the variables and their respective type at the end of the file.which is the input for the platform. The user can either input this code directly inside the Google polymer text box or via a parser we define the part of the code (a collection of Python functions) in the back-end side of the platform which is responsible for translating the model written in LaTeX to Pyomo, the modeling component of the Python programming language. In order to effectively translate the user model input from LaTeX, we need an array of programming functions to carry out the conversion consistently since preserving the equivalence of the two is implied. The aim of the implementation is to provide minimum loss of generality in the ability to express mathematical notation for different modeling needs.As A detailed description of the implemented scheme is given in .tex model file and the .dat AMPL formatted data file are given, the platform then starts processing the model. The conversion starts by reading the variables of the model and their respective types, and then follows with component identification (locating the occurrence of the variables in each constraint) and their inter-relationships . Additionally, any summation and constraint conditional indexing schemes will be processed separately. Constraint-by-constraint the parser gradually builds the .py Pyomo abstract model file. It then merges through Pyomo the model with its data set and calls the selected solver for optimization.Once the \quad command). The platform also supports the use of Greek letters. For instance, if a parameter is declared as α the platform identifies the symbol, removes the backslash and expects to find alpha in the data-file. This takes place also in the pre-processing stage.A significant amount of pre-processing takes place prior of parsing. The minimum and essential is to first tidy up the input; that is, clear empty lines and spaces, as well as reserved (by the platform) keywords that the user can include but do not play any role in functional parsing , no matter if the initial input was done using the frac environment.The user can also opt-out selectively the constraints by putting regular comments in LaTeX, with the insertion of the percentage symbol (%) in the beginning of each expression. Once done, we attempt to simplify some types of mathematical expressions in order to be able to better process them later on. More specifically, we have two main functions that handle fractions and common factor (distributive expressions) simplifications. For example: This keeps the basic component identification functions intact, since their input is transformed first to the acceptable analytical format. Instead of transforming the parsing functions, we transform the input in the acceptable format. However, the user does not lose either functionality or flexibility, as this takes place in the background. To put it simply, either the user inputs the analytic form of an expression or the compact, the parser is still able to function correctly.To frame the capabilities of the parser, we will now describe how the user can define optimization models in the platform with a given example and the successful parsing to Pyomo. The parser first attempts to split the model into its three major distinct parts: •the objective function •the sets of constraints •the types of the variables definedThese three parts are in a way independent but interconnected as well.string manipulation functions, therefore the use of regular expressions in Python was essential and effective.The parser first attempts to read the variables and their respective domain space (type). The platform is case sensitive since it is based on Pyomo. The processing is done using keywords as identifiers while scanning from the top to the bottom of the manually curated .tex file which contains the abstract model in LaTeX. For the three respective different parts mentioned earlier, the corresponding identifiers are:Reasonably, the focus was on consistency and reliability, rather computational performance mainly due to the lightweight workload of the processing demands in general. In order to do that, the parser uses  1.minimize, maximize}Objective function: { 2.leq, \geq, =}Sets of constraints: {\ 3.mathbb , {0, 1}}Variables and their types: {\.py output model file. Variable types can appear in the following way:This helps separate the processing into sections. Each section is analyzed and passes the information in Pyomo syntax in the  •  \in \mathbb R     for Real numbers (∈ℝ)•  \in \mathbb R_+     for non-negative Real numbers (∈ℝ+)•  \in \mathbb R_{*}^{+}     for positive Real numbers •  \in \mathbb Z     for integers (∈ℤ)•  \in \mathbb Z_+     for non-negative integers (∈ℤ+)•  \in \mathbb Z_{*}^{+}     for positive integers , the parser then creates a list of strings for the names of the variables. This is one of the crucial structures of the parser and utilized alongside the entire run-time of the conversion process. A list of the same length, which holds the types of each respective variable, is also created. The platform in general uses Python lists to store information about variables, index sets, parameters, scalars etc.Our approach for understanding the inter-mathematical relationships between the variables and the parameters relied on exploiting the fundamental characteristics of Linear Programming: •Proportionality •Additivity •Divisibilitydecompose them. By decomposition we define the fragmentation of each mathematical expression at each line of the .tex input model file into the corresponding variables, parameters, summations etc. so as we can process the given information accordingly. A simple graphical example is given in These mathematical relationships can help us understand the structure of the expressions and how to ax to describe coefficient a being multiplied by variable x). In some cases however it is imperative to use the asterisk to decompose a multiplication. For example, say Ds is a parameter and s is also a variable in the same model. There is no possible way to tell whether the expression Ds actually means D*s or if it is about a new parameter altogether, since the parameters are not explicitly defined in the model definition (as in AMLs). Adding to that the fact that for the scalars there is no associated underscore character to identify the parameter as those are not associated with index sets, the task is even more challenging. Therefore, we should write D*s if D is a scalar. As for parameters with index sets, for example Dsisi causes no confusion for the parser because the decomposition based on the underscore character clearly reveals two separate components. In this way, the platform also identifies new parameters. This means that since we know, for instance, that s is a variable but Ds is not, we can dynamically identify Ds on the fly (as we scan the current constraint) as being a parameter which is evidently multiplied with variable s, both having index set i associated with them. However, we need to pay attention on components appearing iteratively in different or in the same sets of constraints; did we have the component already appearing previously in the model again? In that case we do not have to declare it again in the Pyomo model as a new quantity, as that would cause a modeling error.The decomposition with the regular expressions is naturally done via the strings of the possible operators found, that is: addition, subtraction, division , since the asterisk to denote multiplication (∗ or ⋅) is usually omitted in the way we describe the mathematical expressions  model. For instance if a set i is identified, the string model.i = Set(dimen = 1) is first written inside the text version of the Pyomo model file, and then on-the-fly executed independently inside the already parsing Python function using the exec command. The execution commands run in a sequential manner. All the different possible cases of relationships between parameters and variables are dynamically identified, and the parser keeps track of the local (per constraint) and global (per model) list of parameters identified while scanning the model in dynamically growing lists.By split function carefully transfers this information intact to the Pyomo model.Dynamic identification of the parameters and index sets is one of the elegant features of the platform, since in most Algebraic Modeling Languages (AMLs) the user explicitly defines the model parameters one-by-one. In our case, this is done in an intelligent automated manner. Another important aspect of the decomposition process is the identification of the constraint type , since the position of the operator is crucial to separate the left and the right hand side of the constraint. This is handled by an independent function. Decomposition also helps identify Quadratic terms. By automatic conversion of the caret symbol to ∗∗ (as this is one of the ways to denote power of a variable in Pyomo language) the Summation terms need to be enclosed inside parentheses (⋯), even with a single component. This accelerates identification of the summation terms with clarity and consistency. Summations are in a way very different than processing a simplified mathematical expression in the sense that we impose restrictions on how a summation can be used. First of all, the corresponding function to process summations tries to identify how many summation expressions exist in each constraint at a time. Their respective indexing expressions are extracted and then sent back to the index identification functions to be processed. The assignment of conditional indexing with the corresponding summation is carefully managed. Then, the summation commands for the Pyomo model file are gradually built. Summations can be expressed in the following form, and two different fields can be utilized to exploit conditional indexing (upper and lower brackets):  ________________________________________________________________ ∖sum∖ l i m i t s _ {p :  X_{n , p} =  1}ˆ{} ________________________________________________________________     which then compiles to: p, (that is for p = 1:|p|) but only when Xn,p = 1 at the same time. If we want to use multiple and stacked summations  we can express them in the same way by adding the indexes for which the summation will be generated, as for example:This means that the summation will be executed for all values of   _______________________________________________________________________ ∖ sum ∖ l i m i t s _ { i , j }ˆ{} ________________________________________________________________________     which then compiles to: i, j. Dynamic (sparse) sets imposed on constraints can be expressed as:and will run for the full cardinality of sets   _____________________________________________________________________________ X _{ i , j } = Y _ { i , j } ∖ f o r a l l  ∖ in  C ∖∖ ______________________________________________________________________________     Xi,j = Yi,j∀ ∈ Cwhich then compiles to: i, j) which belong to the dynamic set C. In order to achieve proper and precise processing of summations and conditional indexing, we have built two separate functions assigned for the respective tasks. Since specific conditional indexing schemes can take place both for the generation of an entire constraint or just simply for a summation inside a constraint, two different sub-functions process this portion of information. This is done using the \forall command at the end of each constraint, which changes how the indexes are being generated for the vertical expansion of the constraints from a specific index set. Concerning summations it is done with the bottom bracket information for horizontal expansion, as we previously saw, for instance, with p:Xn,p = 1.This means that the constraint is being generated only for those values of  or if a more complex expression is used, the for-loop indexes for the summations are found before the colon symbol (:).A series of challenges arise when processing summations. For instance, which components are inside a summation symbol? A variable that might appear in two different summations at the same constraint can cause confusion. Thus, using a binary list for the full length of variables and parameters present in a constraint we identify the terms which belong to each specific summation. This binary list gets re-initialized for each different summation expression. From the lower bracket of each summation symbol, the parser is expecting to understand the indexes for which the summation is being generated. This is done by either simply stating the indexes in a plain way  symbol which then helps understand for which indexes the constraints are being sequentially generated . For instance, ∀ ∈ C makes sure that the constraint is not generated for all combinations of index sets i, j, but only the ones appearing in the sparse set C. The sparse sets are being registered also on the fly, if found either inside summation indexing brackets or in the constraint general indexing (after the ∀ symbol) by using the keywords \in, \notin. The simplest form of constraint indexing is for instance: i and the summation is running for all those values of set j such that i is not equal to j. More advanced cases of constraint conditional indexing are also identified, as long as each expression is separated with the previous one by using a comma. For example in: At the end of each constraint, the parser identifies the “ ∀”’ ( 1.to identify left part (before the operator/reserved keyword/command), 2.the operator and 3.the right-hand part.i < |i|, the left part is set i, the operator is < and the right-hand part is the cardinality of set i. In this way, by adding a new operator in the acceptable operators list inside the code, we allow expansion of supported expressions in a straightforward manner.For example, in transportation problem: Let us now follow the sequential steps that the parser takes to convert a simple example. Consider the well-known We will now provide in-depth analysis of how each of the main three parts in the model can be processed..tex model file that contains the variable symbols and their respective domains. This is done by trying to identify any of the previously presented reserved keywords specifically for this section. The parser reaches the bottom line by identifying the keyword mathbbR_ + in this case. Commas can separate variables belonging to the same domain, and the corresponding parsing function splits the collections of variables of the same domain and processes them separately.The parser first attempts to locate the line of the x. The platform then builds two Python lists with the name of the variables found and their respective types.In this case, the parser identifies the domain and then rewinds back inside the string expression to find the variable symbols. It finds no commas, thus we collect only one variable with the symbol minimize) and tries to identify any involved variables in the objective function. In a different scenario, where not all of the model variables are present in the objective function, a routine identifies one-by-one all the remaining variables and their associated index sets in the block of the given constraint sets.The parser then reads the optimization sense  respectively will be analyzed separately. In this case, the upper one is empty, so the lower one contains all the indexes for which the summation has to scale. Separated by commas, a simple extraction gives i, j to be used for the Pyomo for-loop in the expression. There is no colon identified inside the lower bracket of the summation, thus no further identification of conditional indexing is required.The parser first attempts to locate any summation symbols. Since this is successful, the contained expression is extracted as split function is then applied on the extracted mathematical expression c_{i, j}x_{i, j} to begin identification of the involved terms. Since there are no operators  we have a list containing only one item; the combined expression. It follows that the underscore characters are used to frame the names of the respective components. It is easy to split on these characters and then create a list to store the pairs of the indexes for each component. Thus, a sub-routine detects the case of having more than just one term in the summation-extracted expression. In this example, c is automatically identified as a parameter because of its associated index set which was identified with the underscore character and since it does not belong to the list of variables.A c, as well as the parameters for the current constraint/objective expression. This helps us clarify which parameters are present in each constraint as well as the set of parameters (unique) for the model thus far, as scanning goes on. Once the parameter c and variable x are identified and registered with their respective index sets, we proceed to read the constraint sets. The parser creates expressions as the ones shown below for this kind of operations:The global list of parameters is then updated by adding   ________________________________________________________________________ model . i = Set ( dimen =1) ∖∖ model . j = Set ( dimen =1) ∖∖ model . c = Param  ∖∖ model . x = Var  ∖∖ ________________________________________________________________________    Since the objective function summation symbol was correctly identified with the respective indexes, the following code is generated and executed:  ____________________________________________________________________________________________________________ def  obj  expression ( model ) :   model . F = sum    return  model . F model . OBJ = Objective  ____________________________________________________________________________________________________________     Since the constraints sets are very similar, for shortness we will only analyze the first one. The parser first locates the constraint type by finding either of the following operators ≤,  ≥ ,  =. It then splits the constraint in two parts, left and right across this operator. This is done to carefully identify the position of the constraint type operator for placement into the Pyomo constraint expression later on.a is identified on the fly and since x is already registered as a variable and the parser proceeds to only register the new parameter by generating the following Pyomo expressions:The first component the parser gives is the terms identified raw in the expression  __________________________________________________________________________________________     X in the following piece of code:The platform successfully identifies which terms belong to the summation and which do not and separates them carefully. Eventually the ∀ symbol gives the list of indexes for which the constraints are being generated. This portion of information in the structure of a Pyomo constraint definition goes in replacing  _________________________________________________________________________________________ def  axb _ constraint _  rule _ 1  : __________________________________________________________________________________________    and the full resulting function is:  __________________________________________________________________________________________ def  axb _ constraint _ rule _ 1  :   model . C _ 1 = sum  <= model . a [ i ]   return  model . C _ 1 model . AxbConstraint _ 1 = Constraint  __________________________________________________________________________________________     understand almost every different way of writing mathematical models using LaTeX is nearly impossible; however, even by framing the way the user could write down the models, there are some challenges to overcome. For instance, the naming policy for the variables and parameters. One would assume that these would cause no problems but usually this happens because even in formal modeling languages, the user states the names and the types of every component of the problem. Starting from the sense of the objective function, to the names and the types of the variables and parameters as well as their respective sizes and the names of the index sets, everything is explicitly defined. This is not the case though in this platform; the parser recognizes the parameters and index sets with no prior given information. This in turn imposes trade-offs in the way we write the mathematical notation. For instance, multiple index sets have to be separated by commas as in xi,j instead of writing xij.Developing a parser that would be able to On the other hand, using symbolic representation of the models in LaTeX can enable the user quickly identify errors in the description of the model, the involved variables, parameters or their mathematical relationships therein. This as opposed trying to debug models that have been developed directly in a programming language or in an AML, which would make the detection of such errors or typos more challenging.By scanning a constraint, the parser quickly identifies as mentioned the associated variables. In many cases parameters and variables might have multiple occurrences in the same constraint. This creates a challenging environment to locate the relationships of the parameters and the variables since they appear in multiple locations inside the string expressions and in different ways. On top of this, the name of a parameter can cause identification problems because it might be a sub/super set of the name of another parameter, e.g., parameter AB, and parameter ABC. Therefore naming conflicts are carefully resolved by the platform by meticulously identifying the exact location and occurrences of each term.The CPU time required for each step in the modeling process of the platform  can be found in the Supplementary Information. It can be noted that the parser is the least consuming step, which clearly demonstrates the efficiency of the platform. The Pyomo model generation and solver (CPLEX in our measurements) steps and their associated CPU-time are completely outside of the parser’s control. However, it is essential to get an idea of how these timings compare to each other with the addition of this extra higher level of abstraction in the beginning of the modeling process.Challenges also arise in locating which of the terms appearing in a constraint belong to summations, and to which summations; especially when items have multiple occurrences inside a constraint, there needs to be a unique identification so as to include a parameter (or a variable) inside a specific summation or not. We addressed this with the previously introduced binary lists. Then, for each of those summation symbols, the items activated (1) are included in the summation or not (0) and the list is generated for each different summation within the expression.Additionally, another challenge constitutes the extension of the platform to support nonlinear terms, where each term itself can be a combination of various operators and mathematical functions.Finally, it is worth mentioning that the amount of lines/characters to represent a model in LaTeX in comparison with the equivalent model in Pyomo is substantially smaller. In this respect, the platform accelerates the modeling development process.left) which would capture more complex mathematical relationships.We presented a platform for rapid model generation using LaTeX as the input language for mathematical programming, starting with the classes of LP, MILP and MIQP. The platform is based on Python and parses the input to Pyomo to successfully solve the underlying optimization problems. It uses a simple GUI to facilitate model and data input based on Django as the web-framework. The user can exploit locally installed solvers or redirect to NEOS server. This prototype platform delivers transparency and clarity, speedup of the model design and development process (by significantly reducing the required characters to type the input models) and abstracts the syntax from programming languages and AMLs. It therefore delivers reproducibility and the ability to replicate scientific work in an effective manner from an audience not necessarily versed in coding. Future work could possibly involve expansion to support nonlinear terms as well as differential and algebraic equations, sanity checking and error catching on input, the ability to embed explanatory comments in the input model file which would transfer to the target AML, extending the functionality concerning bounds on the variables as well as adding further support to built-in LaTeX commands (such as ∖10.7717/peerj-cs.161/supp-1Supplemental Information 1Click here for additional data file.


HemaSphere. 2020;4:e448), there was an error in the original published title. “Life-threatening” instead appeared as “Life-threating.” The title has now been corrected online:In the article entitled “Coronavirus Disease 2019 in Recipient of Allogeneic Hematopoietic Stem Cell Transplantation: Life-threatening Features Within the Early Post-engraftment Phase” (https://journals.lww.com/hemasphere/Fulltext/2020/08000/Coronavirus_Disease_2019_in_Recipient_of.14.aspx


Ultrasonic vocalizations (USVs) analysis is a well-recognized tool to investigate animal communication. It can be used for behavioral phenotyping of murine models of different disorders. The USVs are usually recorded with a microphone sensitive to ultrasound frequencies and they are analyzed by specific software. Different calls typologies exist, and each ultrasonic call can be manually classified, but the qualitative analysis is highly time-consuming. Considering this framework, in this work we proposed and evaluated a set of supervised learning methods for automatic USVs classification. This could represent a sustainable procedure to deeply analyze the ultrasonic communication, other than a standardized analysis. We used manually built datasets obtained by segmenting the USVs audio tracks analyzed with the Avisoft software, and then by labelling each of them into 10 representative classes. For the automatic classification task, we designed a Convolutional Neural Network that was trained receiving as input the spectrogram images associated to the segmented audio files. In addition, we also tested some other supervised learning algorithms, such as Support Vector Machine, Random Forest and Multilayer Perceptrons, exploiting informative numerical features extracted from the spectrograms. The performance showed how considering the whole time/frequency information of the spectrogram leads to significantly higher performance than considering a subset of numerical features. In the authors’ opinion, the experimental results may represent a valuable benchmark for future work in this research field. Rodent models are good tools for scientific research because they allow to study and understand the biological mechanisms underlying the different pathologies. Behavioral alterations in animal models offer markers for the symptoms of the human diseases . The stuThe USVs are generally recorded with an ultrasound sensitive microphone and they are analyzed by specific software applications. Each syllable can be classified manually based on specific features, such as frequency, duration, amplitude and general shape. The manual classification provides a detailed characterization, but it is very time-consuming and may be subject to personal interpretation. Even using specific and professional tools, this task usually takes a lot of time and it appears to be only partially automatized. Indeed, it is possible to automatically analyze quantitative parameters , but it is more difficult to evaluate qualitative parameters, such as the different typologies of ultrasonic calls. For these reasons, it could be very interesting to find a method that automatically processes vocalizations starting from the audio tracks. This fundamental and challenging step could speed up the analysis of ultrasonic communication and also provide insight into the meaning of different USVs.ad hoc method for automatic USVs classification on the basis of the well-known USVs classification pattern published by Scattoni and colleagues Reviewers' comments:Reviewer's Responses to QuestionsComments to the Author1. Is the manuscript technically sound, and do the data support the conclusions?The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #1: PartlyReviewer #2: Yes**********2. Has the statistical analysis been performed appropriately and rigorously? Reviewer #1: NoReviewer #2: Yes**********3. Have the authors made all data underlying the findings in their manuscript fully available?PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.The Reviewer #1: NoReviewer #2: No**********4. Is the manuscript presented in an intelligible fashion and written in standard English?PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.Reviewer #1: YesReviewer #2: Yes**********5. Review Comments to the AuthorPlease use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. Reviewer #1: The authors developed several classifiers capable of classifying individual vocalizations according to their spectrographicshapes as described by Scattoni et al, 2008. The work employs a quite comprehensive set of machine learning methodsincluding Support Vector Machines(SVM), Random Forests(RF), fully connected and convolutional artificial neural networks .Two-dimensional spectrograms were used as the input data for CNN-based classifiers. For non-convolutional ANN, SVM andRF classifiers the vocalizations were represented as the sets of 16 automatically extracted features.Although the authors made significant effort in optimizing the hyperparameters of SVM and RF classifiers, the set of features chosenrepresenting the vocalizations seems to be inadequate for the task of spectrogram shape classification.It can be hard to differentiate 'chevron' from 'complex' using only overall and marginal max/mean/min values. The confusionmatrices provided in the paper confirm that.The description of the architecture of ANN provided by the authors lacks details. Two convolutional layers onlycan be insufficient for good performance in image classification tasks. The authors don't mention whether they use any ofthe overfitting-prevention techniques such as batch normalization, dropout, weight regularization and data augmentation.The fact that the CNN applied to full spectrograms performs worse than SVM applied to the ambiguous feature setindicates that more effort can be invested in the optimization of CNN architecture and the training protocol.Note also the performances shown by CNNs in much more complex image classification tasks (CIFAR100) andthe RF performance demonstrated by Vogel at al, 2019 solving a similar problem.According to my assessment, the authors need to address the major and minor points listed below.Major1. For statistical confidence, the cross validation technique(e.g. 10-fold) should be applied for model assessment andthe results should be shown together with their confidence intervals2. The feature set used for RF, SVM and non-convolutional ANN input doesn't seem to be adequate for the desired task.The authors should extend the feature set and/or use entire frequency envelope data as the input.3. The architecture of ANNs should employ widely used techniques for overfitting prevention: batch normalization/dropout. It would begood to employ data augmentation to help the CNN perform better. The batch size alsocan be increased to help improve the performance and/or convergence speed.Minor1. SVM OVA approach requires confidence estimation. Which confidence estimation approach was used?2. ANN architecture should be described more thoroughly: convolutional layers kernel sizes, strides, dropout/batch normalization usage,stochastic gradient descent optimizer used, weight initialization strategy.3. The report of Vogel et al, 2019'Quantifying ultrasonic mouse vocalizations using acoustic analysis in asupervised statistical machine learning framework'solving a similar task should be mentioned4. Strain data is not used in the analysis anyhow.It would be interesting to compare the accuracies of two strains' shape prediction results taken separately.Typos96 'can not applies'124 'postnatal (PND)'Reviewer #2: Overview: The authors manually segmented and labeled an impressive number of mouse USVs (48699) according to categories developed by Scattoni and colleagues (2008). A sampling of 1199 USVs per category was used to train several supervised classification algorithms. Support Vector Machines (SVM), Random Forests (RF) and Artificial Neural Networks (ANN), in several configurations. The authors conclude that the best results are obtained by Support Vector Machines with the One-VS-All configuration. However, no method was particularly accurate. Precision, recall, and accuracy fell between %51.4 and 68.5% for all classifiers. I believe the moderate accuracy of the classifiers described in this report are not due to any deficiencies in methodology employed by the authors, rather they are due to the fundamental inaccuracy of human defined USV classification.Main Issues:The original creator of these particular USV categories, Maria Luisa Scattoni, has already published a paper using support vector machines (SVM) and random forests (RF) to categories USVs. They achieved 85% recall. What more does this paper add?https://doi.org/10.1038/s41598-019-44221-3Vogel, A.P., Tsanas, A. & Scattoni, M.L. Quantifying ultrasonic mouse vocalizations using acoustic analysis in a supervised statistical machine learning framework. Sci Rep 9, 8100 (2019). Experimenter derived call categories are generally falling out of favor. Substantial new evidence suggests that USVs don’t categorize neatly into discrete groups, including the data in this paper. SVMs/RFs are capable of much higher accuracy when the training data actually comes from discrete groups with clear separations. The “short” calls in the present manuscript are a well-defined group and are thus categorized accurately (>90%). But other calls like “complex” and “composite” are frequently miss-categorized. This likely isn’t a fault of the classifier. Rather, it captures the uncertainty within the human created training data.https://doi.org/10.1101/870311Tim Sainburg, Marvin Thielk, Timothy Q Gentner (2019) Latent space visualization, characterization, and generation of diverse vocal communication signals. bioRxiv 870311; doi: Many methods are now available for un-biased call classification. These categories can then be validated through the behavioral/contextual usage of the calls. Call categories that are used identically during behavior can be collapsed. This method is more sophisticated and ethologically relevant than creating categories based on experimenter visual inspection.https://doi.org/10.1038/s41593-020-0584-zSangiamo, D.T., Warren, M.R. & Neunuebel, J.P. Ultrasonic signals associated with different types of social behavior of mice. Nat Neurosci 23, 411–422 (2020). The training data, code, and finalized classifiers are the most valuable elements of this manuscript, but I did not see any indication that they will be distributed to the field. Without this, the manuscript just describes their internally used classifier with moderate accuracy.Final Thoughts:While there is nothing wrong with the scientific methodology employed in this paper, I would like to see all of the issues above addressed before considering the manuscript for publication.**********what does this mean?). If published, this will include your full peer review and any attached files.6. PLOS authors have the option to publish the peer review history of their article  digital diagnostic tool,  30 Sep 2020We would like to take this opportunity to sincerely thank the Reviewers for their valuable comments. We have carefully revised our manuscript, taking all the comments and suggestions into consideration.General comments__________________________________________________________________________First, we list the main efforts we have spent to enhance the experimental phase, in accordance with the Reviewers’ recommendations:1) Expansion of the statistical analysis of the results;2) Rerun of the experiments after:a. Extending the standard feature space, for feature-based learning algorithms;b. Improving the Convolutional Neural Network (CNN) architecture by also including pertinent techniques for overfitting prevention;3) Implementation of two new sets of experiments, namely:a. Classification after splitting the dataset into the two genotype-based datasets;b. Classification performed by removing one class.The new experiments have led to an overall improvement in the classification performance, especially when the CNN is used. Of course, the updated results are reported in the new version of the paper. We change the title of the manuscript in “Automatic classification of mice vocalizations using Machine Learning techniques and Convolutional Neural Networks”, suggesting the importance of use of the Convolutional Neural Networks in our paper.Beyond the experiment aspect, another point that has importantly contributed to the paper revision is related to the work that was suggested by both Reviewers, namely:Vogel, A.P., Tsanas, A. & Scattoni, M.L. Quantifying ultrasonic mouse vocalizations using acoustic analysis in a supervised statistical machine learning framework. Sci Rep 9, 8100 (2019).Accordingly, in the revised paper we discuss the main points that differentiate our work from the above cited paper, which has been properly referenced therein. Furthermore, we want to point out that now we have shifted our attention to the classification method based on CNN architecture, since in the 2019 Scientific Reports paper some traditional machine learning (ML) algorithms were already tested. Consequently, the discussion on the ML techniques results to be more limited with respect to the first submission. This change of focus is also motivated by the fact that the new experiments, carried out in the revision stage, have shown notably superior performance when using CNN as stated above. In addition, as pointed out in the 3) b. item above listed, in order to make the comparison as fair as possible, we have also implemented a further set of experiments, in which the classification has been performed by removing the class harmonic, as done in Vogel’s paper.As a final note, we also mention that we will make both the dataset and the implementation code publicly available.In the following, we reply item by item to the specific Reviewers’ comments.Response to Reviewer #1__________________________________________________________________________Comment 1 (major)• For statistical confidence, the cross-validation technique (e.g. 10-fold) should be applied for model assessment and the results should be shown together with their confidence intervals.Response: Following this valuable suggestion, we have applied 10-fold cross-validation for model assessment. In the revised paper, we have reported the results as mean ± standard deviation.__________________________________________________________________________Comment 2 (major) and Comment 3 (minor)• The feature set used for RF, SVM and non-convolutional ANN input doesn’t seem to be adequate for the desired task. The authors should extend the feature set and/or use entire frequency envelope data as the input.• The report of Vogel et al, 2019 ‘Quantifying ultrasonic mouse vocalization using acoustic analysis in a supervised statistical machine learning framework’ solving similar task should be mentioned.Response: The problem expressed in Comment 2 (major) also implicitly deals with Comment3 (minor), therefore we would like to address both the comments in the same answer.We agree with the Reviewer that the results obtained by using the feature set described in the original paper suggest a perfectible ability to generalize the investigated machine learning models. As a matter of fact, we have extended the original feature set by including other features extracted from the spectrograms.The new features selection has been based from the paper suggested by the Reviewer . We have carefully read that report, in which the authors propose an optimal feature subset of 8 acoustic measures for USV classification. They are duration, quart 50 (start), peak freq (stddeventire), peaktopeak, max freq (stddeventire), peak freq (start), peak freq (end) and quart 75 (start). Since 4 of these 8 features are already included in the original feature set, namely, duration, peak freq (start), peak freq (end) and peak freq (stddeventire), we have just added the remaining 4 features, leading to a final feature space formed by 20 features .The new performances outperform the ones referred to the submitted paper, even if the strongest improvement has been obtained by enhancing the CNN architecture, as pointed out in the next response.______________________________________________________________________________Comment 3 (major) and Comment 2 (minor)• The architecture of ANNs should employ widely used techniques for overfitting prevention: batch normalization/dropout. It would be good to employ data augmentation  to help the CNN perform better. The batch size also can be increased to help improve the performance and/or convergence speed.• ANN architecture should be described more thoroughly: convolutional layers kernel sizes, strides, dropout/batch normalization usage, stochastic gradient descent optimizer used, weight initialization strategy.Response: Regarding the issues detailed in Comment 3 (major), it is undoubtable that overfitting seems to afflict the proposed Convolutional Neural Network (CNN). The previously reported accuracy in training and testing phases are 87.0% and 58.3%, respectively, indicating that our model had some problem to generalize new examples not included in the training dataset.Agreeing with the reviewer suggestions, some techniques for overfitting prevention have been implemented, listed in the following:a. Regularization, i.e., dropout;b. Batch normalization;c. Testing different batch sizes;d. Data augmentation.In order to properly illustrate how we have implemented the just mentioned techniques, it can be helpful to start by describing the CNN architecture, that in turn also answers to Comment 2 (minor). First of all, we want to point out that substantial changes have been applied in the model architecture: indeed, the introduction of methods to prevent overfitting came with an increase of the architecture complexity.The final model is composed by 5 convolutional layers constituted by 32, 64, 64, 128 and 128 filters, respectively. The convolutional layers kernel sizes are 7x7 and 5x5 in the first two layers, and 3x3 in the successive three. The stride is set to 1 for each layer.A max-pooling layer is inserted after each convolutional layer. The pooling size is fixed to 3x3, while the stride is set to 2x2. The tensor is then flattened, and two fully connected layers are inserted, the first of size 1024 with ReLU activation function, and the second of size 10 with Softmax activation function. We mention that the Adam optimization algorithm has been used as stochastic gradient descent optimizer, and that the uniform distribution has been adopted to initialize the network weights. The description of the final CNN architecture has been thoroughly illustrated in the new version of the paper.To prevent overfitting, the strategies that we referred above as a., b., c., and d., have been tested. Dropout, i.e. a., deletes a random sample of the activations  during training. Of course, dropout causes information loss, in particular, losing something in the first layer propagates that loss to the whole network. We have designed several configurations, i.e., we have located dropout layers in different positions and tested a range of rates (the fraction of input units to drop), from 0.2 to 0.8. The batch normalization technique (b.) has been similarly evaluated: it helps to coordinate the update of multiple layers in the model. We have experimented batch normalization layers both before and after the activation functions, and in parallel we have considered different batch sizes (c.) in the training phase.In the end, after this thorough hyperparameters optimization/tuning, the best model resulted the one with a dropout layer after each convolutional/max-pooling layer, except the first one, setting the rates to 0.2, 0.3, 0.4 and 0.5. Then, a further dropout layer was added after the first fully-connected layer , and finally batch normalization was included before the Softmax activation function of the last fully-connected layer. The batch size finally adopted is 32. By applying all of these adjustments, the performance of the final CNN architecture has notably improved: indeed, the test accuracy has been enhanced from 58.3% to 78.8%.Here is a final note on data augmentation (d.): given the characteristics of the spectrograms, some image transformations cannot be analyzed, since they would invalidate the ground-truth classification. For example, rotation or horizontal/vertical flipping could lead the network to confuse the upward class with the downward class, and vice-versa. For this reason, and also as suggested by the reviewer, we have increased the dataset by shifting and cropping the spectrogram images at random: however, the experiments have shown that data augmentation does not significantly improve the performance of the model. As with the other points, the discussion on overfitting prevention has been properly argued in the revised paper.______________________________________________________________________________Comment 1 (minor)• SVM OVA approach requires confidence estimation. Which confidence estimation approach was used? Response: For SVM OVA method, we set a maximum confidence strategy, similar to the weighted voting strategy from OVO systems. So, the output class is taken from the classifier with the largest positive answer. We have reported this detail in the revised version of the paper.______________________________________________________________________________Comment 4 (minor)• Strain data is not used in the analysis anyhow. It would be interesting to compare the accuracies of two strains' shape prediction results taken separately. Response: We have taken into consideration this valuable suggestion. As a matter of fact, in the revised paper we have included a new set of experiments, by testing the classification methods on the genotype-based datasets taken separately.______________________________________________________________________________Typos• 96 'can not applies' • 124 'postnatal (PND)' Response: We have corrected the typos indicated by the Reviewer.Response to Reviewer #2__________________________________________________________________________Comment 1• The original creator of these particular USV categories, Maria Luisa Scattoni, has already published a paper using support vector machines (SVM) and random forests (RF) to categories USVs. They achieved 85% recall. What more does this paper add?Vogel, A.P., Tsanas, A. & Scattoni, M.L. Quantifying ultrasonic mouse vocalizations using acoustic analysis in a supervised statistical machine learning framework. Sci Rep 9, 8100 (2019). Response: We have meticulously read the suggested report, and we agree that it is important to shed light on the improvements led by our work with respect to Vogel’s one. We want to focus here on the two main breakthroughs that in our opinion are worthy to be pointed out.First, we made a significant effort to build our dataset. We performed our experiments on 48699 samples, which is a much more sizable dataset than the 225 samples evaluated in the suggested paper. It is well-known that the bigger the sample size is, the more accurate the research results are. That is because larger dataset cardinality allows to better determine the average values of data and reduce potential errors from testing a small number of possible outliers. As a consequence, the statistical analysis becomes more accurate and the margin of error smaller. Given these crucial considerations, we believe that our results benefit from a very effective statistical soundness, and thus that they may represent a further valuable benchmark for future works in this research field. Of course, the dataset will be made publicly available.The second aspect we want to mention is that in our work we explored the classification ability of a Convolutional Neural Network (CNN), extending the analysis beyond traditional machine learning algorithms, such as Support Vector Machine and Random Forest, employed in Vogel’s paper. To the best of the authors’ knowledge, this work is the first to propose a deep learning architecture aimed to classify the ten categories originally introduced in [17]. Furthermore, the CNN model has been notably modified during the revision phase. Accordingly, we have included all of these modifications in the revised paper. In particular, if compared with the original proposed architecture, we have changed the network topology by inserting more convolutional layers and increased the number of the filters composing them, in parallel to have investigated some techniques for overfitting prevention, such as regularization (drop out) and batch normalization (please find more details in the new version of the paper). As a consequence of such improvements, the performance of the final CNN model now achieves an accuracy of 78% on the test dataset, strongly improving the results reported in the original paper (where the test accuracy was 58.3 %).• Experimenter derived call categories are generally falling out of favor. Substantial new evidence suggests that USVs don’t categorize neatly into discrete groups, including the data in this paper. SVMs/RFs are capable of much higher accuracy when the training data actually comes from discrete groups with clear separations. The “short” calls in the present manuscript are a well-defined group and are thus categorized accurately (>90%). But other calls like “complex” and “composite” are frequently miss-categorized. This likely isn’t a fault of the classifier. Rather, it captures the uncertainty within the human created training data.Tim Sainburg, Marvin Thielk, Timothy Q Gentner (2019) Latent space visualization, characterization, and generation of diverse vocal communication signals. bioRxiv 870311.Response: We think that this different perspective is very interesting. We agree that automatic USVs classification into a fixed number of discrete groups is a challenging task, as also suggested by the paper cited in the comment. Indeed, the results reported in the originally submitted paper confirmed the difficulty of distinguishing particular classes, such as complex and composite.Nevertheless, even if manual classification is an undeniably time-consuming activity and also vulnerable to personal interpretation, it can be performed within a more than reasonable margin of error, especially when the general shape of the spectrogram is analyzed.Even acknowledging an intrinsic uncertainty in the human labelling of data, in our work we addressed the problem of designing efficient and accurate machine learning and deep learning models, aimed to automatically classify vocalizations starting from the audio tracks. The performance obtained from the new set of experiments suggests that crucial improvements have been implemented to the originally proposed models. Indeed, the new results show a lesser uncertainty in USVs categorization, indicating that well defined models can actually lead to significant performance even when discrete vocalizations classification is investigated.This discussion has been inserted also in the revised paper, citing the suggested article.______________________________________________________________________________Comment 2• Many methods are now available for un-biased call classification. These categories can then be validated through the behavioral/contextual usage of the calls. Call categories that are used identically during behavior can be collapsed. This method is more sophisticated and ethologically relevant than creating categories based on experimenter visual inspection.Sangiamo, D.T., Warren, M.R. & Neunuebel, J.P. Ultrasonic signals associated with different types of social behavior of mice. Nat Neurosci 23, 411–422 (2020). doi.org/10.1038/s41593-020-0584-z). This permits to associate distinct vocalizations categories with different types of murine social behavior giving important information to calls meaning. Unfortunately, we do not have these sophisticated instruments to associate behaviors to calls. In addition, we recorded vocalizations emitted by pups and not adults and in pups it is not possible to associate calls to behaviors  as done by Sangiamo and colleagues because pups are still very young (first days after birth) and unable to move or perform more complex actions. They still have closed eyes. Finally, we agree with the Reviewer with the idea that it is very interesting to understand the meaning of mice calls and so, in the future, we would like to perform more sophisticated and ethologically relevant analysis on adult mice.Response: In the literature different methods of calls classification exist. We used a method based on experimenter visual inspection referring to papers of Dr. Scattoni cited in our references. This method permits to the operator to classify vocalizations based on features as frequency content, duration, amplitude and general shape. This classification provides a detailed USV characterization, but it says nothing about the meaning of the calls. Recently, new technologies have been developed such as a microphone array system and sound source localization method, to localize and assign USVs to individual mice during a social context  6, 8 ..." -> "postnatal day (PND) 6, 8..."2. Threshold and hold time parameter values used in Avisoft for vocalization extraction should be provided here.Description of the experiments1. 'the first step consisted into' -> 'the first step consisted of'2. 'giving this way a statistical soundness' -> 'ensuring the statistical soundness' ?Features extraction section1. 'identified by the color...'  -> 'identified by the brightness...'Support vector machines1. 'Maximum confidence strategy was used...' - how the confidence was estimated? SVM provides binary result out of the box, the actual confidence assessment approaches vary.plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.Please submit your revised manuscript by Dec 21 2020 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at Please include the following items when submitting your revised manuscript:A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocolsIf applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: We look forward to receiving your revised manuscript.Kind regards,Gennady Cymbalyuk, Ph.D.Academic EditorPLOS ONE[Note: HTML markup is below. Please do not edit.]Reviewers' comments:Reviewer's Responses to QuestionsComments to the Author1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.Reviewer #1: All comments have been addressedReviewer #2: All comments have been addressed**********2. Is the manuscript technically sound, and do the data support the conclusions?The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #1: YesReviewer #2: Yes**********3. Has the statistical analysis been performed appropriately and rigorously? Reviewer #1: YesReviewer #2: Yes**********4. Have the authors made all data underlying the findings in their manuscript fully available?PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.The Reviewer #1: NoReviewer #2: Yes**********5. Is the manuscript presented in an intelligible fashion and written in standard English?PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.Reviewer #1: NoReviewer #2: Yes**********6. Review Comments to the AuthorPlease use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. Reviewer #1: Though the authors addressed the points noted in the previous review, the document undergone majorchanges that gave rise to several more major and minor inconsistencies to be addressed.Additionally, for the next review round, I would ask the authors to add line numbers in the documentand to provide the source code for the models described in the paper.Major1. The Random Forests, MP, Stacking NN classification performance results are completely omitted in the paper.SVM performance results are hard to find in the text. At the same time, both RF and SVM are mentioned in the abstract andmultiple times throughout the text. The results of all classifiers mentioned in the paper should be presented in a table.If the resulting accuracy is especially bad and is not worth considering than it should be stated explicitly.Note that the results are compared with the results of RF classifier in Vogel et al, 2019 paper, so the omitting of the RF resultslooks strange.2. 'Data-preprocessing' paragraph. 'This operation prevent imbalanced dataset...' - usually, that is not the mainreason to apply data augmentation when training CNN. To deal with imbalanced data one can just use weighted loss function, asyou mention further in the paragraph. Data augmentation is used mostly to prevent overfitting,generating a potentially infinite, though maybe not diverse enough set of training samples.From your description it is not clear which approach for data augmentationis used for the best-performing CNN: statically generated examples with random crops and shifts,just to balance the source data set,or the on-the-fly generation of randomly altered samples during the training,together/without using loss function weighting?That should be stated explicitly.Additionally, I think that is not necessary to explain that one should not rotate on flip images in that case.'data augmentation' is a general term for generation of pseudo-diverse samples andthat is quite obvious that one should not use the techniques applied for object photographs classification here...3. 'Proposed classification methods' paragraph : you briefly explain the principles of CNN and only after that you startexplaining trivial theory of MP, talking about weights, activation functions and neurons.'Multilayer perceptron' sub-section should go first, since it describes the most basic things on which CNN is based as well.Furthermore, I don't think Fig.2 is worth placing in the paper. That diagram is quite trivial, it occupies half of a pageand is known since 70s. It can be found in any textbook about ANN basics.Why not to use that space to depict the actual architecture of the CNN + ANNs you designed,maybe together with some trained kernel weights visualization or other data?For example, see the figures in our recently published workIvanenko et al, 2020, "Classifying sex and strain from mouse ultrasonic vocalizations using deep learning".MinorAbstract1. tested some supervised ... -> tested some other supervised ...2. extracted by the spectrograms -> extracted from the spectrogramsIntroduction1. "... on such a fixed repertoire of calls typologies". I think this should be rephrased.May be "on a predefined set of call types" ?2. Please cite our paperIvanenko et al, 2020, "Classifying sex and strain from mouse ultrasonic vocalizations using deep learning", PLOS CBin the introduction. Though we don't use Scattoni classical vocalization types there, we alsoclassify vocalizations using CNNs basing on their spectrogram shape , thus implementing the 'top-down' approach you mentioned.5. 'Simulation results' - I think the word 'simulation' is misleading here.USV Recording and analysis1. "postnatal (PND) 6, 8 ..." -> "postnatal day (PND) 6, 8..."2. Threshold and hold time parameter values used in Avisoft for vocalization extraction should be provided here.Description of the experiments1. 'the first step consisted into' -> 'the first step consisted of'2. 'giving this way a statistical soundness' -> 'ensuring the statistical soundness' ?Features extraction section1. 'identified by the color...' -> 'identified by the brightness...'Support vector machines1. 'Maximum confidence strategy was used...' - how the confidence was estimated? SVM provides binary result out of the box,the actual confidence assessment approaches vary.Reviewer #2: The authors have made significant efforts to improve their CNN based classification architecture and have at least discussed and considered the theoretical limitations I posed in review. I agree that the scale of the new experiment and improved classification accuracy now expand upon, rather then duplicate, the work of the Scattoni Lab. With the addition of a publicly available dateset and classification CNN, this work now makes a tangible contribution to the field and I recommend it be accepted for publication.**********what does this mean?). If published, this will include your full peer review and any attached files.7. PLOS authors have the option to publish the peer review history of their article  digital diagnostic tool,  4 Dec 2020We would like to take this opportunity to sincerely thank everyone involved in the review process. We are glad that the adjustments introduced in response to reviewers' comments process have been generally appreciated. The further changes suggested by the reviewer #1 are detailed below.Response to Reviewer #1__________________________________________________________________________Comment 1 (major) The Random Forests, MP, Stacking NN classification performance results are completely omitted in the paper. SVM performance results are hard to find in the text. At the same time, both RF and SVM are mentioned in the abstract and multiple times throughout the text. The results of all classifiers mentioned in the paper should be presented in a table. If the resulting accuracy is especially bad and is not worth considering than it should be stated explicitly.Note that the results are compared with the results of RF classifier in Vogel et al, 2019 paper, so the omitting of the RF results looks strange.Response: As mentioned in the previous response process, in the first revised paper we had moved the focus from the “numerical features”-based classification techniques, both for distancing our work with respect to Vogel’s paper (as requested by both the reviewers), and also because the new results obtained by using the Convolutional Neural Networks had outperformed the others. Accordingly, we had considered not interesting and redundant the list of all the performance for all the investigated techniques, with that not for that intending to diminish the importance of studying those methods in our work.However, we agree with the reviewer that a table containing the overview of all the results may benefit the completeness of the paper. As a matter of fact, in the new version of the paper, we have inserted such a table in the “Performance analysis” section. Nonetheless, to streamline the paper, we just report the performance related to the entire mice vocalizations dataset, since the strain-based performances result are similar to the ones obtained for the entire dataset, and so they do not add significant extra information.Finally, the source code has been properly integrated as well.__________________________________________________________________________Comment 2 (major) 'Data-preprocessing' paragraph. 'This operation prevents imbalanced dataset...' - usually, that is not the main reason to apply data augmentation when training CNN. To deal with imbalanced data one can just use weighted loss function, as you mention further in the paragraph. Data augmentation is used mostly to prevent overfitting,generating a potentially infinite, though maybe not diverse enough set of training samples.From your description it is not clear which approach for data augmentationis used for the best-performing CNN: statically generated examples with random crops and shifts, just to balance the source data set, or the on-the-fly generation of randomly altered samples during the training, together/without using loss function weighting? That should be stated explicitly.Additionally, I think that is not necessary to explain that one should not rotate on flip images in that case. 'data augmentation' is a general term for generation of pseudo-diverse samples and that is quite obvious that one should not use the techniques applied for object photographs classification here...Response: Thanks for pointing this out. We take this space to better clarify how data augmentation has been used in our experiments, and we will report these considerations in the new version of the paper.For the CNN design, we have tested data augmentation for both overfitting and imbalanced dataset prevention, separately. In the first case, an offline generation of new examples with random shifts and crops is applied over the entire training dataset; when using data augmentation for this aim, a properly weighted loss function is employed during the training in order to handle the imbalanced dataset problem too. In the second one, data augmentation is performed to balance the source dataset, and so by generating new samples just for the under-represented classes.However, note that data augmentation has been tested in the experimental phase, but it has not been in the end used to obtain the final best-performing CNN model. Indeed, the experimental tests have revealed that just including dropout and batch normalization layers is the most efficient action for overfitting prevention (as reported in the “Classification via CNN” section). Furthermore, the experiments have also provided more uniform accuracy values across the classes when the loss function is properly weighted instead of repopulating the under-represented classes by implementing data augmentation. In conclusion, data augmentation has not been exploited for designing the most-performing CNN architecture.As mentioned before, in the new version of the paper, we have better clarified all these aspects by partially re-writing the former section “Data pre-processing”, which is now titled “Class imbalance handling”. Therein, the reviewer will find that all his/her observations above have been properly considered. __________________________________________________________________________Comment 3 (major) 'Proposed classification methods' paragraph: you briefly explain the principles of CNN and only after that you start explaining trivial theory of MP, talking about weights, activation functions and neurons. 'Multilayer perceptron' sub-section should go first, since it describes the most basic things on which CNN is based as well.Furthermore, I don't think Fig.2 is worth placing in the paper. That diagram is quite trivial, it occupies half of a page and is known since 70s. It can be found in any textbook about ANN basics.Why not to use that space to depict the actual architecture of the CNN + ANNs you designed, maybe together with some trained kernel weights visualization or other data? For example, see the figures in our recently published workIvanenko et al, 2020, "Classifying sex and strain from mouse ultrasonic vocalizations using deep learning".Response: Thanks for the suggestion. We agree with the reviewer that the paper may be clearer by swapping the “Convolutional Neural Networks” and “Multilayer Perceptron” sub-sections.In the revised version of the paper, we have modified the description of the multilayer perceptrons without the support of Fig. 2. In its place, there is a new figure depicting the entire architecture of the designed CNN; such a figure has been suitably added in “Classification via CNN” section.__________________________________________________________________________Minor (Abstract) tested some supervised ... -> tested some other supervised ... extracted by the spectrograms -> extracted from the spectrogramsResponse: We have revised both the sentences as in 1) and 2).__________________________________________________________________________Minor (Introduction) "... on such a fixed repertoire of calls typologies". I think this should be rephrased. May be "on a predefined set of call types"? Please cite our paper Ivanenko et al, 2020, "Classifying sex and strain from mouse ultrasonic vocalizations using deep learning", PLOS CB in the introduction. Though we don't use Scattoni classical vocalization types there, we also classify vocalizations using CNNs basing on their spectrogram shape , thus implementing the 'top-down' approach you mentioned. 'Simulation results' - I think the word 'simulation' is misleading here.Response: We have revised both the sentences as in 1) and 3). We have also carefully read the paper suggested in 2) and cited it in the introduction while mentioning the top-down approach.__________________________________________________________________________Minor  "postnatal (PND) 6, 8 ..." -> "postnatal day (PND) 6, 8..." Threshold and hold time parameter values used in Avisoft for vocalization extraction should be provided here.Response: We have revised the sentences as in 1) and inserted the requested parameters in the appropriate section as suggested in 2).__________________________________________________________________________Minor (Description of the experiments) 'the first step consisted into' -> 'the first step consisted of' 'giving this way a statistical soundness' -> 'ensuring the statistical soundness'?Response: We have revised both the sentences as in 1) and 2).__________________________________________________________________________Minor (Feature extraction section) identified by the color...' -> 'identified by the brightness...'Response: We have revised the sentences as in 1).__________________________________________________________________________Minor (Support Vector Machines) 'Maximum confidence strategy was used...' - how the confidence was estimated? SVM provides binary result out of the box, the actual confidence assessment approaches vary.Response: The confidence is directly given by the decision function, which is the signed distance between the tested sample and the separating hyperplane. More precisely, the decision function is computed as ∑_(i∈SV)▒〖y_i α_i K+b〗, where x is the sample to predict; x_i are the support vectors (SV) that construct the hyperplane; b is the intercept; α_i are the dual coefficients computed in the SVM optimization problem; y_i are the binary labels corresponding to x_i; K=φ(x_i )^T φ(x) is the employed kernel.In the OVA-strategy, we generate 10 classifiers: the first separates class 1 from the remaining ones, the second does the same for class 2, and so on for all 10 classes. The decision function is computed for each class as reported in the previous equation. The classifier that returns the largest positive value identifies the winner class.AttachmentResponse to Reviewers.docxSubmitted filename: Click here for additional data file. 15 Dec 2020Automatic classification of mice vocalizations using Machine Learning techniques and Convolutional Neural NetworksPONE-D-20-09612R2Dear Dr. Premoli,We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.Please, consider minor edits suggested by one of the reviewers.Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at onepress@plos.org.If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact Kind regards,Gennady Cymbalyuk, Ph.D.Academic EditorPLOS ONEAdditional Editor Comments :Reviewers' comments:Reviewer's Responses to QuestionsComments to the Author1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.Reviewer #1: All comments have been addressedReviewer #2: All comments have been addressed**********2. Is the manuscript technically sound, and do the data support the conclusions?The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #1: YesReviewer #2: Yes**********3. Has the statistical analysis been performed appropriately and rigorously? Reviewer #1: YesReviewer #2: Yes**********4. Have the authors made all data underlying the findings in their manuscript fully available?PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.The Reviewer #1: YesReviewer #2: Yes**********5. Is the manuscript presented in an intelligible fashion and written in standard English?PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.Reviewer #1: YesReviewer #2: Yes**********6. Review Comments to the AuthorPlease use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. Reviewer #1: The authors carefully addressed all the comments given in the previous review round. I reccommend acceptance of the paper.Some minor grammar/stylistic/logic errors could be considered:line 391 : generating to -> generating of/forline 432: the CNN architecture has outperformed the other standard features-based methods ->the CNN architecture has outperformed the standard features-based methods (CNN is not a feature-based method in the paper ...)line 455: are arranged on the ten columns -> are arranged in ten columnsline 458 : The values into the matrices are normalized -> the values in the matrices ....line 463, 502: on the x axis there are the predicted labels : "on x axis there are..." sounds a bit ungrammatical to me; usually they use something like "x axis refers to ... " to describe the meaning of axesline 519: The performance showed that by exploiting the whole time/frequency information of the spectrogramleads to significantly higher performance than considering a subset->The performance showed that the exploiting of the whole time/frequency information of the spectrogramleads to significantly higher performance than considering only a subset of numerical featuresline 525: The final set up on an automatic classification method willdefinitely solve the current main problems in USVs manual classification: long time consuming andoperator-dependent.->The final set up of(?) an automatic classification method willdefinitely solve the current main problems of USVs manual classification: its being a time consuming process andoperator bias.Reviewer #2: The authors have done a good job responding to the additional reviewers comments. I recommended to accept the paper in the previous revision.**********what does this mean?). If published, this will include your full peer review and any attached files.7. PLOS authors have the option to publish the peer review history of their article (If you choose “no”, your identity will remain anonymous but your review may still be made public.Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.Yes: Aleksandr IvanenkoReviewer #1: Yes: Kevin CoffeyReviewer #2:  7 Jan 2021PONE-D-20-09612R2 Automatic classification of mice vocalizations using Machine Learning techniques and Convolutional Neural Networks Dear Dr. Premoli:I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department. onepress@plos.org.If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact plosone@plos.org. If we can help with anything else, please email us at Thank you for submitting your work to PLOS ONE and supporting open access. Kind regards, PLOS ONE Editorial Office Staffon behalf ofDr. Gennady Cymbalyuk Academic EditorPLOS ONE