data/users/nshah/seedset-nshah-maple.json

{
  "username": "nshah",
  "s2_authorid": "1737249",
  "papers": [
    {
      "title": "Time is Money: Strategic Timing Games in Proof-of-Stake Protocols",
      "abstract": [
        "We propose a model suggesting that honest-but-rational consensus participants may play timing games, and strategically delay their block proposal to optimize MEV capture, while still ensuring the proposal's timely inclusion in the canonical chain.",
        "In this context, ensuring economic fairness among consensus participants is critical to preserving decentralization.",
        "We contend that a model grounded in honest-but-rational consensus participation provides a more accurate portrayal of behavior in economically incentivized systems such as blockchain protocols.",
        "We empirically investigate timing games on the Ethereum network and demonstrate that while timing games are worth playing, they are not currently being exploited by consensus participants.",
        "By quantifying the marginal value of time, we uncover strong evidence pointing towards their future potential, despite the limited exploitation of MEV capture observed at present."
      ]
    },
    {
      "title": "A Gold Standard Dataset for the Reviewer Assignment Problem",
      "abstract": [
        "Many peer-review venues are either using or looking to use algorithms to assign submissions to reviewers.",
        "The crux of such automated approaches is the notion of the\"similarity score\"--a numerical estimate of the expertise of a reviewer in reviewing a paper--and many algorithms have been proposed to compute these scores.",
        "However, these algorithms have not been subjected to a principled comparison, making it difficult for stakeholders to choose the algorithm in an evidence-based manner.",
        "The key challenge in comparing existing algorithms and developing better algorithms is the lack of the publicly available gold-standard data that would be needed to perform reproducible research.",
        "We address this challenge by collecting a novel dataset of similarity scores that we release to the research community.",
        "Our dataset consists of 477 self-reported expertise scores provided by 58 researchers who evaluated their expertise in reviewing papers they have read previously.",
        "We use this data to compare several popular algorithms employed in computer science conferences and come up with recommendations for stakeholders.",
        "Our main findings are as follows.",
        "First, all algorithms make a non-trivial amount of error.",
        "For the task of ordering two papers in terms of their relevance for a reviewer, the error rates range from 12%-30% in easy cases to 36%-43% in hard cases, highlighting the vital need for more research on the similarity-computation problem.",
        "Second, most existing algorithms are designed to work with titles and abstracts of papers, and in this regime the Specter+MFR algorithm performs best.",
        "Third, to improve performance, it may be important to develop modern deep-learning based algorithms that can make use of the full texts of papers: the classical TD-IDF algorithm enhanced with full texts of papers is on par with the deep-learning based Specter+MFR that cannot make use of this information."
      ]
    },
    {
      "title": "ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing",
      "abstract": [
        "Given the rapid ascent of large language models (LLMs), we study the question: (How) can large language models help in reviewing of scientific papers or proposals?",
        "We first conduct some pilot studies where we find that (i) GPT-4 outperforms other LLMs (Bard, Vicuna, Koala, Alpaca, LLaMa, Dolly, OpenAssistant, StableLM), and (ii) prompting with a specific question (e.g., to identify errors) outperforms prompting to simply write a review.",
        "With these insights, we study the use of LLMs (specifically, GPT-4) for three tasks: 1.",
        "Identifying errors: We construct 13 short computer science papers each with a deliberately inserted error, and ask the LLM to check for the correctness of these papers.",
        "We observe that the LLM finds errors in 7 of them, spanning both mathematical and conceptual errors.",
        "2.",
        "Verifying checklists: We task the LLM to verify 16 closed-ended checklist questions in the respective sections of 15 NeurIPS 2022 papers.",
        "We find that across 119 {checklist question, paper} pairs, the LLM had an 86.6% accuracy.",
        "3.",
        "Choosing the\"better\"paper: We generate 10 pairs of abstracts, deliberately designing each pair in such a way that one abstract was clearly superior than the other.",
        "The LLM, however, struggled to discern these relatively straightforward distinctions accurately, committing errors in its evaluations for 6 out of the 10 pairs.",
        "Based on these experiments, we think that LLMs have a promising use as reviewing assistants for specific reviewing tasks, but not (yet) for complete evaluations of papers or proposals."
      ]
    },
    {
      "title": "Assisting Human Decisions in Document Matching",
      "abstract": [
        "Many practical applications, ranging from paper-reviewer assignment in peer review to job-applicant matching for hiring, require human decision makers to identify relevant matches by combining their expertise with predictions from machine learning models.",
        "In many such model-assisted document matching tasks, the decision makers have stressed the need for assistive information about the model outputs (or the data) to facilitate their decisions.",
        "In this paper, we devise a proxy matching task that allows us to evaluate which kinds of assistive information improve decision makers' performance (in terms of accuracy and time).",
        "Through a crowdsourced (N=271 participants) study, we find that providing black-box model explanations reduces users' accuracy on the matching task, contrary to the commonly-held belief that they can be helpful by allowing better understanding of the model.",
        "On the other hand, custom methods that are designed to closely attend to some task-specific desiderata are found to be effective in improving user performance.",
        "Surprisingly, we also find that the users' perceived utility of assistive information is misaligned with their objective utility (measured through their task performance)."
      ]
    },
    {
      "title": "Counterfactual Evaluation of Peer-Review Assignment Policies",
      "abstract": [
        "Peer review assignment algorithms aim to match research papers to suitable expert reviewers, working to maximize the quality of the resulting reviews.",
        "A key challenge in designing effective assignment policies is evaluating how changes to the assignment algorithm map to changes in review quality.",
        "In this work, we leverage recently proposed policies that introduce randomness in peer-review assignment--in order to mitigate fraud--as a valuable opportunity to evaluate counterfactual assignment policies.",
        "Specifically, we exploit how such randomized assignments provide a positive probability of observing the reviews of many assignment policies of interest.",
        "To address challenges in applying standard off-policy evaluation methods, such as violations of positivity, we introduce novel methods for partial identification based on monotonicity and Lipschitz smoothness assumptions for the mapping between reviewer-paper covariates and outcomes.",
        "We apply our methods to peer-review data from two computer science venues: the TPDP'21 workshop (95 papers and 35 reviewers) and the AAAI'22 conference (8,450 papers and 3,145 reviewers).",
        "We consider estimates of (i) the effect on review quality when changing weights in the assignment algorithm, e.g., weighting reviewers' bids vs. textual similarity (between the review's past papers and the submission), and (ii) the\"cost of randomization\", capturing the difference in expected quality between the perturbed and unperturbed optimal match.",
        "We find that placing higher weight on text similarity results in higher review quality and that introducing randomization in the reviewer-paper assignment only marginally reduces the review quality.",
        "Our methods for partial identification may be of independent interest, while our off-policy approach can likely find use evaluating a broad class of algorithmic matching systems."
      ]
    },
    {
      "title": "Testing for Reviewer Anchoring in Peer Review: A Randomized Controlled Trial",
      "abstract": [
        "Peer review frequently follows a process where reviewers first provide initial reviews, authors respond to these reviews, then reviewers update their reviews based on the authors' response.",
        "There is mixed evidence regarding whether this process is useful, including frequent anecdotal complaints that reviewers insufficiently update their scores.",
        "In this study, we aim to investigate whether reviewers anchor to their original scores when updating their reviews, which serves as a potential explanation for the lack of updates in reviewer scores.",
        "We design a novel randomized controlled trial to test if reviewers exhibit anchoring.",
        "In the experimental condition, participants initially see a flawed version of a paper that is later corrected, while in the control condition, participants only see the correct version.",
        "We take various measures to ensure that in the absence of anchoring, reviewers in the experimental group should revise their scores to be identically distributed to the scores from the control group.",
        "Furthermore, we construct the reviewed paper to maximize the difference between the flawed and corrected versions, and employ deception to hide the true experiment purpose.",
        "Our randomized controlled trial consists of 108 researchers as participants.",
        "First, we find that our intervention was successful at creating a difference in perceived paper quality between the flawed and corrected versions: Using a permutation test with the Mann-Whitney U statistic, we find that the experimental group's initial scores are lower than the control group's scores in both the Evaluation category (Vargha-Delaney A=0.64, p=0.0096) and Overall score (A=0.59, p=0.058).",
        "Next, we test for anchoring by comparing the experimental group's revised scores with the control group's scores.",
        "We find no significant evidence of anchoring in either the Overall (A=0.50, p=0.61) or Evaluation category (A=0.49, p=0.61)."
      ]
    },
    {
      "title": "Batching of Tasks by Users of Pseudonymous Forums: Anonymity Compromise and Protection",
      "abstract": [
        "In a number of applications where anonymity is critical, users act under pseudonyms to preserve their privacy.",
        "For instance, in scientific peer review using forums like OpenReview.net, reviewers make comments on papers that are publicly viewable.",
        "Reviewers who have been assigned multiple papers operate under different pseudonyms across their papers to remain anonymous.",
        "Other examples of publicly visible tasks where users operate under pseudonyms include Wikipedia editing and cryptocurrency transactions.",
        "In these settings, it is common for users to engage in batching - the completion of several similar tasks at the same time.",
        "Batching occurs both due to natural bursts in activity (e.g., a person visits a website and makes many comments at once) or as a productivity strategy used to streamline work.",
        "In peer-review forums such as computer science conferences, reviewers and meta-reviewers are often assigned multiple papers.",
        "We find empirically that reviewers are highly likely to batch their comments and/or reviews across papers.",
        "In analysis of data from a top Computer Science conference with thousands of papers, reviewers, and discussion comments we find that when reviewers and meta-reviewers comment on multiple papers, they have a 30.10% chance of batching their comments within 5 minutes of one other.",
        "In comparison, any randomly chosen pair of reviewers and meta- reviewers had only a 0.66% chance of making comments on different papers within 5 minutes of each other."
      ]
    },
    {
      "title": "To ArXiv or not to ArXiv: A Study Quantifying Pros and Cons of Posting Preprints Online",
      "abstract": [
        "Double-blind conferences have engaged in debates over whether to allow authors to post their papers online on arXiv or elsewhere during the review process.",
        "Independently, some authors of research papers face the dilemma of whether to put their papers on arXiv due to its pros and cons.",
        "We conduct a study to substantiate this debate and dilemma via quantitative measurements.",
        "Specifically, we conducted surveys of reviewers in two top-tier double-blind computer science conferences -- ICML 2021 (5361 submissions and 4699 reviewers) and EC 2021 (498 submissions and 190 reviewers).",
        "Our two main findings are as follows.",
        "First, more than a third of the reviewers self-report searching online for a paper they are assigned to review.",
        "Second, outside the review process, we find that preprints from better-ranked affiliations see a weakly higher visibility, with a correlation of 0.06 in ICML and 0.05 in EC.",
        "In particular, papers associated with the top-10-ranked affiliations had a visibility of approximately 11% in ICML and 22% in EC, whereas the remaining papers had a visibility of 7% and 18% respectively."
      ]
    },
    {
      "title": "Strategyproofing Peer Assessment via Partitioning: The Price in Terms of Evaluators' Expertise",
      "abstract": [
        "Strategic behavior is a fundamental problem in a variety of real-world applications that require some form of peer assessment, such as peer grading of homeworks, grant proposal review, conference peer review of scientific papers, and peer assessment of employees in organizations.",
        "Since an individual's own work is in competition with the submissions they are evaluating, they may provide dishonest evaluations to increase the relative standing of their own submission.",
        "This issue is typically addressed by partitioning the individuals and assigning them to evaluate the work of only those from different subsets.",
        "Although this method ensures strategyproofness, each submission may require a different type of expertise for effective evaluation.",
        "In this paper, we focus on finding an assignment of evaluators to submissions that maximizes assigned evaluators' expertise subject to the constraint of strategyproofness.",
        "We analyze the price of strategyproofness: that is, the amount of compromise on the assigned evaluators' expertise required in order to get strategyproofness.",
        "We establish several polynomial-time algorithms for strategyproof assignment along with assignment-quality guarantees.",
        "Finally, we evaluate the methods on a dataset from conference peer review."
      ]
    },
    {
      "title": "Allocation Schemes in Analytic Evaluation: Applicant-Centric Holistic or Attribute-Centric Segmented?",
      "abstract": [
        "Many applications such as hiring and university admissions involve evaluation and selection of applicants.",
        "These tasks are fundamentally difficult, and require combining evidence from multiple different aspects (what we term \"attributes\").",
        "In these applications, the number of applicants is often large, and a common practice is to assign the task to multiple evaluators in a distributed fashion.",
        "Specifically, in the often-used holistic allocation, each evaluator is assigned a subset of the applicants, and is asked to assess all relevant information for their assigned applicants.",
        "However, such an evaluation process is subject to issues such as miscalibration (evaluators see only a small fraction of the applicants and may not get a good sense of relative quality), and discrimination (evaluators are influenced by irrelevant information about the applicants).",
        "We identify that such attribute-based evaluation allows alternative allocation schemes.",
        "Specifically, we consider assigning each evaluator more applicants but fewer attributes per applicant, termed segmented allocation.",
        "We compare segmented allocation to holistic allocation on several dimensions via theoretical and experimental methods.",
        "We establish various tradeoffs between these two approaches, and identify conditions under which one approach results in more accurate evaluation than the other."
      ]
    },
    {
      "title": "Integrating Rankings into Quantized Scores in Peer Review",
      "abstract": [
        "In peer review, reviewers are usually asked to provide scores for the papers.",
        "The scores are then used by Area Chairs or Program Chairs in various ways in the decision-making process.",
        "The scores are usually elicited in a quantized form to accommodate the limited cognitive ability of humans to describe their opinions in numerical values.",
        "It has been found that the quantized scores suffer from a large number of ties, thereby leading to a significant loss of information.",
        "To mitigate this issue, conferences have started to ask reviewers to additionally provide a ranking of the papers they have reviewed.",
        "There are however two key challenges.",
        "First, there is no standard procedure for using this ranking information and Area Chairs may use it in different ways (including simply ignoring them), thereby leading to arbitrariness in the peer-review process.",
        "Second, there are no suitable interfaces for judicious use of this data nor methods to incorporate it in existing workflows, thereby leading to inefficiencies.",
        "We take a principled approach to integrate the ranking information into the scores.",
        "The output of our method is an updated score pertaining to each review that also incorporates the rankings.",
        "Our approach addresses the two aforementioned challenges by: (i) ensuring that rankings are incorporated into the updates scores in the same manner for all papers, thereby mitigating arbitrariness, and (ii) allowing to seamlessly use existing interfaces and workflows designed for scores.",
        "We empirically evaluate our method on synthetic datasets as well as on peer reviews from the ICLR 2017 conference, and find that it reduces the error by approximately 30% as compared to the best performing baseline on the ICLR 2017 data."
      ]
    },
    {
      "title": "No Rose for MLE: Inadmissibility of MLE for Evaluation Aggregation Under Levels of Expertise",
      "abstract": [
        "A number of applications including crowd-sourced labeling and peer review require aggregation of labels or evaluations sourced from multiple evaluators.",
        "There is often additional information available pertaining to the evaluators\u2019 expertise.",
        "A natural approach for aggregation is to consider the widely studied Dawid-Skene model (or its extensions incorporating evaluators\u2019 expertise), and employ the standard maximum likelihood estimator (MLE).",
        "While MLE is in general widely used in practice and enjoys a number of appealing theoretical guarantees, in this work we provide a negative result for the MLE.",
        "Specifically, we prove that the MLE is asymptotically inadmissible for a special case of evaluation aggregation with expertise level information.",
        "We show this by constructing an alternative estimator that we show is significantly better than the MLE in certain parameter regimes and at least as good elsewhere.",
        "Finally, simulations reveal that our findings may hold in more general conditions than what we theoretically analyze."
      ]
    },
    {
      "title": "Addendum and Erratum to \u201cThe MDS Queue: Analysing the Latency Performance of Erasure Codes\u201d",
      "abstract": [
        "In the above article [1], we introduced two scheduling policies and analyzed their average job latencies.",
        "With an implicit assumption that the scheduling policies provide sample-path bounds by construction, we claimed that their average job latencies serve as upper and lower bounds on that of a centralized MDS queue.",
        "In this note, we present recently discovered counterexamples, disproving the assumption.",
        "We replace the assumption with a conjecture that the average latency bounds still hold.",
        "We also provide an erratum to the original article to correct any confusing or misleading statements."
      ]
    },
    {
      "title": "Cite-seeing and reviewing: A study on citation bias in peer review",
      "abstract": [
        "Citations play an important role in researchers\u2019 careers as a key factor in evaluation of scientific impact.",
        "Many anecdotes advice authors to exploit this fact and cite prospective reviewers to try obtaining a more positive evaluation for their submission.",
        "In this work, we investigate if such a citation bias actually exists: Does the citation of a reviewer\u2019s own work in a submission cause them to be positively biased towards the submission?",
        "In conjunction with the review process of two flagship conferences in machine learning and algorithmic economics, we execute an observational study to test for citation bias in peer review.",
        "In our analysis, we carefully account for various confounding factors such as paper quality and reviewer expertise, and apply different modeling techniques to alleviate concerns regarding the model mismatch.",
        "Overall, our analysis involves 1,314 papers and 1,717 reviewers and detects citation bias in both venues we consider.",
        "In terms of the effect size, by citing a reviewer\u2019s work, a submission has a non-trivial chance of getting a higher score from the reviewer: an expected increase in the score is approximately 0.23 on a 5-point Likert item.",
        "For reference, a one-point increase of a score by a single reviewer improves the position of a submission by 11% on average."
      ]
    },
    {
      "title": "Tradeoffs in Preventing Manipulation in Paper Bidding for Reviewer Assignment",
      "abstract": [
        "Many conferences rely on paper bidding as a key component of their reviewer assignment procedure.",
        "These bids are then taken into account when assigning reviewers to help ensure that each reviewer is assigned to suitable papers.",
        "However, despite the benefits of using bids, reliance on paper bidding can allow malicious reviewers to manipulate the paper assignment for unethical purposes (e.g., getting assigned to a friend's paper).",
        "Several different approaches to preventing this manipulation have been proposed and deployed.",
        "In this paper, we enumerate certain desirable properties that algorithms for addressing bid manipulation should satisfy.",
        "We then offer a high-level analysis of various approaches along with directions for future investigation."
      ]
    },
    {
      "title": "The Price of Strategyproofing Peer Assessment",
      "abstract": [
        "Strategic behavior is a fundamental problem in a variety of real-world applications that require some form of peer assessment, such as peer grading of assignments, grant proposal review, conference peer review, and peer assessment of employees.",
        "Since an individual\u2019s own work is in competition with the submissions they are evaluating, they may provide dishonest evaluations to increase the relative standing of their own submission.",
        "This issue is typically addressed by partitioning the individuals and assigning them to evaluate the work of only those from different subsets.",
        "Although this method ensures strategyproofness, each submission may require a different type of expertise for effective evaluation.",
        "In this paper, we focus on finding an assignment of evaluators to submissions that maximizes assigned expertise subject to the constraint of strategyproofness.",
        "We analyze the price of strategyproofness: that is, the amount of compromise on the assignment quality required in order to get strategyproofness.",
        "We establish several polynomial-time algorithms for strategyproof assignment along with assignment-quality guarantees.",
        "Finally, we evaluate the methods on a dataset from conference peer review."
      ]
    },
    {
      "title": "The role of author identities in peer review",
      "abstract": [
        "There is widespread debate on whether to anonymize author identities in peer review.",
        "The key argument for anonymization is to mitigate bias, whereas arguments against anonymization posit various uses of author identities in the review process.",
        "The Innovations in Theoretical Computer Science (ITCS) 2023 conference adopted a middle ground by initially anonymizing the author identities from reviewers, revealing them after the reviewer had submitted their initial reviews, and allowing the reviewer to change their review subsequently.",
        "We present an analysis of the reviews pertaining to the identification and use of author identities.",
        "Our key findings are: (I) A majority of reviewers self-report not knowing and being unable to guess the authors\u2019 identities for the papers they were reviewing. (",
        "II) After the initial submission of reviews, 7.1% of reviews changed their overall merit score and 3.8% changed their self-reported reviewer expertise. (",
        "III) There is a very weak and statistically insignificant correlation of the rank of authors\u2019 affiliations with the change in overall merit; there is a weak but statistically significant correlation with respect to change in reviewer expertise.",
        "We also conducted an anonymous survey to obtain opinions from reviewers and authors.",
        "The main findings from the 200 survey responses are: (i) A vast majority of participants favor anonymizing author identities in some form. (",
        "ii) The \u201cmiddle-ground\u201d initiative of ITCS 2023 was appreciated. (",
        "iii) Detecting conflicts of interest is a challenge that needs to be addressed if author identities are anonymized.",
        "Overall, these findings support anonymization of author identities in some form (e.g., as was done in ITCS 2023), as long as there is a robust and efficient way to check conflicts of interest."
      ]
    },
    {
      "title": "Calibration with Privacy in Peer Review",
      "abstract": [
        "This paper is eligible for the Jack Keil Wolf ISIT Student Paper Award.",
        "Reviewers in peer review are often miscalibrated: they may be strict, lenient, extreme, moderate, etc.",
        "A number of algorithms have previously been proposed to calibrate reviews.",
        "Such attempts of calibration can however leak sensitive information about which reviewer reviewed which paper.",
        "In this paper, we identify this problem of calibration with privacy, and provide a foundational building block to address it.",
        "Specifically, we present a theoretical study of this problem under a simplified-yet-challenging model involving two reviewers, two papers, and an MAP-computing adversary.",
        "Our main results establish the Pareto frontier of the tradeoff between privacy (preventing the adversary from inferring reviewer identity) and utility (accepting better papers), and design explicit computationally-efficient algorithms that we prove are Pareto optimal."
      ]
    },
    {
      "title": "Batching of Tasks by Users of Pseudonymous Forums: Anonymity Compromise and Protection",
      "abstract": [
        "There are a number of forums where people participate under pseudonyms.",
        "One example is peer review, where the identity of reviewers for any paper is confidential.",
        "When participating in these forums, people frequently engage in \"batching\": executing multiple related tasks (e.g., commenting on multiple papers) at nearly the same time.",
        "Our empirical analysis shows that batching is common in two applications we consider -- peer review and Wikipedia edits.",
        "In this paper, we identify and address the risk of deanonymization arising from linking batched tasks.",
        "To protect against linkage attacks, we take the approach of adding delay to the posting time of batched tasks.",
        "We first show that under some natural assumptions, no delay mechanism can provide a meaningful differential privacy guarantee.",
        "We therefore propose a \"one-sided\" formulation of differential privacy for protecting against linkage attacks.",
        "We design a mechanism that adds zero-inflated uniform delay to events and show it can preserve privacy.",
        "We prove that this noise distribution is in fact optimal in minimizing expected delay among mechanisms adding independent noise to each event, thereby establishing the Pareto frontier of the trade-off between the expected delay for batched and unbatched events.",
        "Finally, we conduct a series of experiments on Wikipedia and Bitcoin data that corroborate the practical utility of our algorithm in obfuscating batching without introducing onerous delay to a system."
      ]
    },
    {
      "title": "A Dataset on Malicious Paper Bidding in Peer Review",
      "abstract": [
        "In conference peer review, reviewers are often asked to provide \u201cbids\u201d on each submitted paper that express their interest in reviewing that paper.",
        "A paper assignment algorithm then uses these bids (along with other data) to compute a high-quality assignment of reviewers to papers.",
        "However, this process has been exploited by malicious reviewers who strategically bid in order to unethically manipulate the paper assignment, crucially undermining the peer review process.",
        "For example, these reviewers may aim to get assigned to a friend\u2019s paper as part of a quid-pro-quo deal.",
        "A critical impediment towards creating and evaluating methods to mitigate this issue is the lack of any publicly-available data on malicious paper bidding.",
        "In this work, we collect and publicly release a novel dataset to fill this gap, collected from a mock conference activity where participants were instructed to bid either honestly or maliciously.",
        "We further provide a descriptive analysis of the bidding behavior, including our categorization of different strategies employed by participants.",
        "Finally, we evaluate the ability of each strategy to manipulate the assignment, and also evaluate the performance of some simple algorithms meant to detect malicious bidding.",
        "The performance of these detection algorithms can be taken as a baseline for future research on detecting malicious bidding."
      ]
    },
    {
      "title": "Challenges, experiments, and computational solutions in peer review",
      "abstract": [
        "Improving the peer review process in a scientific manner shows promise."
      ]
    },
    {
      "title": "How do Authors' Perceptions of their Papers Compare with Co-authors' Perceptions and Peer-review Decisions?",
      "abstract": [
        "How do author perceptions match up to the outcomes of the peer-review process and perceptions of others?",
        "In a top-tier computer science conference (NeurIPS 2021) with more than 23,000 submitting authors and 9,000 submitted papers, we survey the authors on three questions: (i) their predicted probability of acceptance for each of their papers, (ii) their perceived ranking of their own papers based on scientific contribution, and (iii) the change in their perception about their own papers after seeing the reviews.",
        "The salient results are: (1) Authors have roughly a three-fold overestimate of the acceptance probability of their papers: The median prediction is 70% for an approximately 25% acceptance rate. (",
        "2) Female authors exhibit a marginally higher (statistically significant) miscalibration than male authors; predictions of authors invited to serve as meta-reviewers or reviewers are similarly calibrated, but better than authors who were not invited to review. (",
        "3) Authors' relative ranking of scientific contribution of two submissions they made generally agree (93%) with their predicted acceptance probabilities, but there is a notable 7% responses where authors think their better paper will face a worse outcome. (",
        "4) The author-provided rankings disagreed with the peer-review decisions about a third of the time; when co-authors ranked their jointly authored papers, co-authors disagreed at a similar rate -- about a third of the time. (",
        "5) At least 30% of respondents of both accepted and rejected papers said that their perception of their own paper improved after the review process.",
        "The stakeholders in peer review should take these findings into account in setting their expectations from peer review."
      ]
    },
    {
      "title": "An Overview of Challenges, Experiments, and Computational Solutions in Peer Review (Extended Version)",
      "abstract": [
        "In this overview article, we survey a number of challenges in peer review, understand these issues and tradeo\ufb00s involved via insightful experiments, and discuss computational solutions proposed in the literature.",
        "The survey is divided into seven parts: mismatched reviewer expertise, dishonest behavior, miscalibration, subjectivity, biases pertaining to author identities, incentives, and norms and policies."
      ]
    },
    {
      "title": "KDD 2021 Tutorial on Systemic Challenges and Solutions on Bias and Unfairness in Peer Review",
      "abstract": [
        "Introduction.",
        "Peer review is a cornerstone of academic practice [1].",
        "The peer review process is highly regarded by the vast majority of researchers and considered by most to be essential to the communication of scholarly research [2\u20134].",
        "However, there is also an overwhelming desire for improvement [2, 4, 5].",
        "Problems in peer review have consequences much beyond the outcome for a specific paper or grant, particularly due to the widespread prevalence of the Matthew effect (\u201crich get richer\u201d) in academia [6].",
        "As noted by [7] \u201can incompetent review may lead to the rejection of the submitted paper, or of the grant application, and the ultimate failure of the career of the author.\u201d (",
        "See also [8, 9].)",
        "The importance of peer review and the urgent need for improvements, behooves research on principled approaches towards addressing problems in peer review, particularly at scale.",
        "In this tutorial, we discuss a number of key challenges in peer review, outline several directions of research on this topic, and also highlight important open problems that we envisage to be exciting to the community.",
        "This document summarizes the contents of the tutorial and provides relevant references."
      ]
    },
    {
      "title": "A heuristic for statistical seriation",
      "abstract": [
        "We study the statistical seriation problem, where the goal is to estimate a matrix whose rows satisfy the same shape constraint after a permutation of the columns.",
        "This is a important classical problem, with close connections to statistical literature in permutation-based models and also has wide applications ranging from archaeology to biology.",
        "Specifically, we consider the case where the rows are monotonically increasing after an unknown permutation of the columns.",
        "Past work has shown that the least-squares estimator is optimal up to logarithmic factors, but efficient algorithms for computing the least-squares estimator remain unknown to date.",
        "We approach this important problem from a heuristic perspective.",
        "Specifically, we replace the combinatorial permutation constraint by a continuous regularization term, and then use projected gradient descent to obtain a local minimum of the non-convex objective.",
        "We show that the attained local minimum is the global minimum in certain special cases under the noiseless setting, and preserves desirable properties under the noisy setting.",
        "Simulation results reveal that our proposed algorithm outperforms prior algorithms when (1) the underlying model is more complex than simplistic parametric assumptions such as low-rankedness, or (2) the signal-to-noise ratio is high.",
        "Under partial observations, the proposed algorithm requires an initialization, and different initializations may lead to different local minima.",
        "We empirically observe that the proposed algorithm yields consistent improvement over the initialization, even though different initializations start with different levels of quality."
      ]
    },
    {
      "title": "JCDL 2021 Tutorial on Systemic Challenges and Computational Solutions on Bias and Unfairness in Peer Review",
      "abstract": [
        "Peer review is the backbone of scientific research and determines the composition of scientific digital libraries.",
        "Any systemic issues in peer review - such as biases or fraud - can systematically affect the resulting scientific digital library as well as any analyses on that library.",
        "They also affect billions of dollars in research grants made via peer review as well as entire careers of researchers.",
        "The tutorial will discuss various systemic issues in peer review via insightful experiments, several computational solutions proposed to address these issues, and a number of important open problems.",
        "A detailed writeup on the topics of this tutorial as well as a complete list of references is available in [1]."
      ]
    },
    {
      "title": "WSDM 2021 Tutorial on Systematic Challenges and Computational Solutions on Bias and Unfairness in Peer Review",
      "abstract": [
        "Peer review is the backbone of scientific research.",
        "Yet peer review is called \"biased,\" \"broken,\" and \"unscientific\" in many scientific disciplines.",
        "This problem is further compounded with the near-exponentially growing number of submissions in various computer science conferences.",
        "Due to the prevalence of \"Matthew effect'' of rich getting richer in academia, any source of unfairness in the peer review system, such as those discussed in this tutorial, can considerably affect the entire career trajectory of (young) researchers.",
        "This tutorial will discuss a number of systemic challenges in peer review such as biases, subjectivity, miscalibration, dishonest behavior, and noise.",
        "For each issue, the tutorial will first present insightful experiments to understand the issue.",
        "Then the tutorial will present computational techniques designed to address these challenges.",
        "Many open problems will be highlighted which are envisaged to be exciting to the WSDM audience, and will lead to significant impact if solved."
      ]
    },
    {
      "title": "A Heuristic for Statistical Seriation (Supplementary Material)",
      "abstract": [
        "Rii\u2032jj\u2032 = Rii\u2032j\u2032j = Ri\u2032ijj\u2032 = Ri\u2032ij\u2032j .",
        "We say that an entry (i, j) does not contribute to the regularizer if Rii\u2032jj\u2032 = 0 for all i\u2032 \u2208 [n] and j\u2032 \u2208 [d].",
        "We say that a row/column does not contribute to the regularizer if none of the entries in the row/column contributes to the regularizer.",
        "We say that (i, i\u2032, j, j\u2032) is a \u201cconflicting quadruple\u201d if we have (Aij \u2212Aij\u2032)(Ai\u2032j \u2212Ai\u2032j\u2032) < 0.",
        "By the definition (7) of the regularizer, an entry (i, j) does not contribute to the regularizer if and only if the quadruple (i, i\u2032, j, j\u2032) is not a conflicting quadruple for each i\u2032 \u2208 [n] and j\u2032 \u2208 [d]."
      ]
    },
    {
      "title": "Near-Optimal Reviewer Splitting in Two-Phase Paper Reviewing and Conference Experiment Design",
      "abstract": [
        "Many scientific conferences employ a two-phase paper review process, where some papers are assigned additional reviewers after the initial reviews are submitted.",
        "Many conferences also design and run experiments on their paper review process, where some papers are assigned reviewers who provide reviews under an experimental condition.",
        "In this paper, we consider the question: how should reviewers be divided between phases or conditions in order to maximize total assignment similarity?",
        "We make several contributions towards answering this question.",
        "First, we prove that when the set of papers requiring additional review is unknown, a simplified variant of this problem is NP-hard.",
        "Second, we empirically show that across several datasets pertaining to real conference data, dividing reviewers between phases/conditions uniformly at random allows an assignment that is nearly as good as the oracle optimal assignment.",
        "This uniformly random choice is practical for both the two-phase and conference experiment design settings.",
        "Third, we provide explanations of this phenomenon by providing theoretical bounds on the suboptimality of this random strategy under certain natural conditions.",
        "From these easily-interpretable conditions, we provide actionable insights to conference program chairs about whether a random reviewer split is suitable for their conference."
      ]
    },
    {
      "title": "SIGMOD 2020 Tutorial on Fairness and Bias in Peer Review and Other Sociotechnical Intelligent Systems",
      "abstract": [
        "Questions of fairness and bias abound in all socially-consequential decisions pertaining to collection and management of data.",
        "Whether designing protocols for peer review of research papers, setting hiring policies, or framing research question in genetics, any data-management decision with the potential to allocate benefits or confer harms raises concerns about who gains or loses that may fail to surface in naively-chosen performance measures.",
        "Data science interacts with these questions in two fundamentally different ways: (i) as the technology driving the very systems responsible for certain social impacts, posing new questions about what it means for such systems to accord with ethical norms and the law; and (ii) as a set of powerful tools for analyzing existing data management systems, e.g., for auditing existing systems for various biases.",
        "This tutorial will tackle both angles on the interaction between technology and society vis-a-vis concerns over fairness and bias, particularly focusing on the collection and management of data.",
        "Our presentation will cover a wide range of disciplinary perspectives with the first part focusing on the social impacts of technology and the formulations of fairness and bias defined via protected characteristics and the second part taking a deep into peer review and distributed human evaluations, to explore other forms of bias, such as that due to subjectivity, miscalibration, and dishonest behavior."
      ]
    },
    {
      "title": "A large scale randomized controlled trial on herding in peer-review discussions",
      "abstract": [
        "Peer review is the backbone of academia and humans constitute a cornerstone of this process, being responsible for reviewing submissions and making the final acceptance/rejection decisions.",
        "Given that human decision-making is known to be susceptible to various cognitive biases, it is important to understand which (if any) biases are present in the peer-review process, and design the pipeline such that the impact of these biases is minimized.",
        "In this work, we focus on the dynamics of discussions between reviewers and investigate the presence of herding behaviour therein.",
        "Specifically, we aim to understand whether reviewers and discussion chairs get disproportionately influenced by the first argument presented in the discussion when (in case of reviewers) they form an independent opinion about the paper before discussing it with others.",
        "In conjunction with the review process of a large, top tier machine learning conference, we design and execute a randomized controlled trial that involves 1,544 papers and 2,797 reviewers with the goal of testing for the conditional causal effect of the discussion initiator\u2019s opinion on the outcome of a paper.",
        "Our experiment reveals no evidence of herding in peer-review discussions.",
        "This observation is in contrast with past work that has documented an undue influence of the first piece of information on the final decision (e.g., anchoring effect) and analyzed herding behaviour in other applications (e.g., financial markets).",
        "Regarding policy implications, the absence of the herding effect suggests that the current status quo of the absence of a unified policy towards discussion initiation does not result in an increased arbitrariness of the resulting decisions."
      ]
    },
    {
      "title": "Ranking and Rating Rankings and Ratings",
      "abstract": [
        "Cardinal scores collected from people are well known to suffer from miscalibrations.",
        "A popular approach to address this issue is to assume simplistic models of miscalibration (such as linear biases) to de-bias the scores.",
        "This approach, however, often fares poorly because people's miscalibrations are typically far more complex and not well understood.",
        "It is widely believed that in the absence of simplifying assumptions on the miscalibration, the only useful information in practice from the cardinal scores is the induced ranking.",
        "In this paper we address the fundamental question of whether this widespread folklore belief is actually true.",
        "We consider cardinal scores with arbitrary (or even adversarially chosen) miscalibrations that is only required to be consistent with the induced ranking.",
        "We design rating-based estimators and prove that despite making no assumptions on the ratings, they strictly and uniformly outperform all possible estimators that rely on only the ranking.",
        "These estimators can be used as a plug-in to show the superiority of cardinal scores over ordinal rankings for a variety of applications, including A/B testing and ranking.",
        "This work thus provides novel fundamental insights in the eternal debate between cardinal and ordinal data: It ranks the approach of using ratings higher than that of using rankings, and rates both approaches in terms of their estimation errors."
      ]
    },
    {
      "title": "Prior and Prejudice",
      "abstract": [
        "Modern machine learning and computer science conferences are experiencing a surge in the number of submissions that challenges the quality of peer review as the number of competent reviewers is growing at a much slower rate.",
        "To curb this trend and reduce the burden on reviewers, several conferences have started encouraging or even requiring authors to declare the previous submission history of their papers.",
        "Such initiatives have been met with skepticism among authors, who raise the concern about a potential bias in reviewers' recommendations induced by this information.",
        "In this work, we investigate whether reviewers exhibit a bias caused by the knowledge that the submission under review was previously rejected at a similar venue, focusing on a population of novice reviewers who constitute a large fraction of the reviewer pool in leading machine learning and computer science conferences.",
        "We design and conduct a randomized controlled trial closely replicating the relevant components of the peer-review pipeline with $133$ reviewers (master's, junior PhD students, and recent graduates of top US universities) writing reviews for $19$ papers.",
        "The analysis reveals that reviewers indeed become negatively biased when they receive a signal about paper being a resubmission, giving almost 1 point lower overall score on a 10-point Likert item (\u0394 = -0.78, 95% CI = [-1.30, -0.24]) than reviewers who do not receive such a signal.",
        "Looking at specific criteria scores (originality, quality, clarity and significance), we observe that novice reviewers tend to underrate quality the most."
      ]
    },
    {
      "title": "Research Paper: Loss Functions, Axioms, and Peer Review",
      "abstract": [
        "It is common to see a handful of reviewers reject a highly novel paper, because they view, say, extensive experiments as far more important than novelty, whereas the community as a whole would have embraced the paper.",
        "More generally, the disparate mapping of criteria scores to final recommendations by different reviewers is a major source of inconsistency in peer review.",
        "In this paper we present a framework inspired by empirical risk minimization (ERM) for learning the community\u2019s aggregate mapping.",
        "The key challenge that arises is the specification of a loss function for ERM.",
        "We consider the class of L(p, q) loss functions, which is a matrix-extension of the standard class of Lp losses on vectors; here the choice of the loss function amounts to choosing the hyperparameters p, q \u2208 [1,\u221e].",
        "To deal with the absence of ground truth in our problem, we instead draw on computational social choice to identify desirable values of the hyperparameters p and q. Specifically, we characterize p = q = 1 as the only choice of these hyperparameters that satisfies three natural axiomatic properties.",
        "Finally, we implement and apply our approach to reviews from IJCAI 2017."
      ]
    },
    {
      "title": "AAAI 2020 Tutorial on Fairness and Bias in Peer Review and other Sociotechnical Intelligent Systems (Part II on Peer Review)",
      "abstract": [
        "Peer review is the backbone of scholarly research, but it faces a number of challenges pertaining to bias and unfairness.",
        "There is an urgent need to improve peer review.",
        "This AAAI tutorial (part 2) discusses several problems, empirical studies, proposed solutions, and open problems in this domain.",
        "This document serves to provide a summary and references for the"
      ]
    },
    {
      "title": "Mitigating Manipulation in Peer Review via Randomized Reviewer Assignments",
      "abstract": [
        "We consider three important challenges in conference peer review: (i) reviewers maliciously attempting to get assigned to certain papers to provide positive reviews, possibly as part of quid-pro-quo arrangements with the authors; (ii) \"torpedo reviewing,\" where reviewers deliberately attempt to get assigned to certain papers that they dislike in order to reject them; (iii) reviewer de-anonymization on release of the similarities and the reviewer-assignment code.",
        "On the conceptual front, we identify connections between these three problems and present a framework that brings all these challenges under a common umbrella.",
        "We then present a (randomized) algorithm for reviewer assignment that can optimally solve the reviewer-assignment problem under any given constraints on the probability of assignment for any reviewer-paper pair.",
        "We further consider the problem of restricting the joint probability that certain suspect pairs of reviewers are assigned to certain papers, and show that this problem is NP-hard for arbitrary constraints on these joint probabilities but efficiently solvable for a practical special case.",
        "Finally, we experimentally evaluate our algorithms on datasets from past conferences, where we observe that they can limit the chance that any malicious reviewer gets assigned to their desired paper to 50% while producing assignments with over 90% of the total optimal similarity.",
        "Our algorithms still achieve this similarity while also preventing reviewers with close associations from being assigned to the same paper."
      ]
    },
    {
      "title": "Approval Voting and Incentives in Crowdsourcing",
      "abstract": [
        "The growing need for labeled training data has made crowdsourcing a vital tool for developing machine learning applications.",
        "Here, workers on a crowdsourcing platform are typically shown a list of unlabeled items, and for each of these items, are asked to choose a label from one of the provided options.",
        "The workers in crowdsourcing platforms are not experts, thereby making it essential to judiciously elicit the information known to the workers.",
        "With respect to this goal, there are two key shortcomings of current systems: (i) the incentives of the workers are not aligned with those of the requesters; and (ii) the interface does not allow workers to convey their knowledge accurately by forcing them to make a single choice among a set of options.",
        "In this article, we address these issues by introducing approval voting to utilize the expertise of workers who have partial knowledge of the true answer and coupling it with two strictly proper scoring rules.",
        "We additionally establish attractive properties of optimality and uniqueness of our scoring rules.",
        "We also conduct preliminary empirical studies on Amazon Mechanical Turk, and the results of these experiments validate our approach."
      ]
    },
    {
      "title": "Two-Sample Testing on Ranked Preference Data and the Role of Modeling Assumptions",
      "abstract": [
        "A number of applications require two-sample testing on ranked preference data.",
        "For instance, in crowdsourcing, there is a long-standing question of whether pairwise comparison data provided by people is distributed similar to ratings-converted-to-comparisons.",
        "Other examples include sports data analysis and peer grading.",
        "In this paper, we design two-sample tests for pairwise comparison data and ranking data.",
        "For our two-sample test for pairwise comparison data, we establish an upper bound on the sample complexity required to correctly distinguish between the distributions of the two sets of samples.",
        "Our test requires essentially no assumptions on the distributions.",
        "We then prove complementary lower bounds showing that our results are tight (in the minimax sense) up to constant factors.",
        "We investigate the role of modeling assumptions by proving lower bounds for a range of pairwise comparison models (WST, MST,SST, parameter-based such as BTL and Thurstone).",
        "We also provide testing algorithms and associated sample complexity bounds for the problem of two-sample testing with partial (or total) ranking data.",
        "Furthermore, we empirically evaluate our results via extensive simulations as well as two real-world datasets consisting of pairwise comparisons.",
        "By applying our two-sample test on real-world pairwise comparison data, we conclude that ratings and rankings provided by people are indeed distributed differently.",
        "On the other hand, our test recognizes no significant difference in the relative performance of European football teams across two seasons.",
        "Finally, we apply our two-sample test on a real-world partial and total ranking dataset and find a statistically significant difference in Sushi preferences across demographic divisions based on gender, age and region of residence."
      ]
    },
    {
      "title": "TheWebConf 2020 Tutorial on Fairness and Bias in Peer Review and other Sociotechnical Intelligent Systems (Part II on Peer Review)",
      "abstract": [
        "Peer review is the backbone of scholarly research, but it faces a number of challenges pertaining to bias and unfairness.",
        "There is an urgent need to improve peer review.",
        "This TheWebConf tutorial (part 2) discusses several problems, empirical studies, proposed solutions, and open problems in this domain.",
        "This document serves to provide a summary and references for the tutorial."
      ]
    },
    {
      "title": "Catch Me if I Can: Detecting Strategic Behaviour in Peer Assessment",
      "abstract": [
        "We consider the issue of strategic behaviour in various peer-assessment tasks, including peer grading of exams or homeworks and peer review in hiring or promotions.",
        "When a peer-assessment task is competitive (e.g., when students are graded on a curve), agents may be incentivized to misreport evaluations in order to improve their own final standing.",
        "Our focus is on designing methods for detection of such manipulations.",
        "Specifically, we consider a setting in which agents evaluate a subset of their peers and output rankings that are later aggregated to form a final ordering.",
        "In this paper, we investigate a statistical framework for this problem and design a principled test for detecting strategic behaviour.",
        "We prove that our test has strong false alarm guarantees and evaluate its detection ability in practical settings.",
        "For this, we design and conduct an experiment that elicits strategic behaviour from subjects and release a dataset of patterns of strategic behaviour that may be of independent interest.",
        "We use this data to run a series of real and semi-synthetic evaluations that reveal a strong detection power of our test."
      ]
    },
    {
      "title": "Two-Sample Testing on Pairwise Comparison Data and the Role of Modeling Assumptions",
      "abstract": [
        "A number of applications require two-sample testing of pairwise comparison data.",
        "For instance, in crowdsourcing, there is a long-standing question of whether comparison data provided by people is distributed similar to ratings-converted-to-comparisons.",
        "Other examples include sports data analysis and peer grading.",
        "In this paper, we design a two-sample test for pairwise comparison data.",
        "We establish an upper bound on the sample complexity required to correctly distinguish between the distributions of the two sets of samples.",
        "Our test requires essentially no assumptions on the distributions.",
        "We then prove complementary information-theoretic lower bounds showing that our results are tight (in the minimax sense) up to constant factors.",
        "We also investigate the role of modeling assumptions by proving information-theoretic lower bounds for a range of pairwise comparison models (WST, MST, SST, parameter-based such as BTL and Thurstone)."
      ]
    },
    {
      "title": "Uncovering Latent Biases in Text: Method and Application to Peer Review",
      "abstract": [
        "Quantifying systematic disparities in numerical quantities such as employment rates and wages between population subgroups provides compelling evidence for the existence of societal biases.",
        "However, biases in the text written for members of different subgroups (such as in recommendation letters for male and non-male candidates), though widely reported anecdotally, remain challenging to quantify.",
        "In this work, we introduce a novel framework to quantify bias in text caused by the visibility of subgroup membership indicators.",
        "We develop a nonparametric estimation and inference procedure to estimate this bias.",
        "We then formalize an identification strategy to causally link the estimated bias to the visibility of subgroup membership indicators, provided observations from time periods both before and after an identity-hiding policy change.",
        "We identify an application wherein \u201cground truth\u201d bias can be inferred to evaluate our framework, instead of relying on synthetic or secondary data.",
        "Specifically, we apply our framework to quantify biases in the text of peer reviews from a reputed machine-learning conference before and after the conference adopted a double-blind reviewing policy.",
        "We show evidence of biases in the review ratings that serves as \u201cground truth\u201d, and show that our proposed framework accurately detects the presence (and absence) of these biases from the review text without having access to the review ratings."
      ]
    },
    {
      "title": "A SUPER* Algorithm to Optimize Paper Bidding in Peer Review",
      "abstract": [
        "A number of applications involve sequential arrival of users, and require showing each user an ordering of items.",
        "A prime example (which forms the focus of this paper) is the bidding process in conference peer review where reviewers enter the system sequentially, each reviewer needs to be shown the list of submitted papers, and the reviewer then \"bids\" to review some papers.",
        "The order of the papers shown has a significant impact on the bids due to primacy effects.",
        "In deciding on the ordering of papers to show, there are two competing goals: (i) obtaining sufficiently many bids for each paper, and (ii) satisfying reviewers by showing them relevant items.",
        "In this paper, we begin by developing a framework to study this problem in a principled manner.",
        "We present an algorithm called SUPER*, inspired by the A* algorithm, for this goal.",
        "Theoretically, we show a local optimality guarantee of our algorithm and prove that popular baselines are considerably suboptimal.",
        "Moreover, under a community model for the similarities, we prove that SUPER* is near-optimal whereas the popular baselines are considerably suboptimal.",
        "In experiments on real data from ICLR 2018 and synthetic data, we find that SUPER* considerably outperforms baselines deployed in existing systems, consistently reducing the number of papers with fewer than requisite bids by 50-75% or more, and is also robust to various real world complexities."
      ]
    },
    {
      "title": "On the Privacy-Utility Tradeoff in Peer-Review Data Analysis",
      "abstract": [
        "A major impediment to research on improving peer review is the unavailability of peer-review data, since any release of such data must grapple with the sensitivity of the peer review data in terms of protecting identities of reviewers from authors.",
        "We posit the need to develop techniques to release peer-review data in a privacy-preserving manner.",
        "Identifying this problem, in this paper we propose a framework for privacy-preserving release of certain conference peer-review data -- distributions of ratings, miscalibration, and subjectivity -- with an emphasis on the accuracy (or utility) of the released data.",
        "The crux of the framework lies in recognizing that a part of the data pertaining to the reviews is already available in public, and we use this information to post-process the data released by any privacy mechanism in a manner that improves the accuracy (utility) of the data while retaining the privacy guarantees.",
        "Our framework works with any privacy-preserving mechanism that operates via releasing perturbed data.",
        "We present several positive and negative theoretical results, including a polynomial-time algorithm for improving on the privacy-utility tradeoff."
      ]
    },
    {
      "title": "A Novice-Reviewer Experiment to Address Scarcity of Qualified Reviewers in Large Conferences",
      "abstract": [
        "Conference peer review constitutes a human-computation process whose importance cannot be overstated: not only it identifies the best submissions for acceptance, but, ultimately, it impacts the future of the whole research area by promoting some ideas and restraining others.",
        "A surge in the number of submissions received by leading AI conferences has challenged the sustainability of the review process by increasing the burden on the pool of qualified reviewers which is growing at a much slower rate.",
        "In this work, we consider the problem of reviewer recruiting with a focus on the scarcity of qualified reviewers in large conferences.",
        "Specifically, we design a procedure for (i) recruiting reviewers from the population not typically covered by major conferences and (ii) guiding them through the reviewing pipeline.",
        "In conjunction with the ICML 2020 --- a large, top-tier machine learning conference --- we recruit a small set of reviewers through our procedure and compare their performance with the general population of ICML reviewers.",
        "Our experiment reveals that a combination of the recruiting and guiding mechanisms allows for a principled enhancement of the reviewer pool and results in reviews of superior quality compared to the conventional pool of reviews as evaluated by senior members of the program committee (meta-reviewers)."
      ]
    },
    {
      "title": "On Testing for Biases in Peer Review",
      "abstract": [
        "We consider the issue of biases in scholarly research, specifically, in peer review.",
        "There is a long standing debate on whether exposing author identities to reviewers induces biases against certain groups, and our focus is on designing tests to detect the presence of such biases.",
        "Our starting point is a remarkable recent work by Tomkins, Zhang and Heavlin which conducted a controlled, large-scale experiment to investigate existence of biases in the peer reviewing of the WSDM conference.",
        "We present two sets of results in this paper.",
        "The first set of results is negative, and pertains to the statistical tests and the experimental setup used in the work of Tomkins et al. We show that the test employed therein does not guarantee control over false alarm probability and under correlations between relevant variables, coupled with any of the following conditions, with high probability can declare a presence of bias when it is in fact absent: (a) measurement error, (b) model mismatch, (c) reviewer calibration.",
        "Moreover, we show that the setup of their experiment may itself inflate false alarm probability if (d) bidding is performed in non-blind manner or (e) popular reviewer assignment procedure is employed.",
        "Our second set of results is positive, in that we present a general framework for testing for biases in (single vs. double blind) peer review.",
        "We then present a hypothesis test with guaranteed control over false alarm probability and non-trivial power even under conditions (a)--(c).",
        "Conditions (d) and (e) are more fundamental problems that are tied to the experimental setup and not necessarily related to the test."
      ]
    },
    {
      "title": "The Square-Root Agreement Rule for Incentivizing Objective Feedback in Online Platforms",
      "abstract": [
        "A major challenge in obtaining evaluations of products or services on e-commerce platforms is that of eliciting informative feedback in the absence of verifiability.",
        "We propose a simple incentive mechanism for obtaining objective feedback on such platforms.",
        "In this mechanism, an agent gets a reward only if her answer for an evaluation matches that of her peer, where this reward is inversely proportional to a popularity index of each answer.",
        "This index is defined to be the square-root of the empirical frequency at which any two agents performing the same evaluation agree on the particular answer.",
        "Rarely agreed-upon answers thus earn a higher reward than answers for which agreements are relatively more common.",
        "We call this mechanism the Square-Root Agreement Rule (SRA).",
        "\n \nA key feature of platforms that SRA leverages is the existence of a large number of similar entities to be evaluated (e.g., restaurants, sellers, services, etc.);",
        "in this regime, we show that truthful behavior is a strict Bayes-Nash equilibrium of the game induced by SRA.",
        "Further, as the number of evaluation tasks across the platform grows, this equilibrium is asymptotically optimal for the agents across all symmetric equilibria.",
        "Moreover, under a mild condition, we show that any symmetric equilibrium that gives a higher expected payoff to the agents than the truthful equilibrium must be close to being fully informative when the number of evaluations is large.",
        "SRA can thus be an effective approach for administering reward-based incentive schemes (e.g., rebates, reputation score, etc.)",
        "on these platforms."
      ]
    },
    {
      "title": "HITSnDIFFS : A fast algorithm for consecutive ones with applications in item labeling",
      "abstract": [
        "We analyze a general problem in a crowd-sourced setting: users pick a label from a set of candidates for a set of items; the problem is to determine the most likely label for each item, as well as a ranking of the users based on their ability to pick correct labels for the items.",
        "We start by defining an idealized setting for this problem where the relative performance of users is consistent across items, and observe that the response matrices in this ideal case obey the Consecutive Ones Property (C1P).",
        "While the consecutive ones problem is well understood algorithmically with various discrete algorithms, we devise a simple variant of the HITS algorithm called \u201cHITSnDIFFS\u201d and prove that it can recover the ideal C1P-permutation in case it exists.",
        "Unlike fast combinatorial algorithms for finding the consecutive ones permutation (if it exists), HITSnDIFFS also returns an ordering when such a permutation does not exist, thus providing a principled heuristic for the problem that returns the correct answer in the ideal case.",
        "We compare HITSnDIFFS\u2019s performance with previously proposed iterative and spectral algorithms to solve similar real-world problems.",
        "Our experiments on both real and simulated datasets show that HITSnDIFFS produces user rankings and item labelings with superior accuracy compared to the various scalable methods, and is competitive with other slower stateof-the-art methods while providing an asymptotic improvement in running time."
      ]
    },
    {
      "title": "Principled Methods to Improve Peer Review",
      "abstract": [
        "There is an urgent need to improve peer review, particularly due to the explosion in the number of submissions especially at ML and AI venues.",
        "Peer review faces a number of challenges including noise, calibration, subjectivity, and strategic behavior.",
        "This paper presents a survey of our recent works towards addressing these challenges.",
        "Our works take a principled approach to tackle these issues, towards developing an algorithmic toolkit for improved peer-review processes.",
        "Our algorithms focus on achieving objectives of fairness, accuracy, and robustness in these goals.",
        "We supplement our algorithms with strong theoretical guarantees as well as empirical evaluations on conference data.",
        "The ideas, results, and insights of this work as applicable broadly to a variety of applications beyond peer review."
      ]
    },
    {
      "title": "Stretching the Effectiveness of MLE from Accuracy to Bias for Pairwise Comparisons",
      "abstract": [
        "A number of applications (e.g., AI bot tournaments, sports, peer grading, crowdsourcing) use pairwise comparison data and the Bradley-Terry-Luce (BTL) model to evaluate a given collection of items (e.g., bots, teams, students, search results).",
        "Past work has shown that under the BTL model, the widely-used maximum-likelihood estimator (MLE) is minimax-optimal in estimating the item parameters, in terms of the mean squared error.",
        "However, another important desideratum for designing estimators is fairness.",
        "In this work, we consider fairness modeled by the notion of bias in statistics.",
        "We show that the MLE incurs a suboptimal rate in terms of bias.",
        "We then propose a simple modification to the MLE, which \"stretches\" the bounding box of the maximum-likelihood optimizer by a small constant factor from the underlying ground truth domain.",
        "We show that this simple modification leads to an improved rate in bias, while maintaining minimax-optimality in the mean squared error.",
        "In this manner, our proposed class of estimators provably improves fairness represented by bias without loss in accuracy."
      ]
    },
    {
      "title": "Feeling the Bern: Adaptive Estimators for Bernoulli Probabilities of Pairwise Comparisons",
      "abstract": [
        "We study methods for aggregating pairwise comparison data among a collection of <inline-formula> <tex-math notation=\"LaTeX\">$n$ </tex-math></inline-formula> items with the goal of estimating the outcome probabilities for future comparisons.",
        "Working within a flexible model that only imposes a form of strong stochastic transitivity, we introduce an \u201cadaptivity index\u201d which compares the risk of our estimator to that of an oracle, over appropriate sub-models, where the oracle knows the specific sub-model in the ground truth.",
        "In addition to measuring the usual worst-case risk of an estimator, this adaptivity index also captures the extent to which the estimator adapts to instance-specific difficulty relative to an oracle estimator.",
        "First, we propose a three-step estimator termed count-randomize-least squares, and show that it has adaptivity index upper bounded by <inline-formula> <tex-math notation=\"LaTeX\">$\\sqrt {n}$ </tex-math></inline-formula> up to logarithmic factors.",
        "We then show that conditional on the planted clique hypothesis, no computationally efficient estimator can achieve an adaptivity index smaller than <inline-formula> <tex-math notation=\"LaTeX\">$\\sqrt {n}$ </tex-math></inline-formula>. Second, we show that a regularized least squares estimator can achieve a poly-logarithmic adaptivity index, thereby demonstrating a <inline-formula> <tex-math notation=\"LaTeX\">$\\sqrt {n}$ </tex-math></inline-formula>-gap between optimal and computationally achievable adaptivity.",
        "Finally, we prove that the standard least squares estimator, which is known to be optimally adaptive in several closely related problems, fails to adapt in the context of estimating pairwise probabilities."
      ]
    },
    {
      "title": "On Strategyproof Conference Peer Review",
      "abstract": [
        "We consider peer review under a conference setting where there are conflicts between the reviewers and the submissions.",
        "Under such conflicts, reviewers can manipulate their reviews in a strategic manner to influence the final rankings of their own papers.",
        "Present-day peer-review systems are not designed to guard against such strategic behavior, beyond minimal (and insufficient) checks such as not assigning a paper to a conflicted reviewer.",
        "In this work, we address this problem through the lens of social choice, and present a theoretical framework for strategyproof and efficient peer review.",
        "Given the conflict graph which satisfies a simple property, we first present and analyze a flexible framework for reviewer-assignment and aggregation for the reviews that guarantees not only strategyproofness but also a natural efficiency property (unanimity).",
        "Our framework is based on the so-called partitioning method, and can be treated as a generalization of this type of method to conference peer review settings.",
        "We then empirically show that the requisite property on the (authorship) conflict graph is indeed satisfied in the ICLR-17 submissions data, and further demonstrate a simple trick to make the partitioning method more practically appealing under conference peer-review settings.",
        "Finally, we complement our positive results with negative theoretical results where we prove that under slightly stronger requirements, it is impossible for any algorithm to be both strategyproof and efficient."
      ]
    },
    {
      "title": "Loss Functions, Axioms, and Peer Review",
      "abstract": [
        "It is common to see a handful of reviewers reject a highly novel paper, because they view, say, extensive experiments as far more important than novelty, whereas the community as a whole would have embraced the paper.",
        "More generally, the disparate mapping of criteria scores to final recommendations by different reviewers is a major source of inconsistency in peer review.",
        "In this paper we present a framework inspired by empirical risk minimization (ERM) for learning the community's aggregate mapping.",
        "The key challenge that arises is the specification of a loss function for ERM.",
        "We consider the class of L(p,q) loss functions, which is a matrix-extension of the standard class of Lp losses on vectors; here the choice of the loss function amounts to choosing the hyperparameters p and q. To deal with the absence of ground truth in our problem, we instead draw on computational social choice to identify desirable values of the hyperparameters p and q. Specifically, we characterize p=q=1 as the only choice of these hyperparameters that satisfies three natural axiomatic properties.",
        "Finally, we implement and apply our approach to reviews from IJCAI 2017."
      ]
    },
    {
      "title": "An Incentive Mechanism for Crowd Sensing with Colluding Agents",
      "abstract": [
        "Vehicular mobile crowd sensing is a fast-emerging paradigm to collect data about the environment by mounting sensors on vehicles such as taxis.",
        "An important problem in vehicular crowd sensing is to design payment mechanisms to incentivize drivers (agents) to collect data, with the overall goal of obtaining the maximum amount of data (across multiple vehicles) for a given budget.",
        "Past works on this problem consider a setting where each agent operates in isolation---an assumption which is frequently violated in practice.",
        "In this paper, we design an incentive mechanism to incentivize agents who can engage in arbitrary collusions.",
        "We then show that in a \"homogeneous\" setting, our mechanism is optimal, and can do as well as any mechanism which knows the agents' preferences a priori.",
        "Moreover, if the agents are non-colluding, then our mechanism automatically does as well as any other non-colluding mechanism.",
        "We also show that our proposed mechanism has strong (and asymptotically optimal) guarantees for a more general \"heterogeneous\" setting.",
        "Experiments based on synthesized data and real-world data reveal gains of over 30\\% attained by our mechanism compared to past literature."
      ]
    },
    {
      "title": "PeerReview 4 All algorithm . We establish deterministic approximation guarantees on the fairness of our PeerReview 4 All algorithm in Section 5",
      "abstract": [
        "We consider the problem of automated assignment of papers to reviewers in conference peer review, with a focus on fairness and statistical accuracy.",
        "Our fairness objective is to maximize the review quality of the most disadvantaged paper, in contrast to the commonly used objective of maximizing the total quality over all papers.",
        "We design an assignment algorithm based on an incremental max-flow procedure that we prove is near-optimally fair.",
        "Our statistical accuracy objective is to ensure correct recovery of the papers that should be accepted.",
        "We provide a sharp minimax analysis of the accuracy of the peer-review process for a popular objective-score model as well as for a novel subjective-score model that we propose in the paper.",
        "Our analysis proves that our proposed assignment algorithm also leads to a near-optimal statistical accuracy.",
        "Finally, we design a novel experiment that allows for an objective comparison of various assignment algorithms, and overcomes the inherent difficulty posed by the absence of a ground truth in experiments on peer-review.",
        "The results of this experiment corroborate the theoretical guarantees of our algorithm."
      ]
    },
    {
      "title": "Choosing How to Choose Papers",
      "abstract": [
        "It is common to see a handful of reviewers reject a highly novel paper, because they view, say, extensive experiments as far more important than novelty, whereas the community as a whole would have embraced the paper.",
        "More generally, the disparate mapping of criteria scores to final recommendations by different reviewers is a major source of inconsistency in peer review.",
        "In this paper we present a framework --- based on $L(p,q)$-norm empirical risk minimization --- for learning the community's aggregate mapping.",
        "We draw on computational social choice to identify desirable values of $p$ and $q$; specifically, we characterize $p=q=1$ as the only choice that satisfies three natural axiomatic properties.",
        "Finally, we implement and apply our approach to reviews from IJCAI 2017."
      ]
    },
    {
      "title": "Your 2 is My 1, Your 3 is My 9: Handling Arbitrary Miscalibrations in Ratings",
      "abstract": [
        "Cardinal scores (numeric ratings) collected from people are well known to suffer from miscalibrations.",
        "A popular approach to address this issue is to assume simplistic models of miscalibration (such as linear biases) to de-bias the scores.",
        "This approach, however, often fares poorly because people's miscalibrations are typically far more complex and not well understood.",
        "In the absence of simplifying assumptions on the miscalibration, it is widely believed by the crowdsourcing community that the only useful information in the cardinal scores is the induced ranking.",
        "In this paper, inspired by the framework of Stein's shrinkage, empirical Bayes, and the classic two-envelope problem, we contest this widespread belief.",
        "Specifically, we consider cardinal scores with arbitrary (or even adversarially chosen) miscalibrations which are only required to be consistent with the induced ranking.",
        "We design estimators which despite making no assumptions on the miscalibration, strictly and uniformly outperform all possible estimators that rely on only the ranking.",
        "Our estimators are flexible in that they can be used as a plug-in for a variety of applications, and we provide a proof-of-concept for A/B testing and ranking.",
        "Our results thus provide novel insights in the eternal debate between cardinal and ordinal data."
      ]
    },
    {
      "title": "PeerReview4All: Fair and Accurate Reviewer Assignment in Peer Review",
      "abstract": [
        "We consider the problem of automated assignment of papers to reviewers in conference peer review, with a focus on fairness and statistical accuracy.",
        "Our fairness objective is to maximize the review quality of the most disadvantaged paper, in contrast to the commonly used objective of maximizing the total quality over all papers.",
        "We design an assignment algorithm based on an incremental max-flow procedure that we prove is near-optimally fair.",
        "Our statistical accuracy objective is to ensure correct recovery of the papers that should be accepted.",
        "We provide a sharp minimax analysis of the accuracy of the peer-review process for a popular objective-score model as well as for a novel subjective-score model that we propose in the paper.",
        "Our analysis proves that our proposed assignment algorithm also leads to a near-optimal statistical accuracy.",
        "Finally, we design a novel experiment that allows for an objective comparison of various assignment algorithms, and overcomes the inherent difficulty posed by the absence of a ground truth in experiments on peer-review.",
        "The results of this experiment as well as of other experiments on synthetic and real data corroborate the theoretical guarantees of our algorithm."
      ]
    },
    {
      "title": "Learning From People",
      "abstract": [
        "Author(s): Shah, Nihar Bhadresh | Advisor(s): Wainwright, Martin J; Ramchandran, Kannan | Abstract: Learning from people represents a new and expanding frontier for data science.",
        "Crowdsourcing, where data is collected from non-experts online, is now extensively employed in academic research, industry, and also for many societal causes.",
        "Two critical challenges in crowdsourcing and learning form people are that of (i) developing algorithms for maximally accurate learning and estimation that operate under minimal modeling assumptions, and (ii) designing incentive mechanisms to elicit high-quality data from people.",
        "In this thesis, we addresses these fundamental challenges in the context of several canonical problem settings that arise in learning from people.",
        "For the challenge of estimation, there are various algorithms proposed in past literature, but their reliance on strong parameter-based assumptions is severely limiting.",
        "In this thesis, we introduce a class of \"permutation-based\" models that are considerably richer than classical parameter-based models.",
        "We present algorithms for estimation which we show are both statistically optimal and significantly more robust than prior state-of-the-art methods.",
        "We also prove that our estimators automatically adapt and are simultaneously optimal over the classical parameter-based models as well, thereby enjoying a surprising win-win in the statistical bias-variance tradeoff.",
        "As for the second challenge of incentivizing people, we design a class of payment mechanisms that take a \"multiplicative\" form.",
        "For several common interfaces in crowdsourcing, we show that these multiplicative mechanisms are surprisingly the only mechanisms that can guarantee honest responses and satisfy a mild and natural requirement which we call no-free-lunch.",
        "We show that our mechanisms have several additional desirable qualities.",
        "The simplicity of our mechanisms imparts them with an additional practical appeal."
      ]
    },
    {
      "title": "Design and Analysis of the NIPS 2016 Review Process",
      "abstract": [
        "Neural Information Processing Systems (NIPS) is a top-tier annual conference in machine learning.",
        "The 2016 edition of the conference comprised more than 2,400 paper submissions, 3,000 reviewers, and 8,000 attendees.",
        "This represents a growth of nearly 40% in terms of submissions, 96% in terms of reviewers, and over 100% in terms of attendees as compared to the previous year.",
        "The massive scale as well as rapid growth of the conference calls for a thorough quality assessment of the peer-review process and novel means of improvement.",
        "In this paper, we analyze several aspects of the data collected during the review process, including an experiment investigating the efficacy of collecting ordinal rankings from reviewers.",
        "Our goal is to check the soundness of the review process, and provide insights that may be useful in the design of the review process of subsequent conferences."
      ]
    },
    {
      "title": "Low Permutation-Rank Matrices: Structural Properties and Noisy Completion",
      "abstract": [
        "We consider the problem of noisy matrix completion, in which the goal is to reconstruct a structured matrix whose entries are partially observed in noise.",
        "Standard approaches to this underdetermined inverse problem are based on assuming that the underlying matrix has low rank, or is well-approximated by a low rank matrix.",
        "In this paper, we first identify how the classical non-negative rank model enforces restrictions that may be undesirable in practice.",
        "We propose a richer model based on what we term the \u201cpermutation-rank\u201d of a matrix and show how the restrictions due to classical low rank assumptions can be avoided by using the richer permutation-rank model.",
        "We establish information-theoretic lower bounds on the rates of estimation, and design an estimator which we prove is simultaneously optimal (up to logarithmic factors) for both the permutation-rank and the low-rank models.",
        "Our results thus show that the proposed permutation-rank model and estimator enjoy a surprising win-win in terms of the statistical bias-variance tradeoff as compared to the classical low-rank models.",
        "An extended version of this paper is available on arXiv [1]."
      ]
    },
    {
      "title": "A Piggybacking Design Framework for Read-and Download-Efficient Distributed Storage Codes",
      "abstract": [
        "Erasure codes are being extensively deployed in distributed storage systems instead of replication to achieve fault tolerance in a storage efficient manner.",
        "While traditional erasure codes are storage efficient, they can result in a significant increase in the amount of data access and downloaded during rebuilding of failed or otherwise unavailable nodes.",
        "In this paper, we present a new framework, which we call piggybacking, for constructing distributed storage codes that are efficient in the amount of data read and downloaded during rebuilding, while meeting requirements arising out of system considerations in data centers\u2014maximum-distance-separability (MDS), high-rate, and a small number of so-called substripes.",
        "Under this setting, to the best of our knowledge, piggyback codes achieve the minimum average amount of data access and downloaded during rebuilding among all existing explicit solutions.",
        "The piggybacking framework also offers a rich design space for constructing codes for a variety of other settings.",
        "In particular, we construct codes that require minimum amount of data access and downloaded for rebuilding among all existing solutions for: 1) binary MDS array codes with more than two parities and 2) MDS codes with the smallest locality during rebuilding.",
        "In addition, we show how piggybacking can be employed to enable efficient repair of parity nodes in codes that address the rebuilding of only systematic nodes.",
        "The basic idea behind the piggybacking framework is to take multiple instances of existing codes and add carefully designed functions of the data from one instance to the others.",
        "This framework provides 25% to 50% savings in the average amount of data access and downloaded during rebuilding depending on the choice of the code parameters."
      ]
    },
    {
      "title": "Active Ranking from Pairwise Comparisons and the Futility of Parametric Assumptions",
      "abstract": [
        "We consider sequential or active ranking of a set of n items based on noisy pairwise comparisons.",
        "Items are ranked according to the probability that a given item beats a randomly chosen item, and ranking refers to partitioning the items into sets of pre-specified sizes according to their scores.",
        "This notion of ranking includes as special cases the identification of the top-k items and the total ordering of the items.",
        "We first analyze a sequential ranking algorithm that counts the number of comparisons won, and uses these counts to decide whether to stop, or to compare another pair of items, chosen based on confidence intervals specified by the data collected up to that point.",
        "We prove that this algorithm succeeds in recovering the ranking using a number of comparisons that is optimal up to logarithmic factors.",
        "This guarantee does not require any structural properties of the underlying pairwise probability matrix, unlike a significant body of past work on pairwise ranking based on parametric models such as the Thurstone or BradleyTerry-Luce models.",
        "It has been a long-standing open question as to whether or not imposing these parametric assumptions allow for improved ranking algorithms.",
        "Our second contribution settles this issue in the context of the problem of active ranking from pairwise comparisons: by means of tight lower bounds, we prove that perhaps surprisingly, these popular parametric modeling choices offer little statistical advantage."
      ]
    },
    {
      "title": "No Oops, You Won't Do It Again: Mechanisms for Self-correction in Crowdsourcing",
      "abstract": [
        "Crowdsourcing is a very popular means of obtaining the large amounts of labeled data that modern machine learning methods require.",
        "Although cheap and fast to obtain, crowdsourced labels suffer from significant amounts of error, thereby degrading the performance of downstream machine learning tasks.",
        "With the goal of improving the quality of the labeled data, we seek to mitigate the many errors that occur due to silly mistakes or inadvertent errors by crowd-sourcing workers.",
        "We propose a two-stage setting for crowdsourcing where the worker first answers the questions, and is then allowed to change her answers after looking at a (noisy) reference answer.",
        "We mathematically formulate this process and develop mechanisms to incentivize workers to act appropriately.",
        "Our mathematical guarantees show that our mechanism incentivizes the workers to answer honestly in both stages, and refrain from answering randomly in the first stage or simply copying in the second.",
        "Numerical experiments reveal a significant boost in performance that such \"self-correction\" can provide when using crowdsourcing to train machine learning algorithms."
      ]
    },
    {
      "title": "Feeling the bern: Adaptive estimators for Bernoulli probabilities of pairwise comparisons",
      "abstract": [
        "We study methods for aggregating pairwise comparison data in order to estimate outcome probabilities for future comparisons.",
        "We investigate this problem under a flexible class of models satisfying the strong stochastic transitivity (SST) condition.",
        "Prior works have studied the minimax risk for estimation of the pairwise comparison probabilities under the SST model.",
        "The minimax risk, however, is a measure of the worst-case risk of an estimator over a large parameter space, and in general provides only a rudimentary understanding of an estimator in problems where the intrinsic difficulty of estimation varies considerably over the parameter space.",
        "In this paper, we introduce an adaptivity index, in order to benchmark the performance of an estimator against an oracle estimator.",
        "The adaptivity index, in addition to measuring the worst-case risk of an estimator, also captures the extent to which the estimator adapts to the instance-specific difficulty of the underlying problem, relative to an oracle estimator.",
        "In the context of this adaptivity index we provide two main results.",
        "We propose a three-step, Count-Randomize-Least squares (CRL) estimator, and derive upper bounds on the adaptivity index of this estimator.",
        "We complement this result with a complexity-theoretic result, that shows that conditional on the planted clique hardness conjecture, no computationally efficient estimator can achieve a substantially smaller adaptivity index."
      ]
    },
    {
      "title": "Optimal Oblivious Updates in Distributed Storage Networks",
      "abstract": [
        "Constructing truly distributed computer systems requires eliminating central control from several system operations.",
        "One such critical module is of updating data.",
        "In distributed storage systems, storage nodes intermittently go offline for various reasons.",
        "Upon coming back online, nodes need to update their contents to reflect any modifications to the data in the interim.",
        "We consider the problem of designing codes and update protocols where a stale node can be updated by downloading data from already updated nodes without requiring the system to store any additional information about the updates (such as previous versions of the data or a log of which data symbols were updated).",
        "We term such an update as an oblivious update.",
        "We provide lower bounds on communication for oblivious updates under generic settings as well as when the codes are restricted to be Maximum-DistanceSeparable (MDS).",
        "We then present explicit codes and oblivious update algorithms for both settings that meet these bounds tightly, thus establishing optimality."
      ]
    },
    {
      "title": "When Do Redundant Requests Reduce Latency?",
      "abstract": [
        "Many systems possess the flexibility to serve requests in more than one way, such as distributed storage systems that store multiple copies of the data.",
        "In such systems, the latency of serving the requests may potentially be reduced by sending redundant requests: a request may be sent to more servers than needed and deemed served when the requisite number of servers complete service.",
        "Such a mechanism trades off the possibility of faster execution of the request with the increase in the load on the system.",
        "Several recent works empirically evaluate the latency performance of redundant requests in diverse settings.",
        "In this paper, we perform an analytical study of the latency performance of redundant requests, with the primary goals of characterizing under what scenarios sending redundant requests will help (and under what scenarios it will not), and of designing optimal redundant-requesting policies.",
        "We show that when service times are i.i.d.",
        "memoryless or \u201cheavier,\u201d and when the additional copies of already-completed jobs can be removed instantly, maximally scheduling redundant requests achieves the optimal average latency.",
        "On the other hand, when service times are i.i.d. \u201c",
        "lighter\u201d or when service times are memoryless and removal of jobs is not instantaneous, then not having any redundancy in the requests is optimal under high loads.",
        "Our results are applicable to arbitrary arrival processes."
      ]
    },
    {
      "title": "Active ranking from pairwise comparisons and when parametric assumptions do not help",
      "abstract": [
        "We consider sequential or active ranking of a set of n items based on noisy pairwise comparisons.",
        "Items are ranked according to the probability that a given item beats a randomly chosen item, and ranking refers to partitioning the items into sets of pre-specified sizes according to their scores.",
        "This notion of ranking includes as special cases the identification of the top-k items and the total ordering of the items.",
        "We first analyze a sequential ranking algorithm that counts the number of comparisons won, and uses these counts to decide whether to stop, or to compare another pair of items, chosen based on confidence intervals specified by the data collected up to that point.",
        "We prove that this algorithm succeeds in recovering the ranking using a number of comparisons that is optimal up to logarithmic factors.",
        "This guarantee does not require any structural properties of the underlying pairwise probability matrix, unlike a significant body of past work on pairwise ranking based on parametric models such as the Thurstone or Bradley-Terry-Luce models.",
        "It has been a long-standing open question as to whether or not imposing these parametric assumptions allows for improved ranking algorithms.",
        "For stochastic comparison models, in which the pairwise probabilities are bounded away from zero, our second contribution is to resolve this issue by proving a lower bound for parametric models.",
        "This shows, perhaps surprisingly, that these popular parametric modeling choices offer at most logarithmic gains for stochastic comparisons."
      ]
    },
    {
      "title": "A Permutation-Based Model for Crowd Labeling: Optimal Estimation and Robustness",
      "abstract": [
        "The task of aggregating and denoising crowd-labeled data has gained increased significance with the advent of crowdsourcing platforms and massive datasets.",
        "We propose a permutation-based model for crowd labeled data that is a significant generalization of the classical Dawid-Skene model, and introduce a new error metric by which to compare different estimators.",
        "We derive global minimax rates for the permutation-based model that are sharp up to logarithmic factors, and match the minimax lower bounds derived under the simpler Dawid-Skene model.",
        "We then design two computationally-efficient estimators: the WAN estimator for the setting where the ordering of workers in terms of their abilities is approximately known, and the OBI- WAN estimator where that is not known.",
        "For each of these estimators, we provide non-asymptotic bounds on their performance.",
        "We conduct synthetic simulations and experiments on real-world crowdsourcing data, and the experimental results corroborate our theoretical findings."
      ]
    },
    {
      "title": "Approval Voting and Incentives in Crowdsourcing Supplementary Material for Approval Voting and Incentives in Crowdsourcing",
      "abstract": [
        "A. Proofs A.1.",
        "Proof of Theorem 4.1 We assume that there indeed exists some incentive-compatible payment function f , and prove a contradiction.",
        "Let us first consider the special case of N = G = 1 and B = 2.",
        "Since N = G = 1, there is only one question.",
        "Let p1 > 0.5 be the probability, according to the belief of the worker, that option 1 is correct; the worker then believes that option 2 is correct with probability (1 p1).",
        "When p1 = 1, we need the worker to select option 1 alone.",
        "Thus we need"
      ]
    },
    {
      "title": "Parametric Prediction from Parametric Agents",
      "abstract": [
        "Prediction algorithms are often designed under the assumption that the training data is provided to the algorithm, and that the algorithm has no control over the quality of the training data.",
        "In many situations, however, the training data is collected by surveying people, for instance, in the prediction of the future demand for a product by surveying a number of potential customers, or the prediction of the winner of an election by surveying potential voters.",
        "Collecting data from people is much cheaper, easier and faster today due to the emergence of several commercial crowdsourcing platforms such as Amazon Mechanical Turk and others.",
        "In such situations, it is possible to monetarily incentivize the respondents to provide higher quality inputs.",
        "In any realistic setup, the responses obtained from people (\u201cthe agents\u201d) are noisy: one cannot expect a naive customer to gauge the sales of a product accurately.",
        "Moreover, every individual has a different expertise and ability, and will likely react differently to the amount of money paid per task.",
        "For example, some people may be active users of the surveyed product, therefore have a better understanding of its anticipated usage.",
        "We assume that the surveyor (\u201cthe principal\u201d) has no knowledge of the behavior of individual agents.",
        "It is therefore important to design an appropriate incentive mechanism for the prediction procedure that exploits the heterogeneity of the agents, motivating them to participate and exert suitable levels of effort.",
        "An appropriate incentive will provide higher quality data and as result, a superior prediction performance.",
        "This requirement motivates the problem at the interface between statistical estimation and mechanism design considered in this paper.",
        "As compared to problems that tackle only one of the prediction and the mechanism design problems, the problem of joint design poses a significantly greater challenge.",
        "From the statistical prediction point of view, the challenge is that every sample is drawn from a different distribution, whose properties are unknown apriori to the principal.",
        "From the mechanism design perspective, the challenge is that the incentivization procedure not only needs to ensure that agents report truthfully, but also needs to ensure that each agent exerts an effort that minimizes the overall prediction error.",
        "In this paper, we formulate and optimally solve a \u201cparametric\u201d form of this joint design problem.",
        "More specifically, the principal desires to predict a parameter of a known distribution.",
        "Each agent is modeled in a parametric fashion, with her work quality (or expertise) governed by a single parame-"
      ]
    },
    {
      "title": "Regularized Minimax Conditional Entropy for Crowdsourcing",
      "abstract": [
        "There is a rapidly increasing interest in crowdsourcing for data labeling.",
        "By crowdsourcing, a large number of labels can be often quickly gathered at low cost.",
        "However, the labels provided by the crowdsourcing workers are usually not of high quality.",
        "In this paper, we propose a minimax conditional entropy principle to infer ground truth from noisy crowdsourced labels.",
        "Under this principle, we derive a unique probabilistic labeling model jointly parameterized by worker ability and item difficulty.",
        "We also propose an objective measurement principle, and show that our method is the only method which satisfies this objective measurement principle.",
        "We validate our method through a variety of real crowdsourcing datasets with binary, multiclass or ordinal labels."
      ]
    },
    {
      "title": "A Truth Serum for Large-Scale Evaluations",
      "abstract": [
        "A major challenge in obtaining large-scale evaluations, e.g., product or service reviews on online platforms, labeling images, grading in online courses, etc.,",
        "is that of eliciting honest responses from agents in the absence of verifiability.",
        "We propose a new reward mechanism with strong incentive properties applicable in a wide variety of such settings.",
        "This mechanism has a simple and intuitive output agreement structure: an agent gets a reward only if her response for an evaluation matches that of her peer.",
        "But instead of the reward being the same across different answers, it is inversely proportional to a popularity index of each answer.",
        "This index is a second order population statistic that captures how frequently two agents performing the same evaluation agree on the particular answer.",
        "Rare agreements thus earn a higher reward than agreements that are relatively more common.",
        "\nIn the regime where there are a large number of evaluation tasks, we show that truthful behavior is a strict Bayes-Nash equilibrium of the game induced by the mechanism.",
        "Further, we show that the truthful equilibrium is approximately optimal in terms of expected payoffs to the agents across all symmetric equilibria, where the approximation error vanishes in the number of evaluation tasks.",
        "Moreover, under a mild condition on strategy space, we show that any symmetric equilibrium that gives a higher expected payoff than the truthful equilibrium must be close to being fully informative if the number of evaluations is large.",
        "These last two results are driven by a new notion of an agreement measure that is shown to be monotonic in information loss.",
        "This notion and its properties are of independent interest."
      ]
    },
    {
      "title": "Having Your Cake and Eating It Too: Jointly Optimal Erasure Codes for I/O, Storage, and Network-bandwidth",
      "abstract": [
        "Erasure codes, such as Reed-Solomon (RS) codes, are increasingly being deployed as an alternative to data-replication for fault tolerance in distributed storage systems.",
        "While RS codes provide significant savings in storage space, they can impose a huge burden on the I/O and network resources when reconstructing failed or otherwise unavailable data.",
        "A recent class of erasure codes, called minimum-storage-regeneration (MSR) codes, has emerged as a superior alternative to the popular RS codes, in that it minimizes network transfers during reconstruction while also being optimal with respect to storage and reliability.",
        "However, existing practical MSR codes do not address the increasingly important problem of I/O overhead incurred during reconstructions, and are, in general, inferior to RS codes in this regard.",
        "In this paper, we design erasure codes that are simultaneously optimal in terms of I/O, storage, and network bandwidth.",
        "Our design builds on top of a class of powerful practical codes, called the product-matrix-MSR codes.",
        "Evaluations show that our proposed design results in a significant reduction the number of I/Os consumed during reconstructions (a 5\u00d7 reduction for typical parameters), while retaining optimality with respect to storage, reliability, and network bandwidth."
      ]
    },
    {
      "title": "Approval Voting and Incentives in Crowdsourcing",
      "abstract": [
        "The growing need for labeled training data has made crowdsourcing an important part of machine learning.",
        "The quality of crowdsourced labels is, however, adversely affected by three factors: (1) the workers are not experts; (2) the incentives of the workers are not aligned with those of the requesters; and (3) the interface does not allow workers to convey their knowledge accurately, by forcing them to make a single choice among a set of options.",
        "In this paper, we address these issues by introducing approval voting to %judiciously utilize the expertise of workers who have partial knowledge of the true answer, and coupling it with a (\"strictly proper\") incentive-compatible compensation mechanism.",
        "We show rigorous theoretical guarantees of optimality of our mechanism together with a simple axiomatic characterization.",
        "We also conduct preliminary empirical studies on Amazon Mechanical Turk which validate our approach."
      ]
    },
    {
      "title": "Information-Theoretically Secure Erasure Codes for Distributed Storage",
      "abstract": [
        "Repair operations in erasure-coded distributed storage systems involve a lot of data movement.",
        "This can potentially expose data to malicious acts of passive eavesdroppers or active adversaries, putting security of the system at risk.",
        "This paper presents coding schemes and repair algorithms that ensure security of the data in the presence of passive eavesdroppers and active adversaries while maintaining high availability, reliability, and resource efficiency in the system.",
        "The proposed codes are optimal in that they meet previously proposed lower bounds on storage and network-bandwidth requirements for a wide range of system parameters.",
        "The results thus establish the secure storage capacity of such systems.",
        "The proposed codes are based on an optimal class of codes called product-matrix codes.",
        "The constructions presented for security from active adversaries provide an additional appealing feature of \u201con-demand security,\u201d where the desired level of security can be chosen separately for each instance of repair, and the proposed algorithms remain optimal simultaneously for all possible security levels.",
        "This paper also provides necessary and sufficient conditions governing the transformation of any (non-secure) code into one providing on-demand security."
      ]
    },
    {
      "title": "A Simple Truth Serums for Massively Crowdsourced Evaluation Tasks",
      "abstract": [
        "We consider the problem of eliciting truthful responses from agents in the absence of any known answers.",
        "This class of problems, pioneered by the peer-prediction method and the Bayesian truth serum, is now quite well studied in the literature.",
        "In this paper we propose new mechanisms that, unlike most works on this topic, require no extraneous elicitation from the agents, and furthermore allow the agents\u2019 beliefs to be (almost) arbitrary.",
        "Moreover, these mechanisms have the structure of output agreement mechanisms, which are simple, intuitive and have been quite popular in practice.",
        "These mechanisms operate under scenarios where the number of questions is large, and are suitable for most tasks in crowdsourcing and peer-grading."
      ]
    },
    {
      "title": "The Square Root Agreement Rule for Incentivizing Truthful Feedback on Online Platforms",
      "abstract": [
        "A major challenge in obtaining evaluations of products or services on e-commerce platforms is eliciting informative responses in the absence of verifiability.",
        "This paper proposes the square root agreement rule (SRA): a simple reward mechanism that incentivizes truthful responses to objective evaluations on such platforms.",
        "In this mechanism, an agent gets a reward for an evaluation only if the answer matches that of a peer agent, where this reward is inversely proportional to a popularity index of the answer.",
        "This index is defined to be the square root of the empirical frequency at which any two agents performing the same evaluation agree on the particular answer across evaluations of similar entities operating on the platform.",
        "Rarely agreed-on answers thus earn a higher reward than answers for which agreements are relatively more common.",
        "We show that in the many tasks regime, the truthful equilibrium under SRA is strictly payoff-dominant across large classes of natural equilibria that could arise in these settings, thus increasing the likelihood of its adoption.",
        "Although there exist other mechanisms achieving such guarantees, they either impose additional assumptions on the response distribution that are not generally satisfied for objective evaluations or they incentivize truthful behavior only if each agent performs a prohibitively large number of evaluations and commits to using the same strategy for each evaluation.",
        "SRA is the first known incentive mechanism satisfying such guarantees without imposing any such requirements.",
        "Moreover, our empirical findings demonstrate the robustness of the incentive properties of SRA in the presence of mild subjectivity or observational biases in the responses.",
        "These properties make SRA uniquely attractive for administering reward-based incentive schemes (e.g., rebates, discounts, reputation scores, etc.)",
        "on online platforms.",
        "This paper was accepted by Kartik Hosanagar, information systems."
      ]
    },
    {
      "title": "Estimation from Pairwise Comparisons: Sharp Minimax Bounds with Topology Dependence",
      "abstract": [
        "Data in the form of pairwise comparisons arises in many domains, including preference elicitation, sporting competitions, and peer grading among others.",
        "We consider parametric ordinal models for such pairwise comparison data involving a latent vector w* e Rd that represents the \"qualities\" of the d items being compared; this class of models includes the two most widely used parametric models|the Bradley-Terry-Luce (BTL) and the Thurstone models.",
        "Working within a standard minimax framework, we provide tight upper and lower bounds on the optimal error in estimating the quality score vector w* under this class of models.",
        "The bounds depend on the topology of the comparison graph induced by the subset of pairs being compared, via the spectrum of the Laplacian of the comparison graph.",
        "Thus, in settings where the subset of pairs may be chosen, our results provide principled guidelines for making this choice.",
        "Finally, we compare these error rates to those under cardinal measurement models and show that the error rates in the ordinal and cardinal settings have identical scalings apart from constant pre-factors."
      ]
    },
    {
      "title": "Truth Serums for Massively Crowdsourced Evaluation Tasks",
      "abstract": [
        "Incentivizing effort and eliciting truthful responses from agents in the absence of verifiability is a major challenge faced while crowdsourcing many types of evaluation tasks like labeling images, grading assignments in online courses, etc.",
        "In this paper, we propose new reward mechanisms for such settings that, unlike most previously studied mechanisms, impose minimal assumptions on the structure and knowledge of the underlying generating model, can account for heterogeneity in the agents' abilities, require no extraneous elicitation from them, and furthermore allow their beliefs to be (almost) arbitrary.",
        "Moreover, these mechanisms have the simple and intuitive structure of output agreement mechanisms, which, despite not incentivizing truthful behavior, have nevertheless been quite popular in practice.",
        "We achieve this by leveraging a typical characteristic of many of these settings, which is the existence of a large number of similar tasks."
      ]
    },
    {
      "title": "L G ] 2 5 M ar 2 01 5 Regularized Minimax Conditional Entropy for Crowdsourcing",
      "abstract": [
        "There is a rapidly increasing interest in crowdsourcing for data labeling.",
        "By crowdsourcing, a large number of labels can be often quickly gathered at low cost.",
        "However, the labels provided by the crowdsourcing workers are usually not of high quality.",
        "In this paper, we propose a minimax conditional entropy principle to infer ground truth from noisy crowdsourced labels.",
        "Under this principle, we derive a unique probabilistic labeling model jointly parameterized by worker ability and item difficulty.",
        "We also propose an objective measurement principle, and show that our method is the only method which satisfies this objective measurement principle.",
        "We validate our method through a variety of real crowdsourcing datasets with binary, multiclass or ordinal labels."
      ]
    },
    {
      "title": "Stochastically Transitive Models for Pairwise Comparisons: Statistical and Computational Issues",
      "abstract": [
        "There are various parametric models for analyzing pairwise comparison data, including the Bradley\u2013Terry\u2013Luce (BTL) and Thurstone models, but their reliance on strong parametric assumptions is limiting.",
        "In this paper, we study a flexible model for pairwise comparisons, under which the probabilities of outcomes are required only to satisfy a natural form of stochastic transitivity.",
        "This class includes parametric models, including the BTL and Thurstone models as special cases, but is considerably more general.",
        "We provide various examples of models in this broader stochastically transitive class for which classical parametric models provide poor fits.",
        "Despite this greater flexibility, we show that the matrix of probabilities can be estimated at the same rate as in standard parametric models up to logarithmic terms.",
        "On the other hand, unlike in the BTL and Thurstone models, computing the minimax-optimal estimator in the stochastically transitive model is non-trivial, and we explore various computationally tractable alternatives.",
        "We show that a simple singular value thresholding algorithm is statistically consistent but does not achieve the minimax rate.",
        "We then propose and study algorithms that achieve the minimax rate over interesting sub-classes of the full stochastically transitive class.",
        "We complement our theoretical results with thorough numerical simulations."
      ]
    },
    {
      "title": "Simple, Robust and Optimal Ranking from Pairwise Comparisons",
      "abstract": [
        "We consider data in the form of pairwise comparisons of n items, with the goal of precisely identifying the top k items for some value of k < n, or alternatively, recovering a ranking of all the items.",
        "We analyze the Copeland counting algorithm that ranks the items in order of the number of pairwise comparisons won, and show it has three attractive features: (a) its computational efficiency leads to speed-ups of several orders of magnitude in computation time as compared to prior work; (b) it is robust in that theoretical guarantees impose no conditions on the underlying matrix of pairwise-comparison probabilities, in contrast to some prior work that applies only to the BTL parametric model; and (c) it is an optimal method up to constant factors, meaning that it achieves the information-theoretic limits for recovering the top k-subset.",
        "We extend our results to obtain sharp guarantees for approximate recovery under the Hamming distortion metric, and more generally, to any arbitrary error requirement that satisfies a simple and natural monotonicity condition."
      ]
    },
    {
      "title": "Fundamental limits on communication for oblivious updates in storage networks",
      "abstract": [
        "In distributed storage systems, storage nodes intermittently go offline for numerous reasons.",
        "On coming back online, nodes need to update their contents to reflect any modifications to the data in the interim.",
        "In this paper, we consider a setting where no information regarding modified data needs to be logged in the system.",
        "In such a setting, a `stale' node needs to update its contents by downloading data from already updated nodes, while neither the stale node nor the updated nodes have any knowledge as to which data symbols are modified and what their value is.",
        "We investigate the fundamental limits on the amount of communication necessary for such an oblivious update process.",
        "We first present a generic lower bound on the amount of communication that is necessary under any storage code with a linear encoding (while allowing non-linear update protocols).",
        "This lower bound is derived under a set of extremely weak conditions, giving all updated nodes access to the entire modified data and the stale node access to the entire stale data as side information.",
        "We then present codes and update algorithms that are optimal in that they meet this lower bound.",
        "Next, we present a lower bound for an important subclass of codes, that of linear Maximum-Distance-Separable (MDS) codes.",
        "We then present an MDS code construction and an associated update algorithm that meets this lower bound.",
        "These results thus establish the capacity of oblivious updates in terms of the communication requirements under these settings."
      ]
    },
    {
      "title": "On the Impossibility of Convex Inference in Human Computation",
      "abstract": [
        "\n \n Human computation or crowdsourcing involves joint inference of the ground-truth-answers and the worker-abilities by optimizing an objective function, for instance, by maximizing the data likelihood based on an assumed underlying model.",
        "A variety of methods have been proposed in the literature to address this inference problem.",
        "As far as we know, none of the objective functions in existing methods is convex.",
        "In machine learning and applied statistics, a convex function such as the objective function of support vector machines (SVMs) is generally preferred, since it can leverage the high-performance algorithms and rigorous guarantees established in the extensive literature on convex optimization.",
        "One may thus wonder if there exists a meaningful convex objective function for the inference problem in human computation.",
        "In this paper, we investigate this convexity issue for human computation.",
        "We take an axiomatic approach by formulating a set of axioms that impose two mild and natural assumptions on the objective function for the inference.",
        "Under these axioms, we show that it is unfortunately impossible to ensure convexity of the inference problem.",
        "On the other hand, we show that interestingly, in the absence of a requirement to model \"spammers\", one can construct reasonable objective functions for crowdsourcing that guarantee convex inference.",
        "\n \n"
      ]
    },
    {
      "title": "When is it Better to Compare than to Score?",
      "abstract": [
        "When eliciting judgements from humans for an unknown quantity, one often has the choice of making direct-scoring (cardinal) or comparative (ordinal) measurements.",
        "In this paper we study the relative merits of either choice, providing empirical and theoretical guidelines for the selection of a measurement scheme.",
        "We provide empirical evidence based on experiments on Amazon Mechanical Turk that in a variety of tasks, (pairwise-comparative) ordinal measurements have lower per sample noise and are typically faster to elicit than cardinal ones.",
        "Ordinal measurements however typically provide less information.",
        "We then consider the popular Thurstone and Bradley-Terry-Luce (BTL) models for ordinal measurements and characterize the minimax error rates for estimating the unknown quantity.",
        "We compare these minimax error rates to those under cardinal measurement models and quantify for what noise levels ordinal measurements are better.",
        "Finally, we revisit the data collected from our experiments and show that fitting these models confirms this prediction: for tasks where the noise in ordinal measurements is sufficiently low, the ordinal approach results in smaller errors in the estimation."
      ]
    },
    {
      "title": "A \u201cHitchhiker\u2019s\u201d Guide to Fast and Efficient Data Reconstruction in Erasure-coded Data Centers",
      "abstract": [
        "Erasure codes such as Reed-Solomon (RS) codes are being extensively deployed in data centers since they offer significantly higher reliability than data replication methods at much lower storage overheads.",
        "These codes however mandate much higher resources with respect to network bandwidth and disk IO during reconstruction of data that is missing or otherwise unavailable.",
        "Existing solutions to this problem either demand additional storage space or severely limit the choice of the system parameters.",
        "In this paper, we present Hitchhiker, a new erasure-coded storage system that reduces both network traffic and disk IO by around 25% to 45% during reconstruction of missing or otherwise unavailable data, with no additional storage, the same fault tolerance, and arbitrary flexibility in the choice of parameters, as compared to RS-based systems.",
        "Hitchhiker \u201crides\u201d on top of RS codes, and is based on novel encoding and decoding techniques that will be presented in this paper.",
        "We have implemented Hitchhiker in the Hadoop Distributed File System (HDFS).",
        "When evaluating various metrics on the data-warehouse cluster in production at Facebook with real-time traffic and workloads, during reconstruction, we observe a 36% reduction in the computation time and a 32% reduction in the data read time, in addition to the 35% reduction in network traffic and disk IO.",
        "Hitchhiker can thus reduce the latency of degraded reads and perform faster recovery from failed or decommissioned machines."
      ]
    },
    {
      "title": "Some Scaling Laws for MOOC Assessments",
      "abstract": [
        "One problem that arises with the increasing numbers of students in Massive Open Online Courses (MOOCs) is that of student evaluation.",
        "The large number of students makes it infeasible for the instructors or the teaching assistants to grade all assignments, while the present auto-grading technology is not feasible for many topics of interest.",
        "As a result, there has recently been a great push for employing peergrading, where students grade each other, since the number of graders automatically scale with the number of students.",
        "However, in practice, peer-grading has been observed to have high error rates and has come under serious criticism.",
        "In this paper, we take a statistical approach towards understanding the feasibility of peer-grading for MOOCs.",
        "Under simple (yet general) models, we show that peer-grading as a standalone will not scale, i.e., as the number of students increases, the expected number of students misgraded will grow proportionately.",
        "We then consider a hybrid approach that combines peer-grading with auto-grading.",
        "In this setting, an automated approach is used for \u2018dimensionality reduction\u2019, a classical technique in statistics and machine learning, and peer-grading is employed to evaluate this lower dimensional set of answers.",
        "We show that this alternative approach has the potential to scale.",
        "While most current research on assessment in MOOCs is empirical, our more theoretical approach provides a fundamental understanding of the errors observed in current grading systems, and provides a direction for future research to overcome those errors."
      ]
    },
    {
      "title": "Optimality of the product-matrix construction for secure MSR regenerating codes",
      "abstract": [
        "In this paper, we consider the security of exact-repair regenerating codes operating at the minimum-storage-regenerating (MSR) point.",
        "The security requirement (introduced in Shah et.",
        "al.) is that no information about the stored data file must be leaked in the presence of an eavesdropper who has access to the contents of \u21131 nodes as well as all the repair traffic entering a second disjoint set of \u21132 nodes.",
        "We derive an upper bound on the size of a data file that can be securely stored that holds whenever \u21132 \u2264 d - k + 1.",
        "This upper bound proves the optimality of the product-matrix-based construction of secure MSR regenerating codes by Shah et.",
        "al."
      ]
    },
    {
      "title": "Double or Nothing: Multiplicative Incentive Mechanisms for Crowdsourcing",
      "abstract": [
        "Crowdsourcing has gained immense popularity in machine learning applications for obtaining large amounts of labeled data.",
        "Crowdsourcing is cheap and fast, but suffers from the problem of low-quality data.",
        "To address this fundamental challenge in crowdsourcing, we propose a simple payment mechanism to incentivize workers to answer only the questions that they are sure of and skip the rest.",
        "We show that surprisingly, under a mild and natural \"no-free-lunch\" requirement, this mechanism is the one and only incentive-compatible payment mechanism possible.",
        "We also show that among all possible incentive-compatible mechanisms (that may or may not satisfy no-free-lunch), our mechanism makes the smallest possible payment to spammers.",
        "Interestingly, this unique mechanism takes a \"multiplicative\" form.",
        "The simplicity of the mechanism is an added benefit.",
        "In preliminary experiments involving over several hundred workers, we observe a significant reduction in the error rates under our unique mechanism for the same or lower monetary expenditure."
      ]
    },
    {
      "title": "A \"hitchhiker's\" guide to fast and efficient data reconstruction in erasure-coded data centers",
      "abstract": [
        "Erasure codes such as Reed-Solomon (RS) codes are being extensively deployed in data centers since they offer significantly higher reliability than data replication methods at much lower storage overheads.",
        "These codes however mandate much higher resources with respect to network bandwidth and disk IO during reconstruction of data that is missing or otherwise unavailable.",
        "Existing solutions to this problem either demand additional storage space or severely limit the choice of the system parameters.",
        "In this paper, we present \"Hitchhiker\", a new erasure-coded storage system that reduces both network traffic and disk IO by around 25% to 45% during reconstruction of missing or otherwise unavailable data, with no additional storage, the same fault tolerance, and arbitrary flexibility in the choice of parameters, as compared to RS-based systems.",
        "Hitchhiker 'rides' on top of RS codes, and is based on novel encoding and decoding techniques that will be presented in this paper.",
        "We have implemented Hitchhiker in the Hadoop Distributed File System (HDFS).",
        "When evaluating various metrics on the data-warehouse cluster in production at Facebook with real-time traffic and workloads, during reconstruction, we observe a 36% reduction in the computation time and a 32% reduction in the data read time, in addition to the 35% reduction in network traffic and disk IO.",
        "Hitchhiker can thus reduce the latency of degraded reads and perform faster recovery from failed or decommissioned machines."
      ]
    },
    {
      "title": "One extra bit of download ensures perfectly private information retrieval",
      "abstract": [
        "Private information retrieval (PIR) systems allow a user to retrieve a record from a public database without revealing to the server which record is being retrieved.",
        "The literature on PIR considers only replication-based systems, wherein each storage node stores a copy of the entire data.",
        "However, systems based on erasure codes are gaining increasing popularity due to a variety of reasons.",
        "This paper initiates an investigation into PIR in erasure-coded systems by establishing its capacity and designing explicit codes and algorithms.",
        "The notion of privacy considered here is information-theoretic, and the metric optimized is the amount of data downloaded by the user during PIR.",
        "In this paper, we present four main results.",
        "First, we design an explicit erasure code and PIR algorithm that requires only one extra bit of download to provide perfect privacy.",
        "In contrast, all existing PIR algorithms require a download of at least twice the size of the requisite data.",
        "Second, we derive lower bounds proving the necessity of downloading at least one additional bit.",
        "This establishes the precise capacity of PIR with respect to the metric of download.",
        "These results are also applicable to PIR in replication-based systems, which are a special case of erasure codes.",
        "Our third contribution is a negative result showing that capacity-achieving codes necessitate super-linear storage overheads.",
        "This motivates the fourth contribution of this paper: an erasure code and PIR algorithm that requires a linear storage overhead, provides high reliability to the data, and is a small factor away from the capacity."
      ]
    },
    {
      "title": "Solu % on ' \u201c X \u201d ' Your ' solu % on ' \u201c Y \u201d",
      "abstract": [
        "MOOCs have been highly successful due to the ease of disseminating information: anyone with an Internet connection can watch videos of the lectures and download study material.",
        "However, they still lag far behind conventional classrooms in one critical aspect\u2014feedback to and evaluation of the students\u2014due to severe mismatches in the number of students enrolled and the number of experts available.",
        "One means of performing evaluation and feedback is peer evaluation wherein the answers submitted by a student are anonymized and provided to a set of other students to evaluate.",
        "In current peer evaluation techniques, these peer evaluators assign cardinal scores to the given solutions.",
        "In this paper, we explore an alternative approach to peer evaluation based on pairwise comparisons.",
        "We present evidence that such an ordinal approach can be significantly more robust to the lack of expertise of the evaluators, as compared to the conventional cardinal approaches.",
        "This work is a first step in understanding the trade-off between the precision of cardinal scores and the robustness of ordinal evaluations for peer grading."
      ]
    },
    {
      "title": "A Solution to the Network Challenges of Data Recovery in Erasure-coded Distributed Storage Systems: A Study on the Facebook Warehouse Cluster",
      "abstract": [
        "Erasure codes, such as Reed-Solomon (RS) codes, are being increasingly employed in data centers to combat the cost of reliably storing large amounts of data.",
        "Although these codes provide optimal storage efficiency, they require significantly high network and disk usage during recovery of missing data.",
        "In this paper, we first present a study on the impact of recovery operations of erasure-coded data on the data-center network, based on measurements from Facebook's warehouse cluster in production.",
        "To the best of our knowledge, this is the first study of its kind available in the literature.",
        "Our study reveals that recovery of RS-coded data results in a significant increase in network traffic, more than a hundred terabytes per day, in a cluster storing multiple petabytes of RS-coded data.",
        "\nTo address this issue, we present a new storage code using our recently proposed \"Piggybacking\" framework, that reduces the network and disk usage during recovery by 30% in theory, while also being storage optimal and supporting arbitrary design parameters.",
        "The implementation of the proposed code in the Hadoop Distributed File System (HDFS) is underway.",
        "We use the measurements from the warehouse cluster to show that the proposed code would lead to a reduction of close to fifty terabytes of cross-rack traffic per day."
      ]
    },
    {
      "title": "When do redundant requests reduce latency ?",
      "abstract": [
        "Several systems possess the flexibility to serve requests in more than one way.",
        "For instance, a distributed storage system storing multiple replicas of the data can serve a request from any of the multiple servers that store the requested data, or a computational task may be performed in a compute-cluster by any one of multiple processors.",
        "In such systems, the latency of serving the requests may potentially be reduced by sending redundant requests: a request may be sent to an excess number of servers, and it is deemed served when the requisite number of servers complete service.",
        "Such a mechanism trades off the possibility of faster execution of at least one copy of the request with the increase in the delay due to an increased load on the system.",
        "Due to this tradeoff, it is unclear when redundant requests may actually help.",
        "Several recent works empirically evaluate the latency performance of redundant requests in diverse settings.",
        "This work aims at a rigorous analytical study of the latency performance of redundant requests, with the primary goals of characterizing the situations when sending redundant requests will help (and when not), and designing optimal redundant-requesting policies.",
        "We first present a model that captures the key features of such systems.",
        "We show that when service times are i.i.d.",
        "memoryless or \u201cheavy\u201d, and when the additional copies of already-completed jobs can be removed with negligible costs, redundant requests reduce the average latency.",
        "On the other hand, when service times are \u201clight\u201d or when service times are memoryless and removal of jobs results in a non-negligible penalty, not having any redundancy in the request is optimal under high loads.",
        "Our results hold for arbitrary arrival processes."
      ]
    },
    {
      "title": "Secure network coding for distributed secret sharing with low communication cost",
      "abstract": [
        "Shamir's (n, k) threshold secret sharing is an important component of several cryptographic protocols, such as those for secure multiparty-computation.",
        "These protocols typically assume the presence of direct communication links from the dealer to all participants, in which case the dealer can directly pass the shares of the secret to every participant.",
        "In this paper, we consider the problem of secret sharing when the dealer does not have direct communication links to all participants, and instead, they form a general network.",
        "We present an algorithm for secret sharing over networks that satisfy what we call the k-propagating-dealer condition.",
        "The algorithm is communication-efficient, distributed and deterministic.",
        "Interestingly, the solution constitutes an instance of a network coding problem admitting a distributed and deterministic solution, and furthermore, handles the case of nodal-eavesdropping, about which very little appears to be known in the literature.",
        "In the second part of the paper, we derive information-theoretic lower bounds on the communication complexity of secret sharing over any network, which may also be of independent interest.",
        "We show that for networks satisfying the k-propagating-dealer condition, the communication complexity of our algorithm is \u0398(n), and furthermore, is always within a constant factor of the lower bound.",
        "We also show that, in contrast, existing solutions in the literature entail a communication-complexity that is superlinear for a wide class of networks, and is \u0398(n2) in the worst case.",
        "Our algorithm thus allows for efficient generalization of several cryptographic protocols to a large class of networks."
      ]
    },
    {
      "title": "A Case for Ordinal Peer-evaluation in MOOCs",
      "abstract": [
        "MOOCs have been highly successful due to the ease of disseminating information: anyone with an Internet connection can watch videos of the lectures and download study material.",
        "However, they still lag far behind conventional classrooms in one critical aspect\u2014feedback to and evaluation of the students\u2014due to severe mismatches in the number of students enrolled and the number of experts available.",
        "One means of performing evaluation and feedback is peer evaluation wherein the answers submitted by a student are anonymized and provided to a set of other students to evaluate.",
        "As per current peer evaluation techniques, these peer evaluators assign cardinal scores to the given solutions.",
        "In this paper, we posit an alternative approach to peer evaluation based on pairwise comparisons.",
        "We argue that such an ordinal approach is signi\ufb01cantly more robust to the lack of expertise of the evaluators, as compared to the conventional cardinal approaches."
      ]
    },
    {
      "title": "A piggybacking design framework for read-and download-efficient distributed storage codes",
      "abstract": [
        "We present a new piggybacking framework for designing distributed storage codes that are efficient in the amount of data read and downloaded during node-repair.",
        "We illustrate the power of this framework by constructing explicit codes that attain the smallest amount of data to be read and downloaded for repair among all existing solutions for three important settings: (a) codes meeting the constraints of being maximum distance separable (MDS), high-rate, and having a small number of substripes, (b) binary MDS codes for all parameters where binary MDS codes exist, and (c) MDS codes with the smallest repair-locality.",
        "In addition, we show how to use this framework to enable efficient repair of parity nodes in existing codes that are constructed to address the repair of only the systematic nodes.",
        "The basic idea behind this framework is to take multiple stripes of existing codes and add carefully designed functions of the data of one stripe to other stripes.",
        "Typical savings in the amount of data read and downloaded during repair are 25% to 50% depending on the choice of the system parameters."
      ]
    },
    {
      "title": "Towards Computation , Space , and Data Efficiency in de novo DNA Assembly : A Novel Algorithmic Framework",
      "abstract": [
        "We consider the problem of de novo DNA sequencing from shot gun data, wherein an underlying (unknown) DNA sequence is to be reconstructed from several short substrings of the sequence.",
        "We propose a de novo assembly algorithm which requires only the minimum amount of data and is efficient with respect to space and computation.",
        "We design the algorithm from an information theoretic perspective of using minimum amount of data.",
        "The key idea to achieve space and computational efficiency is to break the procedure into two phases, an online and an offline phase.",
        "We remark that this can serve as an evidence of the feasibility of using an information-theoretic perspective to guide practical algorithmic design in DNA sequencing.",
        "Preliminary work on extending this algorithmic framework to more realistic settings is also reported."
      ]
    },
    {
      "title": "List Aggregation of Ranked Preferences",
      "abstract": [
        "We consider the problem of aggregating a given set of orderings (rankings) into a (smaller) set of aggregate orderings that maintains certain properties such as fairness.",
        "Classical results such as Arrow\u2019s impossibility theorem show the impossibility of doing so with just one aggregate ordering.",
        "This work thus considers doing so with a \u2018list\u2019 of aggregate orderings, and in particular, with a list of the smallest possible size, under a (stronger) notion of fairness.",
        "Upper and lower bounds on the size of such a list are derived, and these bounds are a logarithmic factor away from each other.",
        "The algorithm presented for aggregating these orderings also forms the basis of a new game \u2018Match to Race\u2019 available online."
      ]
    },
    {
      "title": "Secret Share Dissemination across a Network",
      "abstract": [
        "Secret sharing is an important component of several cryptographic protocols.",
        "These include protocols for secure multiparty function computation, key management, and secure archival storage.",
        "Most protocols assume that the dealer has direct communication links with every participant, in which case, the dealer can directly communicate the respective shares to all participants.",
        "In this paper, we consider the problem of disseminating shares of a secret when the dealer and the participants form a general network.",
        "We provide an algorithm for secret share dissemination that is communication-efficient, distributed and deterministic.",
        "Interestingly, the solution constitutes an instance of a network coding problem admitting a distributed and deterministic solution, and furthermore, handles the case of nodal-eavesdropping, about which very little appears to be known in the literature."
      ]
    },
    {
      "title": "On Minimizing Data-Read and Download for Storage-Node Recovery",
      "abstract": [
        "We consider the problem of efficient recovery of the data stored in any individual node of a distributed storage system, from the rest of the nodes.",
        "Applications include handling failures and degraded reads.",
        "We measure efficiency in terms of the amount of data-read and the download required.",
        "To minimize the download, we focus on the minimum bandwidth setting of the 'regenerating codes' model for distributed storage.",
        "Under this model, the system has a total of n nodes, and the data stored in any node must be (efficiently) recoverable from any d of the other (n-1) nodes.",
        "Lower bounds on the two metrics under this model were derived previously; it has also been shown that these bounds are achievable for the amount of data-read and download when d=n-1, and for the amount of download alone when d\u2260 n-1.",
        "In this paper, we complete the picture by proving the converse result, that when d\u2260 n-1, these lower bounds are strictly loose with respect to the amount of read required.",
        "The proof is information-theoretic, and hence applies to non-linear codes as well.",
        "We also show that under two (practical) relaxations of the problem setting, these lower bounds can be met for both read and download simultaneously."
      ]
    },
    {
      "title": "Regenerating codes for errors and erasures in distributed storage",
      "abstract": [
        "Regenerating codes are a class of codes proposed for providing reliability of data and efficient repair of failed nodes in distributed storage systems.",
        "In this paper, we address the fundamental problem of handling errors and erasures at the nodes or links, during the data-reconstruction and node-repair operations.",
        "We provide explicit regenerating codes that are resilient to errors and erasures, and show that these codes are optimal with respect to storage and bandwidth requirements.",
        "As a special case, we also establish the capacity of a class of distributed storage systems in the presence of malicious adversaries.",
        "While our code constructions are based on previously constructed Product-Matrix codes, we also provide necessary and sufficient conditions for introducing resilience in any regenerating code."
      ]
    },
    {
      "title": "Secret Sharing Across a Network with Low Communication Cost: Distributed Algorithm and Bounds",
      "abstract": [
        "Shamir's (n, k) threshold secret sharing is an important component of several cryptographic protocols, such as those for secure multiparty-computation and key management.",
        "These protocols typically assume the presence of direct communication links from the dealer to all participants, in which case the dealer can directly pass the shares of the secret to each participant.",
        "In this paper, we consider the problem of secret sharing when the dealer does not have direct communication links to all the participants, and instead, the dealer and the participants form a general network.",
        "Existing methods are based on secure message transmissions from the dealer to each participant requiring considerable coordination in the network.",
        "In this paper, we present a distributed algorithm for disseminating shares over a network, which we call the SNEAK algorithm, requiring each node to know only the identities of its one-hop neighbours.",
        "While SNEAK imposes a stronger condition on the network by requiring the dealer to be what we call k-propagating rather than k-connected as required by the existing solutions, we show that in addition to being distributed, SNEAK achieves significant reduction in the communication cost and the amount of randomness required."
      ]
    },
    {
      "title": "Distributed Secret Dissemination Across a Network",
      "abstract": [
        "Shamir's (n,k) threshold secret sharing is an important component of several cryptographic protocols, such as those for secure multiparty-computation and key management.",
        "These protocols typically assume the presence of direct communication links from the dealer to all participants, in which case the dealer can directly pass the shares of the secret to each participant.",
        "In this paper, we consider the problem of secret sharing when the dealer does not have direct communication links to all the participants, and instead, the dealer and the participants form a general network.",
        "Existing methods are based on separate secure message transmissions from the dealer to each participant, requiring considerable coordination and communication in the network.",
        "We present a distributed algorithm for disseminating shares over a network, which we call the \u201cSNEAK\u201d algorithm, requiring each node to know only the identities of its one-hop neighbors.",
        "While SNEAK imposes a stronger condition on the network by requiring the dealer to be what we call k-propagating rather than k-connected as required by the existing solutions, we show that in addition to being distributed, it achieves significant reduction in the amount of communication and the randomness required.",
        "We also derive information-theoretic lower bounds on the amount of communication for secret sharing over networks, which may be of independent interest."
      ]
    },
    {
      "title": "Efficient and Distributed Secret Sharing in General Networks",
      "abstract": [
        "Shamir's (n, k) threshold secret sharing is an important component of several cryptographic protocols, such as those for secure multiparty-computation, key management, and Byzantine agreement.",
        "These protocols typically assume the presence of direct communication links from the dealer to all participants, in which case, the dealer can directly pass the shares of the secret to each participant.",
        "In this paper, we consider the problem of secret sharing when the dealer does not have direct links to all the participants, and instead, the dealer and the participants form a general network.",
        "We present an efficient and distributed algorithm for secret sharing over general networks that satisfy what we call the k-propagating-dealer condition.",
        "\nWe derive information-theoretic lower bounds on the communication complexity of secret sharing over any network, which may also be of independent interest.",
        "We show that for networks satisfying the k-propagating-dealer condition, the communication complexity of our algorithm is {\\Theta}(n), and furthermore, is a constant factor away from the lower bounds.",
        "We also show that, in contrast, the existing solution entails a communication-complexity that is super-linear for a wide class of networks, and is {\\Theta}(n^2) in the worst case.",
        "Moreover, the amount of randomness required under our algorithm is a constant, while that required under the existing solution increases with n for a large class of networks, and in particular, is {\\Theta}(n) whenever the degree of the dealer is bounded.",
        "Finally, while the existing solution requires considerable coordination in the network and knowledge of the global topology, our algorithm is completely distributed and requires each node to know only the identities of its neighbours.",
        "Our algorithm thus allows for efficient generalization of several cryptographic protocols to a large class of general networks."
      ]
    },
    {
      "title": "The MDS Queue: Analysing Latency Performance of Codes and Redundant Requests",
      "abstract": [
        "In order to scale economically, data centers are increasingly evolving their data storage methods from the use of simple data replication to the use of more powerful erasure codes, which provide the same level of reliability as replication-based methods at a significantly lower storage cost.",
        "In particular, it is well known that MaximumDistance-Separable (MDS) codes, such as Reed-Solomon codes, provide the maximum storage efficiency.",
        "While the use of codes for providing improved reliability in archival storage systems, where the data is less frequently accessed (or so-called \u201ccold data\u201d), is well understood, the role of codes in the storage of more frequently accessed and active \u201chot data\u201d, where latency is the key metric, is less clear.",
        "In this paper, we study data storage systems based on MDS codes through the lens of queueing theory, and term this the \u201cMDS queue.\u201d",
        "We analytically characterize the latency performance of MDS queues, for which we present insightful scheduling policies that form upper and lower bounds to performance, and show that they are quite tight.",
        "Extensive simulations using Monte Carlo methods are also provided and used to validate our theoretical analysis.",
        "As a side note, our lower-bound analytical method based on the so-called MDS-Reservation(t) queue, represents an elegant practical scheme that requires the maintenance of considerably smaller state, depending on the parameter t, than that of the full-fledged MDS queue (which corresponds to t =\u221e), and may be of independent interest in practical systems.",
        "Comparisons with replication-based systems reveal that codes provide a superior latency-performance (by up to 70%) than replication.",
        "The second part of the paper considers an alternative method of (potentially) reducing latency in data centers, that of sending redundant requests.",
        "Here, a request is sent to more servers than required, and is deemed served when any requisite number of servers complete service.",
        "Several recent works provide empirical evidence of the benefits of redundant requests in various settings, and in this paper, we aim to analytically characterize the situations when can redundant requests actually help.",
        "We show that under the MDS queue model (with exponential service times and negligible costs of cancelling jobs), in a replication-based system, the average latency strictly reduces with more redundancy in the requests, and that under a general MDS code, the average latency is minimized when requests are sent to all servers.",
        "To the best of our knowledge, these are the first analytical results that prove the benefits of sending redundant requests."
      ]
    },
    {
      "title": "The MDS queue: Analysing the latency performance of erasure codes",
      "abstract": [
        "In order to scale economically, data centers are increasingly evolving their data storage methods from the use of simple data replication to the use of more powerful erasure codes, which provide the same level of reliability as replication but at a significantly lower storage cost.",
        "In particular, it is well known that Maximum-Distance-Separable (MDS) codes, such as Reed-Solomon codes, provide the maximum storage efficiency.",
        "While the use of codes for providing improved reliability in archival storage systems, where data is less frequently accessed (or so-called \u201ccold data\u201d), is well understood, the role of codes in the storage of more frequently accessed and active \u201chot data\u201d, where latency is the key metric, is less clear.",
        "In this paper, we study data storage systems based on MDS codes through the lens of queueing theory, and term the queueing system arising under codes as an \u201cMDS queue.\u201d",
        "We present insightful scheduling policies that form upper and lower bounds to its performance, and use these to obtain easily computable analytical bounds on the average latency of the MDS queue.",
        "These bounds were observed to be quite tight in the settings we simulated.",
        "We additionally derive closed-form expressions of the throughputs of these systems.",
        "Finally, we employ the framework of the MDS queue to analyse different methods of performing so-called degraded reads (reading of partial data) in distributed data storage."
      ]
    },
    {
      "title": "Information-Theoretically Secure Regenerating Codes for Distributed Storage",
      "abstract": [
        "Regenerating codes are a class of codes for distributed storage networks that provide reliability and availability of data, and also perform efficient node repair.",
        "Another important aspect of a distributed storage network is its security.",
        "In this paper, we consider a threat model where an eavesdropper may gain access to the data stored in a subset of the storage nodes, and possibly also, to the data downloaded during repair of some nodes.",
        "We provide explicit constructions of regenerating codes that achieve information-theoretic secrecy capacity in this setting."
      ]
    },
    {
      "title": "Enabling node repair in any erasure code for distributed storage",
      "abstract": [
        "Erasure codes are an efficient means of storing data across a network in comparison to data replication, as they tend to reduce the amount of data stored in the network and offer increased resilience in the presence of node failures.",
        "The codes perform poorly though, when repair of a failed node is called for, as they typically require the entire file to be downloaded to repair a failed node.",
        "A new class of erasure codes, termed as regenerating codes were recently introduced, that do much better in this respect.",
        "However, given the variety of efficient erasure codes available in the literature, there is considerable interest in the construction of coding schemes that would enable traditional erasure codes to be used, while retaining the feature that only a fraction of the data need be downloaded for node repair.",
        "In this paper, we present a simple, yet powerful, framework that does precisely this.",
        "Under this framework, the nodes are partitioned into two types and encoded using two codes in a manner that reduces the problem of node-repair to that of erasure-decoding of the constituent codes.",
        "Depending upon the choice of the two codes, the framework can be used to avail one or more of the following advantages: simultaneous minimization of storage space and repair-bandwidth, low complexity of operation, fewer disk reads at helper nodes during repair, and error detection and correction."
      ]
    },
    {
      "title": "Interference alignment as a tool in network coding as applied to distributed storage",
      "abstract": [
        "In this paper, we outline an approach to the task of designing network codes in a non-multicast setting.",
        "Our approach makes use of the concept of interference alignment.",
        "As an example, we consider the distributed storage problem where the data is stored across the network in n nodes and where a data collector can recover the data by connecting to any k of the n nodes and where furthermore, upon failure of a node, a new node can replicate the data stored in the failed node while minimizing the repair bandwidth."
      ]
    },
    {
      "title": "Explicit and optimal codes for distributed storage",
      "abstract": [
        "In the distributed storage coding problem we consider, data is stored across n nodes in a network, each capable of storing \u03b1 symbols.",
        "It is required that the complete data can be reconstructed by downloading data from any k nodes.",
        "There is also the key additional requirement that a failed node be regenerated by connecting to any d nodes and downloading \u03b2 symbols from each of them.",
        "Our goal is to minimize the repair bandwidth d\u03b2.",
        "In this paper we provide explicit constructions for several parameter sets of interest."
      ]
    },
    {
      "title": "A flexible class of regenerating codes for distributed storage",
      "abstract": [
        "In the distributed storage setting introduced by Dimakis et al., B units of data are stored across n nodes in the network in such a way that the data can be recovered by connecting to any k nodes.",
        "Additionally one can repair a failed node by connecting to any d nodes while downloading at most \u03b2 units of data from each node.",
        "In this paper, we introduce a flexible framework in which the data can be recovered by connecting to any number of nodes as long as the total amount of data downloaded is at least B. Similarly, regeneration of a failed node is possible if the new node connects to the network using links whose individual capacity is bounded above by \u03b2max and whose sum capacity equals or exceeds a predetermined parameter \u03b3.",
        "In this flexible setting, we obtain the cut-set lower bound on the repair bandwidth along with a constructive proof for the existence of codes meeting this bound for all values of the parameters.",
        "An explicit code construction is provided which is optimal in certain parameter regimes."
      ]
    },
    {
      "title": "Explicit and optimal exact-regenerating codes for the minimum-bandwidth point in distributed storage",
      "abstract": [
        "In the distributed storage setting that we consider, data is stored across n nodes in the network such that the data can be recovered by connecting to any subset of k nodes.",
        "Additionally, one can repair a failed node by connecting to any d nodes while downloading \u03b2 units of data from each.",
        "Dimakis et al. show that the repair bandwidth d\u03b2 can be considerably reduced if each node stores slightly more than the minimum required and characterize the tradeoff between the amount of storage per node and the repair bandwidth."
      ]
    },
    {
      "title": "Interference Alignment in Regenerating Codes for Distributed Storage: Necessity and Code Constructions",
      "abstract": [
        "Regenerating codes are a class of recently developed codes for distributed storage that, like Reed-Solomon codes, permit data recovery from any arbitrary k of n nodes.",
        "However regenerating codes possess in addition, the ability to repair a failed node by connecting to any arbitrary d nodes and downloading an amount of data that is typically far less than the size of the data file.",
        "This amount of download is termed the repair bandwidth.",
        "Minimum storage regenerating (MSR) codes are a subclass of regenerating codes that require the least amount of network storage; every such code is a maximum distance separable (MDS) code.",
        "Further, when a replacement node stores data identical to that in the failed node, the repair is termed as exact.",
        "The four principal results of the paper are (a) the explicit construction of a class of MDS codes for d = n - 1 \u2265 2k - 1 termed the MISER code, that achieves the cut-set bound on the repair bandwidth for the exact repair of systematic nodes, (b) proof of the necessity of interference alignment in exact-repair MSR codes, (c) a proof showing the impossibility of constructing linear, exact-repair MSR codes for d <; 2k - 3 in the absence of symbol extension, and (d) the construction, also explicit, of high-rate MSR codes for d = k + 1.",
        "Interference alignment (IA) is a theme that runs throughout the paper: the MISER code is built on the principles of IA and IA is also a crucial component to the nonexistence proof for d <; 2k - 3.",
        "To the best of our knowledge, the constructions presented in this paper are the first explicit constructions of regenerating codes that achieve the cut-set bound."
      ]
    },
    {
      "title": "Explicit codes uniformly reducing repair bandwidth in distributed storage",
      "abstract": [
        "A distributed storage setting is considered where a file of size B is to be stored across n storage nodes.",
        "A data collector should be able to reconstruct the entire data by downloading the symbols stored in any k nodes.",
        "When a node fails, it is replaced by a new node by downloading data from some of the existing nodes.",
        "The amount of download is termed as repair bandwidth.",
        "One way to implement such a system is to store one fragment of an (n, k) MDS code in each node, in which case the repair bandwidth is B. Since repair of a failed node consumes network bandwidth, codes reducing repair bandwidth are of great interest.",
        "Most of the recent work in this area focuses on reducing the repair bandwidth of a set of k nodes which store the data in uncoded form, while the reduction in the repair bandwidth of the remaining nodes is only marginal.",
        "In this paper, we present an explicit code which reduces the repair bandwidth for all the nodes to approximately B/2.",
        "To the best of our knowledge, this is the first explicit code which reduces the repair bandwidth of all the nodes for all feasible values of the system parameters."
      ]
    },
    {
      "title": "Optimal Exact-Regenerating Codes for Distributed Storage at the MSR and MBR Points via a Product-Matrix Construction",
      "abstract": [
        "Regenerating codes are a class of distributed storage codes that allow for efficient repair of failed nodes, as compared to traditional erasure codes.",
        "An [n, k, d] regenerating code permits the data to be recovered by connecting to any k of the n nodes in the network, while requiring that a failed node be repaired by connecting to any d nodes.",
        "The amount of data downloaded for repair is typically much smaller than the size of the source data.",
        "Previous constructions of exact-regenerating codes have been confined to the case n=d+1 .",
        "In this paper, we present optimal, explicit constructions of (a) Minimum Bandwidth Regenerating (MBR) codes for all values of [n, k, d] and (b) Minimum Storage Regenerating (MSR) codes for all [n, k, d \u2265 2k-2], using a new product-matrix framework.",
        "The product-matrix framework is also shown to significantly simplify system operation.",
        "To the best of our knowledge, these are the first constructions of exact-regenerating codes that allow the number n of nodes in the network, to be chosen independent of the other parameters.",
        "The paper also contains a simpler description, in the product-matrix framework, of a previously constructed MSR code with [n=d+1, k, d \u2265 2k-1]."
      ]
    },
    {
      "title": "Distributed Storage Codes With Repair-by-Transfer and Nonachievability of Interior Points on the Storage-Bandwidth Tradeoff",
      "abstract": [
        "Regenerating codes are a class of recently developed codes for distributed storage that, like Reed-Solomon codes, permit data recovery from any subset of nodes within the -node network.",
        "However, regenerating codes possess in addition, the ability to repair a failed node by connecting to an arbitrary subset of nodes.",
        "It has been shown that for the case of functional repair, there is a tradeoff between the amount of data stored per node and the bandwidth required to repair a failed node.",
        "A special case of functional repair is exact repair where the replacement node is required to store data identical to that in the failed node.",
        "Exact repair is of interest as it greatly simplifies system implementation.",
        "The first result of this paper is an explicit, exact-repair code for the point on the storage-bandwidth tradeoff corresponding to the minimum possible repair bandwidth, for the case when .",
        "This code has a particularly simple graphical description, and most interestingly has the ability to carry out exact repair without any need to perform arithmetic operations.",
        "We term this ability of the code to perform repair through mere transfer of data as repair by transfer.",
        "The second result of this paper shows that the interior points on the storage-bandwidth tradeoff cannot be achieved under exact repair, thus pointing to the existence of a separate tradeoff under exact repair.",
        "Specifically, we identify a set of scenarios which we term as \u201chelper node pooling,\u201d and show that it is the necessity to satisfy such scenarios that overconstrains the system."
      ]
    },
    {
      "title": "Explicit construction of optimal exact regenerating codes for distributed storage",
      "abstract": [
        "Erasure coding techniques are used to increase the reliability of distributed storage systems while minimizing storage overhead.",
        "Also of interest is minimization of the bandwidth required to repair the system following a node failure.",
        "In a recent paper, Wu et al. characterize the tradeoff between the repair bandwidth and the amount of data stored per node.",
        "They also prove the existence of regenerating codes that achieve this tradeoff.",
        "In this paper, we introduce Exact Regenerating Codes, which are regenerating codes possessing the additional property of being able to duplicate the data stored at a failed node.",
        "Such codes require low processing and communication overheads, making the system practical and easy to maintain.",
        "Explicit construction of exact regenerating codes is provided for the minimum bandwidth point on the storage-repair bandwidth tradeoff, relevant to distributed-mail-server applications.",
        "A subspace based approach is provided and shown to yield necessary and sufficient conditions on a linear code to possess the exact regeneration property as well as prove the uniqueness of our construction.",
        "Also included in the paper, is an explicit construction of regenerating codes for the minimum storage point for parameters relevant to storage in peer-to-peer systems.",
        "This construction supports a variable number of nodes and can handle multiple, simultaneous node failures.",
        "All constructions given in the paper are of low complexity, requiring low field size in particular."
      ]
    },
    {
      "title": "Explicit codes minimizing repair bandwidth for distributed storage",
      "abstract": [
        "We consider the problem of minimizing the bandwidth required to repair a failed node when data is stored across n nodes in a distributed manner, so as to facilitate reconstruction of the entire data by connecting to any k out of the n nodes.",
        "We provide explicit and optimal constructions which permit exact replication of a failed systematic node."
      ]
    },
    {
      "title": "Exact Regenerating Codes for Distributed Storage",
      "abstract": [
        "Erasure coding techniques are used to increase the reliability of distributed storage systems while minimizing storage overhead.",
        "The bandwidth required to repair the system after a node failure also plays a crucial role in the system performance.",
        "In [1] authors have shown that a tradeoff exists between storage and repair bandwidth.",
        "They also have introduced the scheme of regenerating codes which meet this tradeoff.",
        "In this paper, a scheme of Exact Regenerating Codes is introduced, which are regenerating codes with an additional property of regenerating back the same node upon failure.",
        "For the minimum bandwidth point, which is suitable for applications like distributed mail servers, explicit construction for exact regenerating codes is provided.",
        "A subspace approach is provided, using which the necessary and sufficient conditions for a linear code to be an exact regenerating code are derived.",
        "This leads to the uniqueness of our construction.",
        "For the minimum storage point which suits applications such as storage in peer-to-peer systems, an explicit construction of regenerating codes for certain suitable parameters is provided.",
        "This code supports variable number of nodes and can handle multiple simultaneous node failures.",
        "The constructions given for both the points require a low field size and have low complexity."
      ]
    }
  ],
  "user_kps": [
    "advanced bibliometric methods",
    "citations",
    "cooperative regenerating codes",
    "crowd labeling",
    "crowdsourced data",
    "crowdsourced knowledge",
    "crowdsourced tasks",
    "distributed storage codes",
    "fault-tolerant storage",
    "highest-scoring documents",
    "latency demands",
    "locally repairable codes",
    "malicious recommendation",
    "peer selection",
    "population bias",
    "ranking mechanisms",
    "review selection",
    "reviewers",
    "secret sharing algorithms",
    "truthful incentive mechanisms"
  ]
}