Publications

2023

1. arXiv

   “It’s a Fair Game”, or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agents

   Zhiping Zhang, Michelle Jia, Hao-Ping (Hank) Lee, Bingsheng Yao, Sauvik Das, Ada Lerner, Dakuo Wang, and Tianshi Li

   Sep 2023

   Abs arXiv

   The widespread use of Large Language Model (LLM)-based conversational agents (CAs), especially in high-stakes domains, raises many privacy concerns. Building ethical LLM-based CAs that respect user privacy requires an in-depth understanding of the privacy risks that concern users the most. However, existing research, primarily model-centered, does not provide insight into users’ perspectives. To bridge this gap, we analyzed sensitive disclosures in real-world ChatGPT conversations and conducted semi-structured interviews with 19 LLM-based CA users. We found that users are constantly faced with trade-offs between privacy, utility, and convenience when using LLM-based CAs. However, users’ erroneous mental models and the dark patterns in system design limited their awareness and comprehension of the privacy risks. Additionally, the human-like interactions encouraged more sensitive disclosures, which complicated users’ ability to navigate the trade-offs. We discuss practical design guidelines and the needs for paradigmatic shifts to protect the privacy of LLM-based CA users.

2. SOUPS Poster

   Measuring the Effectiveness of Spinner-based Randomized-Response Differential Privacy Communication for Sensitive Data Sharing

   Seoyoung Ko, Sriram Viswanathan, Alan Esquenazi, Swadhin Routray, Jatan Loya, Tianshi Li, and Lorrie Faith Cranor

   In SOUPS 2023 Poster Aug 2023

   HTML
3. SOUPS Poster

   Analyzing Developers’ Perceptions, Attitudes and Challenges in Complying with Mobile App Store Privacy Requirements

   Geetika Gopi, Peter Meyers, Thanyanun Charoensiritanasin, Jayson Jin, Ragashree Mysuru Chandrashekar, Lorrie Faith Cranor, and Tianshi Li

   In SOUPS 2023 Poster Aug 2023

   HTML
4. PhD Dissertation

   Privacy Annotations: Designing Privacy Support for Developers

   Tianshi Li

   Aug 2023

   Abs PDF

   While data has driven many technological advancements, the ubiquitous collec- tion and sharing of data have caused a privacy trust crisis in our society. Developers play a crucial role in creating apps that respect user expectations and data usage norms, as they have a deep understanding of app behavior and can adjust the design accordingly. However, developers are not privacy experts. Developing a privacy- friendly app is often a challenging task due to their lack of 1) awareness of privacy issues, 2) knowledge of privacy best practices, and 3) time for handling privacy re- quirements. These problems have become more and more salient with the advent of a flurry of privacy requirements from platform providers (e.g., Google Play and Apple App Store) and laws (e.g., GDPR, CCPA), creating urgent needs for effective, opportune, and usable privacy support for developers. Hence, I propose Privacy Support for Developers as a new area of interest at the intersection of privacy, HCI, and software engineering research. The first challenge is that although there has been some research on developers’ challenges for handling privacy requirements, they tend to be more descriptive than prescriptive. Therefore, our community still lacks a clear direction of how to solve the problems. To fill in this gap, I first synthesize developers’ needs for designing privacy-enhancing devel- oper support based on my work and past literature to provide a roadmap for future explorations into this problem. Informed by the identified needs, I present my work that pioneers a novel type of developer tooling: Privacy-Enhancing Integrated Development Environment (IDE) Plugins. I propose privacy annotation, a type of structured metadata that embeds pri- vacy information such as data use purposes directly in code. Based on this concept, I designed, implemented, and evaluated three plugins for Android Studio, the official IDE for Android development, to increase developers’ awareness and knowledge of privacy best practices and to reduce the work required for complying with privacy requirements. With one set of annotations, my tools offer privacy support in multiple aspects, including 1) detection of sensitive API calls and third-party SDKs to support accurate understanding, documentation, and disclosure of data practices, 2) just-in- time reminders and lightweight code repair features (quick-fixes) to help develop- ers conform to best practices, and 3) annotation-based declarative programming to generate in-app privacy notices and privacy nutrition labels required by app stores. My studies demonstrated that my tools effectively improved developers’ awareness and adoption of privacy best practices, reduced the workload for completing privacy compliance tasks, and enhanced the accuracy of the generated privacy notices.

5. CSCW SIG

   Shaping the Emerging Norms of Using Large Language Models in Social Computing Research

   Hong Shen, Tianshi Li, Toby Jia-jun Li, Joon Sung Park, and Diyi Yang

   In Computer Supported Cooperative Work and Social Computing (CSCW’23 Companion) Oct 2023

   Abs HTML PDF

   The emergence of Large Language Models (LLMs) has brought both excitement and concerns to social computing research. On the one hand, LLMs offer unprecedented capabilities in analyzing vast amounts of textual data and generating human-like responses, enabling researchers to delve into complex social phenomena. On the other hand, concerns are emerging regarding the validity, privacy, and ethics of the research when LLMs are involved. This SIG aims at offering an open space for social computing researchers who are interested in understanding the impacts of LLMs to discuss their current practices, perspectives, challenges when engaging with LLMs in their everyday work and collectively shaping the emerging norms of using LLMs in social computing research.

2022

1. CHI

   Understanding Challenges for Developers to Create Accurate Privacy Nutrition Labels

   Tianshi Li, Kayla Reiman, Yuvraj Agarwal, Lorrie Faith Cranor, and Jason I Hong

   In CHI Conference on Human Factors in Computing Systems Apr 2022

   Best Paper Honorable Mention Award 🏅

   Abs HTML PDF

   Apple announced the introduction of app privacy details to their App Store in December 2020, marking the first ever real-world, large-scale deployment of the privacy nutrition label concept, which had been introduced by researchers over a decade earlier. The Apple labels are created by app developers, who self-report their app’s data practices. In this paper, we present the first study examining the usability and understandability of Apple’s privacy nutrition label creation process from the developer’s perspective. By observing and interviewing 12 iOS app developers about how they created the privacy label for a real-world app that they developed, we identified common challenges for correctly and efficiently creating privacy labels. We discuss design implications both for improving Apple’s privacy label design and for future deployment of other standardized privacy notices.

2. PETS

   Understanding privacy-related advice on Stack Overflow

   Mohammad Tahaei, Tianshi Li, and Kami Vaniea

   Proc. Priv. Enhancing Technol. Apr 2022

   Abs HTML PDF

   Abstract Privacy tasks can be challenging for developers, resulting in privacy frameworks and guidelines from the research community which are designed to assist developers in considering privacy features and applying privacy enhancing technologies in early stages of software development. However, how developers engage with privacy design strategies is not yet well understood. In this work, we look at the types of privacy-related advice developers give each other and how that advice maps to Hoepman’s privacy design strategies. We qualitatively analyzed 119 privacy-related accepted answers on Stack Overflow from the past five years and extracted 148 pieces of advice from these answers. We find that the advice is mostly around compliance with regulations and ensuring confidentiality with a focus on the inform, hide, control, and minimize of the Hoepman’s privacy design strategies. Other strategies, abstract, separate, enforce, and demonstrate, are rarely advised. Answers often include links to official documentation and online articles, highlighting the value of both official documentation and other informal materials such as blog posts. We make recommendations for promoting the under-stated strategies through tools, and detail the importance of providing better developer support to handle third-party data practices.

3. PETS

   Charting app developers’ journey through privacy regulation features in ad networks

   Mohammad Tahaei, Kopo M Ramokapane, Tianshi Li, Jason I Hong, and Awais Rashid

   Proc. Priv. Enhancing Technol. Jul 2022

   Abs HTML PDF

   Mobile apps enable ad networks to collect and track users. App developers are given “configurations” on these platforms to limit data collection and adhere to privacy regulations; however, the prevalence of apps that violate privacy regulations because of third parties, including ad networks, begs the question of how developers work through these configurations and how easy they are to utilize. We study privacy regulations-related interfaces on three widely used ad networks using two empirical studies, a systematic review and think-aloud sessions with eleven developers, to shed light on how ad networks present privacy regulations and how usable the provided configurations are for developers. We find that information about privacy regulations is scattered in several pages, buried under multiple layers, and uses terms and language developers do not understand. While ad networks put the burden of complying with the regulations on developers, our participants, on the other hand, see ad networks responsible for ensuring compliance with regulations. To assist developers in building privacy regulations-compliant apps, we suggest dedicating a section to privacy, offering easily accessible configurations (both in graphical and code level), building testing systems for privacy regulations, and creating multimedia materials such as videos to promote privacy values in the ad networks’ documentation.

4. TOCHI

   Alert now or never: Understanding and predicting notification preferences of smartphone users

   Tianshi Li, Julia Katherine Haines, Miguel Flores Ruiz Eguino, Jason I Hong, and Jeffrey Nichols

   ACM Trans. Comput. Hum. Interact. Feb 2022

   Abs HTML PDF

   Notifications are an indispensable feature of mobile devices, but their delivery can interrupt and distract users. Prior work has examined interventions, such as deferring notification delivery to opportune moments, but has not systematically studied how users might prefer an intelligent system to manage their notifications. Hence, we directly probed Android smartphone users’ notification preferences via a one-week experience-sampling study ( N = 35). We found that users prefer mitigating undesired interruptions by suppressing alerts over deferring them and referred to notification content factors more frequently than contextual factors for explaining their preferences. Then we demonstrated the challenges and potentials of leveraging user actions to help predict notification preferences. Specifically, we showed that a model personalized using user actions achieved a performance gain of 39% than a generic model. This improvement is similar to the 42% performance gain using labels solicited from the user while using observable user actions causes no extra disruption.

5. TOCHI

   C-PAK: Correcting and completing variable-length Prefix-based Abbreviated Keystrokes

   Tianshi Li, Philip Quinn, and Shumin Zhai

   ACM Trans. Comput. Hum. Interact. Jul 2022

   Abs HTML PDF

   Improving keystroke savings is a long-term goal of text input research. We present a study into the design space of an abbreviated style of text input called C-PAK (Correcting and completing variable-length Prefix-based Abbreviated Keystrokes) for text entry on mobile devices. Given a variable length and potentially inaccurate input string (e.g. “li g t m”), C-PAK aims to expand it into a complete phrase (e.g. “looks good to me”). We develop a C-PAK prototype keyboard, PhraseWriter , based on a current state-of-the-art mobile keyboard consisting of 1.3 million n-grams and 164,000 words. Using computational simulations on a large dataset of realistic input text, we found that, in comparison to conventional single-word suggestions, PhraseWriter improves the maximum keystroke savings rate by 6.7% (from \(46.3% \) to \(49.4% \) ), reduces the word error rate by 14.7%, and is particularly advantageous for common phrases. We conducted a lab study of novice user behavior and performance which found that users could quickly utilize the C-PAK style abbreviations implemented in PhraseWriter, achieving a higher keystroke savings rate than forward suggestions (25% vs. 16%). Furthermore, they intuitively and successfully abbreviated more with common phrases. However, users had a lower overall text entry rate due to their limited experience with the system (28.5 words per minute vs. 37.7). We outline future technical directions to improve C-PAK over the PhraseWriter baseline, and further opportunities to study the perceptual, cognitive, and physical action trade-offs that underlie the learning curve of C-PAK systems.

6. CHI LBW

   Understanding iOS privacy nutrition labels: An exploratory large-scale analysis of app store data

   Yucheng Li, Deyuan Chen, Tianshi Li, Yuvraj Agarwal, Lorrie Faith Cranor, and Jason I Hong

   In CHI Conference on Human Factors in Computing Systems Extended Abstracts Apr 2022

   HTML PDF

2021

1. arXiv

   The design of the user interfaces for Privacy Enhancements for Android

   Jason I Hong, Yuvraj Agarwal, Matt Fredrikson, Mike Czapik, Shawn Hanna, Swarup Sahoo, Judy Chun, Won-Woo Chung, and 17 more authors

   Apr 2021

   Abs arXiv

   We present the design and design rationale for the user interfaces for Privacy Enhancements for Android (PE for Android). These UIs are built around two core ideas, namely that developers should explicitly declare the purpose of why sensitive data is being used, and these permission-purpose pairs should be split by first party and third party uses. We also present a taxonomy of purposes and ways of how these ideas can be deployed in the existing Android ecosystem.

2. IMWUT

   Honeysuckle: Annotation-Guided Code Generation of In-App Privacy Notices

   Tianshi Li, Elijah B Neundorfer, Yuvraj Agarwal, and Jason I Hong

   Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. Sep 2021

   Abs HTML PDF Talk Demo

   In-app privacy notices can help smartphone users make informed privacy decisions. However, they are rarely used in real-world apps, since developers often lack the knowledge, time, and resources to design and implement them well. We present Honeysuckle, a programming tool that helps Android developers build in-app privacy notices using an annotation-based code generation approach facilitated by an IDE plugin, a build system plugin, and a library. We conducted a within-subjects study with 12 Android developers to evaluate Honeysuckle. Each participant was asked to implement privacy notices for two popular open-source apps using the Honeysuckle library as a baseline as well as the annotation-based approach. Our results show that the annotation-based approach helps developers accomplish the task faster with significantly lower cognitive load. Developers preferred the annotation-based approach over the library approach because it was much easier to learn and use and allowed developers to achieve various types of privacy notices using a unified code format, which can enhance code readability and benefit team collaboration.

3. CSCW

   How Developers Talk About Personal Data and What It Means for User Privacy: A Case Study of a Developer Forum on Reddit

   Tianshi Li, Elizabeth Louie, Laura Dabbish, and Jason I Hong

   Proc. ACM Hum. Comput. Interact. Jan 2021

   Abs HTML PDF Talk

   While online developer forums are major resources of knowledge for application developers, their roles in promoting better privacy practices remain underexplored. In this paper, we conducted a qualitative analysis of a sample of 207 threads (4772 unique posts) mentioning different forms of personal data from the /r/androiddev forum on Reddit. We started with bottom-up open coding on the sampled posts to develop a typology of discussions about personal data use and conducted follow-up analyses to understand what types of posts elicited in-depth discussions on privacy issues or mentioned risky data practices. Our results show that Android developers rarely discussed privacy concerns when talking about a specific app design or implementation problem, but often had active discussions around privacy when stimulated by certain external events representing new privacy-enhancing restrictions from the Android operating system, app store policies, or privacy laws. Developers often felt these restrictions could cause considerable cost yet fail to generate any compelling benefit for themselves. Given these results, we present a set of suggestions for Android OS and the app store to design more effective methods to enhance privacy, and for developer forums(e.g., /r/androiddev) to encourage more in-depth privacy discussions and nudge developers to think more about privacy.

4. PMC

   What makes people install a COVID-19 contact-tracing app? Understanding the influence of app design and individual difference on contact-tracing app adoption intention

   Tianshi Li, Camille Cobb, Jackie (junrui) Yang, Sagar Baviskar, Yuvraj Agarwal, Beibei Li, Lujo Bauer, and Jason I Hong

   Pervasive Mob. Comput. Aug 2021

   Best Research Paper 2019-2021 Award 🏆

   Abs HTML PDF

   Smartphone-based contact-tracing apps are a promising solution to help scale up the conventional contact-tracing process. However, low adoption rates have become a major issue that prevents these apps from achieving their full potential. In this paper, we present a national-scale survey experiment (N=1963) in the U.S. to investigate the effects of app design choices and individual differences on COVID-19 contact-tracing app adoption intentions. We found that individual differences such as prosocialness, COVID-19 risk perceptions, general privacy concerns, technology readiness, and demographic factors played a more important role than app design choices such as decentralized design vs. centralized design, location use, app providers, and the presentation of security risks. Certain app designs could exacerbate the different preferences in different sub-populations which may lead to an inequality of acceptance to certain app design choices (e.g., developed by state health authorities vs. a large tech company) among different groups of people (e.g., people living in rural areas vs. people living in urban areas). Our mediation analysis showed that one’s perception of the public health benefits offered by the app and the adoption willingness of other people had a larger effect in explaining the observed effects of app design choices and individual differences than one’s perception of the app’s security and privacy risks. With these findings, we discuss practical implications on the design, marketing, and deployment of COVID-19 contact-tracing apps in the U.S.

2020

1. arXiv

   Decentralized is not risk-free: Understanding public perceptions of privacy-utility trade-offs in COVID-19 contact-tracing apps

   Tianshi Li,  Jackie,  Yang, Cori Faklaris, Jennifer King, Yuvraj Agarwal, Laura Dabbish, and Jason I Hong

   May 2020

   Abs arXiv

   Contact-tracing apps have potential benefits in helping health authorities to act swiftly to halt the spread of COVID-19. However, their effectiveness is heavily dependent on their installation rate, which may be influenced by people’s perceptions of the utility of these apps and any potential privacy risks due to the collection and releasing of sensitive user data (e.g., user identity and location). In this paper, we present a survey study that examined people’s willingness to install six different contact-tracing apps after informing them of the risks and benefits of each design option (with a U.S.-only sample on Amazon Mechanical Turk, N=208). The six app designs covered two major design dimensions (centralized vs decentralized, basic contact tracing vs. also providing hotspot information), grounded in our analysis of existing contact-tracing app proposals. Contrary to assumptions of some prior work, we found that the majority of people in our sample preferred to install apps that use a centralized server for contact tracing, as they are more willing to allow a centralized authority to access the identity of app users rather than allowing tech-savvy users to infer the identity of diagnosed users. We also found that the majority of our sample preferred to install apps that share diagnosed users’ recent locations in public places to show hotspots of infection. Our results suggest that apps using a centralized architecture with strong security protection to do basic contact tracing and providing users with other useful information such as hotspots of infection in public places may achieve a high adoption rate in the U.S.

2. CHI EA

   Analyzing the Role of General App Creators in Protecting the Privacy of Vulnerable Populations

   Tianshi Li, and Jason I. Hong

   Apr 2020

2019

1. SIGMETRICS

   Demystifying Complex Workload–DRAM Interactions: An Experimental Study

   Saugata Ghose, Tianshi Li, Nastaran Hajinazar, Damla Senol Cali, and Onur Mutlu

   Proc. ACM meas. anal. comput. syst. Dec 2019

   HTML PDF

2018

1. IMWUT

   Coconut: An IDE Plugin for Developing Privacy-Friendly Apps

   Tianshi Li, Yuvraj Agarwal, and Jason I Hong

   Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. Dec 2018

   Abs HTML PDF Demo Website

   Although app developers are responsible for protecting users’ privacy, this task can be very challenging. In this paper, we present Coconut, an Android Studio plugin that helps developers handle privacy requirements by engaging developers to think about privacy during the development process and providing real-time feedback on potential privacy issues. We start by presenting new findings based on a series of semi-structured interviews with Android developers, probing into the difficulties with privacy that developers face when building apps. Based on these findings, we implemented a proof-of-concept prototype of Coconut and evaluated it in a controlled lab study with 18 Android developers (including eight professional developers). Our study results suggest that apps developed with Coconut handled privacy concerns better, and the developers that used Coconut had a better understanding of their code’s behavior and wrote a better privacy policy for their app. We also found that requiring developers to do a small amount of annotating work regarding their apps’ personal data practices during the development process may result in a significant improvement in app privacy.

2017

1. arXiv

   Using ECC DRAM to adaptively increase memory capacity

   Yixin Luo, Saugata Ghose, Tianshi Li, Sriram Govindan, Bikash Sharma, Bryan Kelly, Amirali Boroumand, and Onur Mutlu

   Jun 2017

   Abs arXiv

   Modern DRAM modules are often equipped with hardware error correction capabilities, especially for DRAM deployed in large-scale data centers, as process technology scaling has increased the susceptibility of these devices to errors. To provide fast error detection and correction, error-correcting codes (ECC) are placed on an additional DRAM chip in a DRAM module. This additional chip expands the raw capacity of a DRAM module by 12.5%, but the applications are unable to use any of this extra capacity, as it is used exclusively to provide reliability for all data. In reality, there are a number of applications that do not need such strong reliability for all their data regions (e.g., some user batch jobs executing on a public cloud), and can instead benefit from using additional DRAM capacity to store extra data. Our goal in this work is to provide the additional capacity within an ECC DRAM module to applications when they do not need the high reliability of error correction. In this paper, we propose Capacity- and Reliability-Adaptive Memory (CREAM), a hardware mechanism that adapts error correcting DRAM modules to offer multiple levels of error protection, and provides the capacity saved from using weaker protection to applications. For regions of memory that do not require strong error correction, we either provide no ECC protection or provide error detection using multibit parity. We evaluate several layouts for arranging the data within ECC DRAM in these reduced-protection modes, taking into account the various trade-offs exposed from exploiting the extra chip. Our experiments show that the increased capacity provided by CREAM improves performance by 23.0% for a memory caching workload, and by 37.3% for a commercial web search workload executing production query traces. In addition, CREAM can increase bank-level parallelism within DRAM, offering further performance improvements.

2016

1. SIGMETRICS

   Understanding Latency Variation in Modern DRAM Chips: Experimental Characterization, Analysis, and Optimization.

   Kevin K Chang, Abhijith Kashyap, Hasan Hassan, Saugata Ghose, Kevin Hsieh, Donghyuk Lee, Tianshi Li, Gennady Pekhimenko, and 2 more authors

   In Proceedings of the 2016 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Science Jun 2016

   HTML PDF