Clear, useful ethical standards for software engineers, data scientists, and designers are somewhat lacking.

While many existing guidelines  gesture toward noble goals and seek to ensure work is being done in the public interest with transparency and respect for privacy, there remain significant blind spots. Questions of inclusion, principles for ensuring external consistency, and other issues affecting edge users either largely go ignored or do not offer clear guidelines on how to navigate such problems effectively. 

At the same time, there is a growing body of literature being produced by designers, companies, and other institutions that begin codifying general principles and tips based on market research and industry learnings. Generally, these are shared in small trade publications, blogs, and mini-sites. Overall, there is notable overlap in the key goals they are trying to establish: things like transparency, equity, inclusion, and putting the human user first. However, these have not been institutionalized or formally distilled into a general set of professional principles. This is precisely why a new organization can play an important role in filling this gap.

Overview of existing guidelines

There are a few existing lists of principles and ethical standards for data scientists and software designers that are good points of reference. Chief among these are the ACM/IEEE-CS Software Engineering Code and the ACM Code of Ethics and Professional Conduct. These guidelines are published by professional associations and largely reflect an institutional perspective that looks at designers and data scientists as professionals in their field, rather than as people with tremendous influence over the shape of our world and how different people can engage with it. This professional perspective is a limited one. 

There are other organizations doing similar work that is much closer in focus to the kind of approach we’re exploring in our research. One excellent example is Accenture’s universal principles of data ethics. These principles look at software and systems design from an ethical perspective by highlighting the importance of putting human concerns first, emphasizing the ethics of how the system engages the user at every stage, and ask designers to proactively consider negative downstream implications. Another key focus here is on governance and how it’s vital that these systems be auditable, transparent, and accountable at every stage – since it is likely that they are being built with implicit bias in the first place. 

The other really key set of principles worth mentioning here is Deon, an ethics checklist for data scientists created by Driven Data (a company that organizes crowdsourcing solutions for data science issues). Deon has made the rounds across GitHub, Reddit, and other forums and hubs where software developers gather – underscoring the pent-up demand for usable, relevant and forward-thinking ethical principles. Again, we see emphasis on accountability and transparency at every stage, from data collection to storage, analysis, modeling, and deployment. And we continue to see guidelines around ensuring transparency with users, privacy, configuration options, opt-in and opt-out options, bias in both methodologies and data sets, and other blindspots related to inclusion. 

Inclusive Design Principles is another excellent checklist primarily focused on ensuring that design is inclusive of people with disabilities. They also provide practical examples for each of the points on the list, which makes this a great tool to understand how exactly these principles can be enacted in different fields. Overall, these guidelines also continue to emphasize user choice, consistent experiences across different types of users, and proactive considerations of how edge users might engage with the products. 

Finally, stepping away from the more institutional checklists, there are many bloggers and writers who have put together their own checklists for ethical design and are sharing best practices across small trade outlets, blogs, and forums. These can be articles in places like UX Planet or posts on blogs like Towards Data Science. These lists largely echo the points we find in other guidelines, but are typically more geared to specific applications (like UX design, or focusing on inclusion in software development). 

Key themes in data ethics principles

Across existing guidelines, checklists, and principles for ethical data science, several key themes emerge. Below, we’ve highlighted a few of the themes most relevant to design automation, and those that are on the cutting-edge of developments in the space. ‍

Inclusion

One common theme is the importance of inclusion in software design. In particular, there is a growing dichotomy between accessibility and inclusion: in short, accessibility refers to the outcome of a design that makes it usable, while inclusivity refers more to the process by which these systems are designed in the first place. While the two go hand in hand, the growing importance of inclusivity reflects an approach where designers are increasingly expected (or should be expected) to use methodologies that explicitly and proactively consider edge users in the design process. Julian Kilker, Associate Professor, Emerging Technologies at UNLV, puts it another way: “Design ethics, in short, needs to move beyond making technologies accessible to all people, to making all types of people accessible to designers.”

The primary emphasis on inclusive design remains focused on people with disabilities, but is increasingly incorporating considerations around people with language barriers or low tech literacy. Overall, new organizations must emerge to bring more traditionally overlooked communities into consideration and to act as a leader in developing standards and guidelines by which to design better systems to meet their needs. This can be done through direct consultations with experts or with advocates from affected populations, through user feedback, or even by simply imagining how people with different abilities might engage with a given situation. Theoretically, there could even be a useful checklist that asks designers to consider how different people might approach their product (i.e., “How would a hard of hearing person use this?”; “How would a blind person use this?”; “How would a person with limited English proficiency use this?”; “How would this translate along XYZ metrics for right-left script?”). 

Customization vs. streamlining processes

Another key theme that emerges in the automation and UX space is the importance of balancing customization and choice with streamlining processes and maximizing efficiency. Naturally, systems want to offer as much choice and customization to users as possible. But knowing that complex systems can be overwhelming and have a steep learning curve, it is increasingly important for designers to begin with a clear sense of what their system aims to accomplish, so they can ensure the final product offers a reasonable range of configuration options without sacrificing speed or efficiency. That necessarily varies from project to project, but the principle stands. There is a pressing need to develop guidelines for thinking these types of systems through, particularly from a perspective that aims to promote external consistency with other tools.

Transparency, confidentiality, privacy

As one might expect, transparency, confidentiality, and privacy continue to be major topics in virtually every one of these existing sets of guidelines and principles. Generally, the common point is that systems should be designed to ensure privacy, to respect confidentiality, and to be transparent with what types of data they collect and how they are used (ideally with customization and opt-in/out options for users). This will continue to be one of the biggest issues in ethical data science and design automation for the foreseeable future, but there is definitely more work to be done in elaborating on what is encompassed in our conception of privacy, and how far it extends (e.g., how does data privacy work after someone’s data is collected by one organization, but then handed to another organization to work with in different ways that the user may not have been aware of or consented to?). 

Related to this, we are also increasingly seeing concerns not only with how data are collected (since methodology could have implicit bias), but also thinking ahead about how the data might be used (even down the line) in negative ways. As user data is increasingly commodified (being bought and sold to advertising companies to better target individuals), there is far more opportunity for the data that was originally collected to be used in negative ways downstream.

Two quotes from Accenture really drive this home:

"What is done with datasets is ultimately more consequential to subjects/users than the type of data or the context in which it is collected. Correlative uses of repurposed data in research and industry represents both the greatest promise and the greatest risk posed by data analytics.

‍

All datasets and accompanying analytic tools carry a history of human decision-making. As much as possible, that history should be auditable, including mechanisms for tracking the context of collection, methods of consent, the chain of responsibility, and assessments of quality and accuracy of the data."

Legal considerations

Finally, we see that exhortations to follow the law come up again and again. Naturally, this is a key way to avoid liability. But increasingly, there is a growing recognition that our existing laws are often not enough to go by as guidelines for designing ethical systems. Given that the pace of technological development moves much faster than the pace of regulatory bodies, many of the laws on our books are either outdated, rendered moot, or have gaping loopholes that less-conscientious actors might exploit. 

With that in mind, many guidelines ask that designers consider not just the letter, but also the spirit of the law. Others remind designers that they should always follow the law, but remember that the law is the minimum bar. This is perhaps why we see so many companies and organizations abiding primarily by internal standards, rather than more universal standards tied to regulatory structures or legal policy. 

Conclusion

If the sheer number of these types of blog checklists and lists of principles is any indication, there is a strong appetite for more thorough and universal guidelines from committed experts. To read more about how ExperienceFutures.org is helping advance the ethical use of AI and Machine Learning in digital design to drive equity and empowerment online, see our areas of focus here.