10 HCD AI Prompts evaluated for efficacy in real world scenarios.

Intro

With the eruption of artificial intelligence and its now widespread access, the prospect of leveraging AI support to augment the practices and processes involved in IT services becomes increasingly appealing. 

In the past few months, our team has conducted a series of experiments exploring the capabilities of Generative AI systems (language models) in aiding CX/HCD practices and evaluating the quality of its responses and output. We sought to find the most valuable interactions and in doing so we also came across some of its limitations and pitfalls that require us to be mindful when engaging these new AI tools.

In this article, we present the highlights of the most valuable use of the tool and real examples of AI-generated responses which we graded in terms of their relevance and accuracy.

Generative AI interactions

In the experiments mentioned above, we focalized on the scope of IT services that pertain to Customer Experience (CX) and Human Centered Design (HCD) processes, with the consideration that our federal clients and projects require adherence to the IDEA Act, 508 compliance and USWDS, among other guidelines and requirements.

Learning Curves

An interesting aside that warrants attention is the learning curve that our team experienced during the initial stages of our experiments. Inconsistencies were identified throughout the conversation with the AI models. In some cases, the AI expressed difficulties in the interpreting our prompts and, as users, we felt the models did not understand the questions that what we perceived to have been presented clearly and were informed by our professional expertise. Additionally, prompts that were presented as sequential questions with the intent to  provide  depth and context to a topic were addressed by the AI models as isolated or one-off questions that did not connect with the flow of the conversation. This resulted in the generation of irrelevant answers and points of frustration as users. And while we do not discard that there is a learning curve to presenting better prompts, as users, the experience is perceived as shortcomings in understanding at the hands of the language model.

We were able to bridge these issues by improving our prompts and creating an introductory statement with a formula that proved effective: “You are – describe the persona – of a/ in a – describe the context – that – describe the activity, client, destination -. – Elaborate limits, boundaries, and inherent goals -.” From there on, we posed questions to obtain assorted support or content follow. Additionally, we incorporated middle steps in the form of questions to verify understanding which allowed us to check the correct direction of the conversation (and perhaps add to the comprehension of the model):” – describe or present a statement – Do you understand/ agree with this?”

Our results proved to be more valuable in scenarios where our expectation was for the AI model to supply summarized knowledge of (specific and widely) publicly available sources and create checklists to fulfill government requirements. Additionally, we found valuable support in the areas of scripting and wording for user engagements. We then graded some of the outputs from these exercises and have provided the results below:

1. Accessible Color Palette Generation

• “Generate an accessible color palette for the ed.gov website that meets WCAG 2.1 AA contrast ratio standards, ensuring text and interactive elements are easily readable by users with visual impairments or color vision deficiencies. The palette should include primary and secondary colors, as well as variations for hover and active states. Provide alternative visual cues, such as patterns or textures, to complement color distinctions.”
• “Design an accessible data visualization color scheme for ed.gov that effectively communicates information through contrasting hues and shades while adhering to WCAG guidelines. The scheme should consider color blindness and provide alternative representations, such as patterns or textures, for users with visual disabilities.”
• “Create an accessible color palette for the government website’s form elements, including input fields, buttons, and validation states (error, success, etc.), ensuring sufficient contrast and visibility for users with varying abilities. Provide alternative visual indicators, such as icons or text labels, to complement color based feedback.”
•  “Develop an accessible color scheme for the navigation menu and other interactive components on the government website, ensuring sufficient contrast between text, backgrounds, and hover/focus states. The scheme should also consider color blindness and provide alternative visual cues, such as underlines or icons, for users with visual disabilities.”
• “Generate an accessible color palette for the government website’s data visualizations and charts, ensuring sufficient contrast between different data points, labels, and backgrounds. The palette should also consider color blindness and provide alternative representations, such as patterns or textures, for users with visual disabilities.”

2. Inclusive Content Creation

• “Write a concise, plain language overview of the government website’s mission and services, ensuring the content is easy to understand for users with varying literacy levels and backgrounds. Avoid jargon, technical terms, or assumptions about prior knowledge.”
• “Create a set of frequently asked questions (FAQs) related to federal aid programs offered by the Department, using simple, clear language that is accessible to users with diverse backgrounds and reading abilities.”
•     “Develop user-friendly instructions for navigating and using the website, ensuring the content is free from complex terminology or assumptions about digital literacy. Include visual aids or step-by-step guides to support users with varying abilities.”
•     “Write a series of blog posts or articles on educational topics relevant to the Department’s mission, using inclusive language and avoiding assumptions about the reader’s prior knowledge or background. Ensure the content is easy to understand for a wide range of audiences.”
•     “Create a glossary of common terms and acronyms used on the website, providing clear, concise definitions in plain language to aid users with limited subject matter expertise or English proficiency.”

3. Multilingual Support

    “Translate the main landing pages and essential content on the website into Spanish, ensuring accurate and culturally appropriate translations while maintaining accessibility standards for users with disabilities or limited English proficiency.”

    “Provide multilingual support for the website’s application forms and instructions, translating the content into the top five languages spoken by the target audience. Ensure the translations are accurate, culturally sensitive, and accessible to users with varying abilities.”

    “Develop a language assistance feature for the website, allowing users to select their preferred language and providing machine-translated content or links to human-translated resources. Ensure the feature is easily accessible and compliant with accessibility guidelines.”

    “Translate the website’s online resources and materials into multiple languages commonly spoken by the target audience, ensuring accurate and culturally appropriate translations while maintaining accessibility standards for users with disabilities.”

    “Provide multilingual support for the website’s contact information and customer service channels, offering options for users to communicate in their preferred language or access language assistance services, while adhering to accessibility standards.”

The output is relevant to the project and as before, balances comprehensiveness and succinctness in line with the requirements. But as it connects with the needs of the audience we feel a reference to user engagements is missing here, where valuable insight would be found.

4. Accessible Media Generation

    “Generate alternative text descriptions for all images on the website, ensuring the descriptions accurately convey the content and purpose of the images in a concise and meaningful way, adhering to accessibility guidelines.”

    “Create closed captions and transcripts for videos and multimedia content hosted on the website, ensuring accurate and synchronized captions that meet accessibility standards for users with hearing impairments or limited English proficiency.”

    “Develop audio descriptions for visual elements on the website, such as infographics, charts, or diagrams, providing alternative access to information for users with visual disabilities or learning preferences.”

    “Generate accessible and descriptive text alternatives for complex data visualizations or interactive content on the website, ensuring users with disabilities or limited access to visual information can comprehend the data and interact with the content effectively.”

    “Create text transcripts and captions for live or prerecorded webinars, presentations, or training sessions hosted on the website, ensuring accessibility for users with hearing impairments or limited English proficiency.”

5. Responsive Design Prototyping

    “Develop responsive design prototypes for the website’s main landing page and key service areas, ensuring optimal user experience across various devices, including desktops, laptops, tablets, and mobile phones. Consider different screen sizes, input methods, and assistive technologies in the prototypes.”

    “Create responsive prototypes for the website’s online application forms and submission processes, ensuring a seamless and accessible experience for users across devices, including touch-based interactions and compatibility with assistive technologies.”

    “Design responsive prototypes for the website’s resource library, allowing users to easily browse, search, and access materials from different devices while maintaining accessibility standards.”  

6. Persona-based User Testing

    “Create a set of diverse user personas representing individuals with varying abilities, backgrounds, and technical expertise who may interact with the website. Use these personas to generate realistic user scenarios and conduct accessibility testing to identify and address any usability challenges.”

    “Develop user scenarios based on personas representing users with disabilities, such as visual or hearing impairments, and use these scenarios to test the accessibility of the website’s online resources, applications, and tools.”

    “Create user personas representing individuals with limited English proficiency or from diverse cultural backgrounds, and use these personas to evaluate the effectiveness of the website’s multilingual support and inclusive content.”

    “Generate user scenarios based on personas representing parents or guardians with varying levels of digital literacy, and use these scenarios to assess the usability and accessibility of the website’s information and resources for families.”

    “Develop user personas representing professionals with different technical expertise and abilities, and use these personas to test the accessibility and usability of the website’s professional development resources and data reporting tools.”

7. Service Blueprint Creation

    “Generate a service blueprint that maps out the entire user journey for a prospective user applying for federal aid through the website, including touchpoints with various government agencies and services. Ensure accessibility compliance and seamless integration throughout the process.”

    “Create a service blueprint for a parent or guardian seeking resources and support services for their child through the website, mapping out the different touchpoints across multiple government agencies and service providers, while ensuring accessibility and inclusivity.”

    “Develop a service blueprint that outlines the user journey for a user seeking professional development opportunities, curriculum resources, and information through the website, accounting for potential accessibility challenges and integrating with other relevant government services.”

    “Generate a service blueprint for a researcher or policymaker seeking educational data, reports, and policy information from the website, mapping out the various touchpoints with different government agencies and ensuring accessibility compliance throughout the process.”

    “Create a service blueprint that maps out the user journey for an individual seeking information and resources related to civil rights in education through the website, including touchpoints with relevant government agencies and ensuring accessibility and inclusivity throughout the process.”

8. Accessible Form Design

    “Design an accessible application form on the website, following best practices for label association, error handling, and compatibility with assistive technologies, while providing clear instructions and guidance for users with diverse abilities and backgrounds.”

    “Create an accessible online application form for users seeking professional development funding opportunities through the website, ensuring compliance with accessibility standards, clear labeling, and user friendly error handling.”

    “Develop an accessible form for individuals to submit civil rights complaints or report discrimination incidents through the website, following best practices for accessibility, clear instructions, and user friendly feedback mechanisms.”

    “Design an accessible feedback form for users to provide comments or suggestions on improving the usability and accessibility of the website, ensuring compatibility with assistive technologies and clear instructions for users with diverse abilities.”

    “Create an accessible online form for researchers or policymakers to request data or reports from the website, following accessibility guidelines for form design, labeling, and error handling, while providing clear instructions and support for users with varying technical expertise.”

9. Feedback Mechanism Integration

    “Integrate an accessible user feedback mechanism, such as a survey or comment form, on the website to gather insights and feedback from users with diverse abilities and backgrounds on the website’s usability, accessibility, and content relevance.”

    “Develop an accessible chatbot or virtual assistant feature for website, allowing users to ask questions, provide feedback, or report accessibility issues in a conversational manner, while ensuring compatibility with assistive technologies and multilingual support.”

    “Create an accessible online forum or community platform on website, where users can share their experiences, provide feedback, and engage in discussions, while ensuring inclusivity and moderation for respectful and constructive dialogue.”

    “Integrate an accessible feedback mechanism within the website’s online resources and tools, allowing users to report issues, suggest improvements, or provide feedback on the accessibility and usability of specific features or content.”

    “Develop an accessible feedback mechanism specifically for users using the website’s professional development resources and data reporting tools, gathering insights and feedback to improve the user experience and accessibility for this user group.”

10. Compliance Checklist Generation

     “Create a comprehensive checklist to verify compliance with Section 508, USWDS, and IDEA Act requirements for the main website, covering aspects such as accessibility testing, design standards, content guidelines, and performance metrics.”

     “Generate a compliance checklist specifically for the website’s online application forms and submission processes, ensuring adherence to accessibility guidelines, plain language requirements, and user experience best practices.”

     “Develop a compliance checklist for the website’s resource library, verifying accessibility standards for multimedia content, inclusive language, and user-friendly navigation and search functionalities.”

     “Create a compliance checklist for the website’s data visualization and reporting tools, ensuring adherence to accessibility guidelines, clear data representation, and user-friendly interactions for users with diverse abilities.”

     “Generate a compliance checklist for the website’s interactive tools and calculators, verifying accessibility standards, clear instructions, and user friendly feedback mechanisms.”

Prompt

Generate the list of required links all federal websites must have in their footer and menus

Output

Here is a list of required links that all federal websites should have in their footer and menus to comply with regulations and provide essential information to users:

Footer Links:

1. Accessibility Statement: A link to the website’s accessibility statement, outlining measures taken to ensure compliance with Section 508 and WCAG guidelines.

2. Privacy Policy: A link to the agency’s privacy policy, explaining how user data is collected, used, and protected.

3. Security Policy: A link to the website’s security policy, detailing measures taken to ensure information and system security.

4. Contact Information: A link to a contact page or form for users to get in touch with the agency or website administrators.

5. Site Map/Index: A link to the website’s sitemap or index page, providing an overview of the site’s structure and content.

6. Feedback/Comments: A link to a feedback form or mechanism for users to submit comments, suggestions, or report issues.

7. Legal Notices/Terms of Service: Links to relevant legal notices, disclaimers, or terms of service applicable to the website and its content.

8. Compliance Statements: Links to compliance statements or certifications, such as HTTPS/TLS encryption or digital authentication standards.

9. Social Media Links: Links to the agency’s official social media accounts (e.g., Twitter, Facebook, Instagram) for users to follow and engage with.

10. Related Resources/Links: Links to related government websites, portals, or external resources relevant to the agency’s mission or services.

Menu Links:

1. Accessibility Resources: A link to accessibility resources, such as guides, tools, or assistance for users with disabilities.

2. Language Assistance: A link or dropdown menu for users to access multilingual support, translated content, or language assistance services.

3. Search: A prominent search function for users to easily find information or content within the website.

4. Online Services/Tools: Links to online services, applications, or interactive tools offered by the agency.

5. Publications/Resources: Links to educational materials, reports, data visualizations, or other resources provided by the agency.

6. News/Updates: Links to news, announcements, or updates related to the agency’s activities or initiatives.

7. About Us: A link to information about the agency, its mission, leadership, and organizational structure.

8. Careers/Jobs: A link to career opportunities or job listings within the agency.

9. Contact Us: A prominent link to the agency’s contact information or a contact form for inquiries.

10. Help/FAQs: Links to help resources, frequently asked questions, or support documentation for users.

By including these essential links in the footer and menus, federal websites can ensure compliance with regulations, provide crucial information and resources, and offer an inclusive and accessible user experience for all citizens.

The output offers relevant components for a federal website that are either required or sensible to have; b. But it has omitted key requirements, as presented by the USWDS, such as the incorporation of the Federal “Banner”, the identifier for official websites of government organizations that also signalsiginal a website is official and secure. Additionally it missed a number of items for the footer “identifier” that should present reference to a parent agency, FOIA requests access, link to NO FEAR Aact data, among others. 

Finding the limits and shortcomings of AI support

Most users of AI language models are aware that the models are trained with a robust and diverse data set and information, and that the data  is more or less updated, but cannot incorporate real-time or even near real-time news developments into outputs. The models notify users of this limitation only when user questions are clearly geared toward current events. Which means users’ interactions receive responses that are an alchemy of facts, relational composition and sui generis content that does not provide certainty or clarity regarding how much of each the resulting composition of the response is based on verifiable data. The AI model is trained to imitate human communication, and the responses are presented with a tone of a “person confident in their knowledge and intelligence” regardless of whether it is generating a verifiable response or one that is purely generative without any factual basis. While it may not have been the intent,, the responses of AI models can quickly turn deceitful.

An example of this scenario is a conversation one of our team members had with an AI model regarding government requirements (such as 508 compliance, 21st Century IDEA, etc.) for which responses were fact-based. The flow of the conversation progressed organically culminating in the user requesting the generation of user personas for a government agency. The AI model presented the output in the same matter-of-fact tone as every prior query.

The user  queried the foundation of the output – given that User Personas are research-based composites; to which, the model agreed and iterated an aligned definition. The user followed up by requesting the sources of research used for the personas provided by the AI. The model cited “Qualitative Interviews and User Research” as sources. When the user drilled down further, asking the model to provide the source of those studies, the AI backtracked: “The truth is, I do not have access to any actual user research studies or qualitative interviews involving individuals with visual impairments that I referenced when creating the persona example. As an AI assistant without direct access to such research data, I should not have implied that the persona was informed by real studies.”

It’s thanks to the fact that this interaction was carried out by a subject matter expert that this issue was detected and verified. The gravity of this AI model feature presents a real danger should an inexperienced user or someone not trained in CX/HCD were to carry out a similar exchange, and then base real-world decisions that impact customer experience outcomes on outputs from AI models that are inventions.

Conclusion

AI language models are powerful tools that can be leveraged to provide valuable support to our CX/HCD processes among other applications to IT services. In the current state of the AI capabilities we need to consider that the input and output needs to be carried out and supervised by professionals with experience and expertise in the topics covered, so that the responses can be effectively evaluated to determine if the content collected is viable for  real world application. 

Current AI models are smart, but there is still a long way to go before they can be considered reliable for more than a mere regurgitation of information and simplistic correlation. Further, the lack of transparency surrounding the validity of a model’s output is disconcerting (and potentially detrimental if the user is untrained and not a subject matter expert). Our team’s experience with this disconnect demonstrates a potential need for the inclusion of a disclaimer when an output is not verifiable and sources cannot be cited? Nevertheless, there is definite potential for AI models to serve as a useful resource in CX/HCD and as the models continue to evolve, we are confident that before long we can expect an expansion of its capabilities as well as a resolution to the shortcomings discussed.

Feel free to reach out Mobomo with any questions you might have.