Implementing and Evaluating Your Selected Products

Working with an AI-enabled mental health technology? See AI-Enabled Mental Health Technologies in K–12 Schools: A Practical Guide for School and District Leaders for additional guidance on evaluating, implementing, and monitoring AI-enabled products.

‍

‍

Plan Thoroughly Before Implementation

Revisit key considerations made during needs assessment and product selection to help inform planning and logistics and ensure your district has the people, resources, and processes required to use it effectively. 

Re-engage your team of community members

This should include administrators, teachers, mental health professionals, students, parents and caregivers, and community partners. A diverse community group can help to ensure cultural relevance by aligning the project with the community's values, practices, and needs. Each group brings a unique perspective and expertise, all of which are needed to ensure successful implementation and ongoing support.

Clearly outline the specific responsibilities and contributions of each community group

For example, your administrators might oversee the overall direction and allocate resources, staff can advise on how the product will fit within their existing work and programs. School-based mental health professionals can provide insight on students’ needs, and evaluate the product's impact on well-being. Students and parents and caregivers can provide insight into usability, accessibility, trust, and their experiences with the product. Community partners might support with additional resources or expertise. Establish governance and accountabilityIdentify who will lead implementation and who has authority to make decisions about the product over time. Include the relevant perspectives from student services, school mental health, technology, privacy and security, procurement, legal, communications, and district leadership.

Clearly assign responsibility for:

• overseeing implementation and evaluation;
• managing the relationship with the company;
• determining where professional judgment or human review is required;
• responding to safety, privacy, technical, or implementation concerns;
• reviewing significant changes to the product or its AI-enabled features; and
• deciding whether use should be adjusted, expanded, paused, or stopped.

Use existing district leadership and student support structures wherever possible rather than creating a separate process solely for the product.

Set clear, measurable objectives

These objectives should outline what you aim to achieve and should be specific, measurable, attainable, relevant, and time-bound (SMART). For example, you might aim to increase teacher engagement with the product by 50% within six months or ensure 100% participation in training sessions by the end of the first quarter. Include measures of participation and use, but do not rely on engagement alone. Objectives should also reflect whether the product improves access to appropriate support, strengthens connections to trusted adults and services, and contributes to student well-being.Setting clear objectives will help guide the implementation process, track progress, and evaluate the success of the initiative. Keep in mind that goals can be adjusted as implementation progresses.

Review or establish a district AI policy to guide the implementation of AI-enabled products

If your district has existing AI policies or guidance, review them before implementing an AI-enabled product to ensure they address acceptable use, data privacy, family communication, staff training, human oversight, and student safety, and that they ensure they are evaluated and governed consistently with your broader commitments to student safety and well-being. If your district does not yet have an AI policy, developing one should be part of your pre-implementation planning. 

Learn more: See AI-Enabled Mental Health Technologies in K–12 Schools: A Practical Guide for School and District Leaders for additional guidance and resources on district AI governance, leadership, and ongoing oversight of AI-enabled mental health technologies. 

Allocate resources to facilitate smooth implementation and overall success

Ensure that all necessary resources are available to support the project effectively, including funding, staffing, training, technology, and time. Consider hardware, software, technical support, accessibility needs, staff capacity, and ongoing oversight. You'll also need to set aside enough budget and staff to manage these resources, making sure there are no gaps that could slow down successful implementation.

Review evidence-based practices and case studies of successful implementations

Make sure to research and analyze data from other schools or districts that have successfully used similar digital well-being products. Look for peer-reviewed studies, reports, and evaluations that highlight best practices, challenges, and strategies that led to positive outcomes. By evaluating this evidence, you can identify proven methods, anticipate potential obstacles, and adapt successful strategies to fit your specific context, increasing the chances of a successful implementation.

For products with AI-enabled features, determine whether the evidence applies to those features specifically rather than only to the broader product.

Assess cost feasibility including initial expenses, ongoing maintenance, and hidden costs, to ensure financial sustainability

Consider initial costs as well as licensing, maintenance, training, technical support, staffing, product updates, evaluation, and potential expansion. Assess whether the district can sustain the product without reducing other essential student supports.

For guidance on vendor contracts consideration, including protections specific to digital mental health tools and AI-enabled features, see the Select Relevant Products page.

Outline what you hope to learn from the pilot

This should include feedback you want to gather and what metrics you will use to measure success. Once you have a  timeline for the pilot, with a set duration and key milestones, you can establish a process for collecting and analyzing feedback from participants to make necessary adjustments before a wider rollout.

Develop a pilot plan to test the technology in a select group of schools.

Your plan should include selecting a diverse group of schools or classrooms that represent different demographics and needs. Establish a timeline, milestones, feedback process, and criteria for determining what happens after the pilot.

‍

Train Staff and Engage Community Members

Successful implementation of any new initiative relies heavily on effectively training staff, providing them with ongoing support, and engaging community members. In making sure everyone involved has the information they need to implement and understand the technology, your district not only fosters buy-in to use the product, but will also encourage open dialogue and feedback.

Provide role-specific training

Effective training is essential for the successful use of new initiatives. These sessions should be role-specific, and cover the project's goals, the purpose of the product, how it works, appropriate and best practices for use, relevant district procedures, and what to do when problems arise. 

For products with AI-enabled features, training should also address:

• what the AI does and does not do;
• what information may shape its responses, recommendations, alerts, summaries, or other outputs;
• known limitations and risks;
• how product-generated information should be interpreted;
• when professional judgment or direct knowledge of a student must take priority;
• how the product connects with referral and crisis response procedures; and
• how to respond if a student reports an inaccurate, harmful, biased, confusing, or distressing interaction.

Training should include hands-on practice, time for questions, and ongoing support and follow-up sessions to address any issues and reinforce learning.

Establish systems for ongoing support and troubleshooting during the initial roll-out

This includes setting up a support team that can quickly address any issues or concerns that arise and providing multiple channels for support, such as email, phone, and a comprehensive FAQ and troubleshooting guide to help users solve common problems independently. Consider scheduling regular check-ins and feedback sessions with staff to identify any issues and make necessary adjustments.

Develop an effective communication plan to inform and involve students, parents and caregivers, and the broader school community

Your plan should include regular updates through newsletters, emails, and school meetings to share progress and important information. When the product includes automated, adaptive, or AI-enabled features, explain in plain language:

• when and how those features are used;
• what information they collect or generate;
• how student information is protected;
• where human review or support is involved;
• what students and families should expect; and
• how they can ask questions, raise concerns, or exercise available choices about participation.

Update communications when the product, evidence, or district practices change.

Use multiple platforms such as social media, the school website, and community events to reach people where they are, and make sure that communication is clear and consistent.

Set up mechanisms for continuous feedback from all community

It’s important to encourage feedback to not only foster a sense of involvement and support from the entire community, but to also get buy-in for use. Provide accessible ways for students, families, educators, school mental health professionals, and other affected groups to share their experiences throughout implementation. Be clear about how feedback will be reviewed, who is responsible for acting on it, and how the district will communicate what it learns.

‍

‍Implement and Evaluate the Pilot

Implementation requires a clear and structured approach from the beginning. It’s helpful to pilot before rolling out to the entire district. By choosing a few schools that are ready to pilot a new initiative, you can follow the strategic plan closely, monitor each step, and make adjustments as needed. Collecting smaller sets of initial data on usage, engagement, and outcomes allows your district to assess early impact. It also means you can actively seek feedback from students, parents and caregivers, teachers, and other community members that can be used to make improvements to help ensure a wider roll-out will effectively meet your goals.

Follow your plan for piloting closely to ensure fidelity to the program design, monitoring each step along the way

Track whether the product is being used as planned, whether staff understand their responsibilities, and whether the necessary technical and student support processes are working.

Also monitor significant product or feature changes that may affect performance, student experience, staff responsibilities, privacy, safety, or access to human support.

Gather data on program usage, engagement, and initial outcomes to evaluate adoption and early impact

This data should include quantitative data like participation rates and qualitative data to provide a comprehensive understanding of the program's effectiveness, including:

• who is and is not using the product;
• how students and staff experience it;
• whether it is functioning as expected;
• whether it is helping achieve the district’s stated objectives;
• how it affects access to support and trusted adults; and
• whether unexpected problems or barriers are emerging.

For AI-enabled products, pay particular attention to inaccurate or confusing information, inappropriate recommendations, overreliance on the technology, reduced help-seeking, unclear human oversight, or other effects on students and school mental health services and supports.

Actively seek feedback from students, parents and caregivers, teachers, and other community groups involved in the pilot

Use surveys, focus groups, and informal discussions to review progress, address challenges, and foster the collaboration needed to identify and understand any emerging issues.

Analyze and use the collected feedback to identify areas for improvement

This allows you to make informed adjustments to enhance effectiveness. Your analysis will guide targeted changes, ensuring the program better meets the needs of all students and achieves its intended goals. Create ways for concerns to be raised promptly rather than waiting until the pilot ends.

Adjust the implementation plan as needed

Use the information gathered to address problems, improve training or communication, clarify responsibilities, and change how the product is used. If concerns cannot be adequately addressed, be prepared to limit, pause, or stop implementation.

‍

Evaluate Outcomes for Continuous Improvement

To ensure ongoing success and improvement, it's crucial to evaluate use, user satisfaction, and outcomes regularly. Collecting continuous data on key performance indicators related to student mental health, program usage, and overall well-being through both quantitative and qualitative methods will allow the district or school to gather comprehensive insights.

Those data help identify trends, patterns, and gaps, which can be shared and talked about with community members. Together, your team can make iterative improvements, document changes for future reference, and plan for long-term sustainability and potential expansion.

Regularly collect data on key performance indicators related to student mental health, program usage, and overall well-being

Data on key performance indicators should be gathered consistently to track progress and identify any areas that need attention or improvement over time.

Use mixed methods to collect continuous data

Employ both quantitative methods, such as surveys and attendance records, and qualitative methods, like focus groups and interviews, reviewing any automated indicators or product-generated insights alongside feedback and observations from students, families, and staff. This provides a more complete and accurate picture of impact.

Analyze for Disparities

If your school can disaggregate data to identify inequities and disparities, it will help determine the most pressing areas of opportunity, the effectiveness of current services, patterns of emotional, behavioral, medical, basic, and social support needs, and how implementation practices, training, product design, AI-enabled features, or product-generated recommendations and workflows may be contributing to positive, negative or uneven outcomes across groups.

Look for trends, gaps, and unintended consequences

Assess what is working, what needs improvement, and where the product may not be meeting its objectives. Review intended benefits alongside potential negative or unexpected effects, including impacts on help-seeking, trusted relationships, staff responsibilities, access to support, and student well-being.

Hold regular review meetings

Bring together the people responsible for governance, implementation, and evaluation to review:

• progress toward objectives;
• student, family, and staff feedback;
• safety, privacy, technical, or implementation concerns;
• differences across student groups;
• emerging research or evidence; and
• significant product or AI model changes.

Use these reviews to make adjustments, including whether implementation should continue as planned or be adjusted, expanded, limited, paused, or stopped. Regular updates ensure it remains effective and responsive to students' needs. 

Share results and communicate findings

Communicate relevant findings and decisions to students, families, staff, district leaders, and other community members. Explain what the district learned, what changes it is making, and how community feedback informed those decisions.

Make iterative improvements based on data analysis and feedback

Make adjustments to how the district or school is using the technology, as needed. Regular updates ensure it remains effective and responsive to students' needs.

Keep detailed records of changes made and their impacts to inform future decision-making

Maintain records of significant implementation decisions, concerns, product changes, district responses, and their effects. This documentation can support accountability, future evaluation, staff transitions, and decisions about continued use.

Plan for sustainability

Determine whether the district can continue providing the funding, staffing, training, technical support, governance, and evaluation the product requires. Periodically reassess whether the product, including any AI-enabled features, still aligns with district goals and continues to support student well-being, privacy, safety, fairness, and access to trusted human support.

‍

Looking for more guidance?

AI-Enabled Mental Health Technologies in K–12 Schools: A Practical Guide for School and District Leaders includes additional recommendations for implementing, monitoring, and evaluating AI-enabled mental health technologies over time, including governance, training, communication, and continuous improvement.