#1 IN THE SERIES
Welcome to the “Signals of Quality” series — a look at what separates genuinely effective AI-powered ed tech from flashy technology, and how state leaders and advocates can build procurement practices that put student learning first. Given the growing skepticism of artificial intelligence (AI) and ed tech, how should K-12 education leaders think about what “good” tools look like? Safety-oriented frameworks for AI tools abound, but little exists on assessing efficacy and quality for AI tools used by students and teachers in classrooms. Building on early research, this series 1) surfaces five signals of quality in AI-powered tools, and 2) identifies concrete steps state leaders and advocates can take to prioritize efficacy and learning throughout a procurement ecosystem. These insights arise from the AI Policy Hub, a partnership between Bellwether and PIE Network to connect advocates with resources, support, and national education experts.
Not all ed tech is the same, but the “techlash” sweeping K-12 education isn’t making any distinctions. The confluence of debates about phones in schools, social media, screen time in classrooms, and the rise of AI has created a perfect storm that blurs the lines between good AI-powered ed tech tools and bad ones. Yet research shows that a well-designed educational tool is a completely different experience from the addictive scrolling of social media. To ensure that students are getting more of the former than the latter, state and district leaders must be able to differentiate high-quality tools from low-quality ones, then translate those differences into procurement processes.
The field has made significant progress on helping leaders do this for safety. For example, Digital Promise’s* Responsibly Designed AI Product Certification, Common Sense Media’s AI Risk Assessments, and EdSafe AI Alliance’s SAFE Benchmarks all evaluate whether AI products meet baseline expectations around child safety, data privacy, responsible design, and bias mitigation. State regulation on AI tool safety is also emerging: A recent California executive order directs state agencies to develop AI vendor certification standards covering bias mitigation, content safety, and civil rights protections.
However, there hasn’t been similar momentum on quality-oriented frameworks for AI tools. Only two procurement frameworks have emerged, and both represent early thinking. The State Educational Technology Directors Association (SETDA)’s ed tech procurement guidance is comprehensive but not AI-specific. Meanwhile, Opportunity Labs and F3Law’s report, Procurement Benchmarks for AI in K-12 Education, directly addresses AI tool procurement but provides a starting point rather than immediately applicable recommendations.
This asymmetry has practical consequences: Leaders can increasingly verify whether an AI tool meets safety principles but have far fewer resources for evaluating efficacy, pedagogical quality, and vendor sustainability. This piece elevates five signals of high-quality AI tools:
- Emphasis on learning outcomes, not technology features.
- Productive struggle as a primary pathway for learning.
- Sound pedagogy and coherence with existing instructional practice.
- Technical configurations designed to maximize quality.
- Attention to market sustainability and long-term planning.
In the rest of this series, we suggest concrete steps state leaders and advocates can take to leverage these signals in procurement and identify the tools or developers focused on students’ learning rather than the latest trend.
Signal 1: Emphasis on learning outcomes, not technology features.
Traditional efficacy research on AI tools is often unavailable, can take years to produce, and is challenging given the pace of AI evolution, especially for smaller vendors. As a result, engagement metrics (e.g., logins, time on task, Net Promoter Scores) are often shared as proof of quality even though they do not indicate whether students are learning. But regardless of their capacity to conduct rigorous evaluations, high-quality vendors should be able to articulate a clear theory of change that includes the instructional problem being solved, the long-term outcomes expected, and the leading indicators being tracked and reported in the meantime.
One familiar way to identify whether developers are focused on outcomes is to use a logic model. Procurement officers considering student-facing tools might outline a theory of action with named outcomes such as higher math scores or increased collaboration skills. Logic models are also useful for identifying expected short-, medium-, and long-term outcomes. Teacher-facing tools, for example, might target increased familiarity with formative assessments in the short term and higher standardized test scores over time (Figure 1). Any of these can be more meaningful efficacy measurements than usage, but only if named clearly and matched to appropriate metrics.
FIGURE 1: SAMPLE LOGIC MODEL FOR AN AI-POWERED LESSON PLAN GENERATOR
Source: Bellwether, “Measuring Artificial Intelligence in Education,” 2025.
Signal 2: Productive struggle as a primary pathway for learning.
Quality AI tools support learning by scaffolding student thinking, providing meaningful feedback, and preserving productive struggle (the effortful engagement with difficult problems that elicits deeper learning). Yet, these dimensions are difficult to surface from product demos alone, and existing frameworks typically focus on teacher workflow and system integration rather than instructional practice.
For example, Quill, an AI platform that supports writing instruction, guides students on how to improve their work through specific, personalized feedback, but still expects students to do the thinking needed and decide what changes will enhance their writing (Figure 2). EdLight, another AI platform, uses a similar approach for math: Students must tackle problems on paper first, by hand. Only then does the AI scan their work, identify misconceptions, and ask them scaffolded questions to assist their understanding.
Technology inherently makes certain workflows easier, but that doesn’t always have to mean bypassing student effort. These examples demonstrate that when thoughtfully designed, AI tools can guard against cognitive offloading and actually increase productive struggle by calibrating the level of difficulty to meet students where they are.
FIGURE 2: SCREENSHOT FROM A SAMPLE QUILL EXERCISE SHOWING FEEDBACK FOR A STUDENT
Source: Quill, “The Power of Feedback.”
Signal 3: Sound pedagogy and coherence with existing instructional practice.
Beyond protecting productive struggle, a tool’s instructional approach and coherence are key indicators of its quality. This could manifest through several design strategies, such as using existing best practices (e.g., timely, personalized feedback), aligning with high-quality instructional materials (HQIMs), or augmenting rather than automating teacher judgment. But across these examples is a critical theme: high-quality tools are built on strong instructional practices with robust evidence bases, and generative AI is just the enabling underlying technology.
District and school leaders should also consider what this means for their specific context. Beyond a tool’s internal design, “quality” can depend on fit, or whether its pedagogical approach aligns with an instructional model. For example, introducing a tool built for traditional instructional models into a school using an inquiry-based approach can hurt a student’s learning experience. Alternatively, selecting a tool with a contrasting approach could add value — if deployed thoughtfully.
Signal 4: Technical configurations designed to maximize quality.
Technical decisions are often not visible in product demos and can feel inaccessible to those without deep technical expertise. Yet the choices vendors make about model architecture, response refinement, and evaluation infrastructure reveal much about their pedagogical commitment. For example, model architecture choices can demonstrate developers’ priorities: Some vendors rely on a “single-shot” approach with one model; others use multiple models in a “pipeline” structure, allowing individual models to specialize in different tasks (e.g., audio, visual input, text generation) and providing redundancy against downtime. Developers who invest in quality also typically maintain rubric-based benchmarks (often developed with subject-matter experts or validated by third parties) against which they regularly evaluate AI-generated responses.
Regardless of their exact methods, vendors should be able to clearly articulate, without technical jargon, how their decisions enhance the quality of their product. Developers committed to learning can describe their tool’s design and trade-offs in detail, including where and why they set boundaries on AI use (e.g., refusing to use AI in high-stakes decisions such as teacher evaluations, or avoiding student-facing chatbot interactions where current capabilities fall short). Vendors that cannot articulate these choices in plain terms likely have not invested in the underlying quality infrastructure.
Signal 5: Attention to market sustainability and long-term planning.
Given the relative immaturity of many AI ed tech vendors and the pace of technological change, procurement must include due diligence on vendors’ business models. Cases of instability — such as the collapse of AI startup AllHere, which left Los Angeles Unified School District without services and uncertain whether student data had been exposed — illustrate that “quality” should also weigh business model sustainability, governance, and contingency planning.
This shouldn’t disqualify startups; they may be the most innovative or the most closely aligned with certain priorities. But asking vendors about sustainability early, before a contract is signed, can help leaders assess any risks alongside the intended scale of adoption (e.g., pilot versus statewide) and any pedagogical or technical strengths.
The flood of AI-powered ed tech entering the market has left K-12 district leaders and educators overwhelmed and unsure of how to distinguish high-quality tools. Their confusion converges with many parents’ and educators’ concerns about ed tech, which has led at least eight states to pass laws limiting screen time. But many of these policies treat all screens alike, so a learning platform that builds a specific math skill gets lumped in with doomscrolling. As a result, students who might otherwise benefit from technology that makes learning more accessible could suffer from blunt policy instruments that pull back across the board. That conflation is precisely why the ability to distinguish between high- and low-quality AI tools matters.
The five signals elevated here are not a comprehensive framework per se, but they offer a detailed starting point for the education leaders who ultimately decide when, where, and how students interact with AI tools in the classroom. In the next two posts, we’ll dive deeper into how state leaders and advocates can combine these signals with concrete actions to build an ecosystem where students use ed tech that genuinely works for them.
*Editor’s note: Digital Promise is a former Bellwether client. For a full list of our partners, visit our website.
