NIST 的 AI for Manufacturing 研究计划：面向人机协作的度量与标准

AI for Manufacturing is a research initiative focused on measurement science and standards to advance AI adoption in manufacturing. The project integrates four research threads into a cohesive agenda: (1) engaging industry to identify real-world use cases and barriers, (2) developing foundational metrics for human-AI teaming, (3) building methodologies to certify operator understanding of AI solutions, and (4) validating AI-assisted system approaches through interoperability benchmark studies. Working in partnership with national manufacturing institutes such as CESMII, MxD, America Makes, NIMBL, and the planned AI for Resilient Manufacturing institute, the project targets Human-AI teaming benchmarking and evaluation methodologies and contributes to standards through ISO, IEC or other professional societies. The outcomes will provide manufacturers, technology providers, and policymakers with evidence-based tools, metrics, and guidelines to ensure that AI adoption in manufacturing is reliable, resilient, and interoperable, reinforcing U.S. leadership in digital manufacturing innovation. 

Objective

To develop the methods, metrics, models, tools, and consensus-based standards necessary to empower the next-generation manufacturing systems with the trustworthy and interoperable advanced data infrastructure and AI technologies. More specifically, the project focuses on developing methodology, datasets, tools and benchmark studies that enable reliable human-AI teaming for manufacturing system design, operation and automation.

Technical Idea

This project creates guidance for selecting AI methods that are fit-for-purpose in manufacturing applications. To accomplish this, our project develops a holistic, standards-aligned framework, with foundational models, taxonomy and metrics. Through several deep dive case studies, we will demonstrate the effectiveness of our framework for human-AI teaming in manufacturing, with a focus on robust evaluation, and future-ready interoperability.

Both top-down and bottom-up strategies will be applied to address the core research challenge. For the top-down approach, a Human-AI Teaming Workshop will convene experts from manufacturing, automation, digital systems, and research institutes. This event is designed to gather expert-driven use cases, clarify challenges, and spark future collaborations toward standards development.

Field use cases will then be systematically collected and analyzed, mapped onto established smart manufacturing and AI reference models. This mapping will underpin the development of robust metrics and taxonomies for human-AI teaming effectiveness. These metrics will be validated through in-depth case studies, including AI-assisted production scheduling, manufacturing system integration, and preventive maintenance.

In parallel, the bottom-up approach will tackle immediate, practical questions in applying AI to manufacturing. This line of inquiry will support the standards community’s efforts to investigate how human-AI teaming can improve and streamline manufacturing systems engineering through standards.

Research Plan

The research plan weaves together the four interconnected themes.

Stakeholder Engagement & Use Case Mobilization

With the TPO funding, we will organize a focused workshop and working sessions with stakeholders from manufacturing, automation, digital systems, and relevant manufacturing institutes (e.g., CESMII, MxD, America Makes, NIMBL, SMART, AI for Resilient Manufacturing) to collect, document, and classify real-world use cases that highlight gaps, requirements, and priorities for both human-AI teaming and manufacturing system engineering and identify technical, organizational, and standards-related barriers to the adoption of AI across manufacturing workflows.

Metrics Development & Taxonomy Mapping

We will curate the documented use cases into an open reference library mapped to established smart manufacturing reference models and AI technology taxonomies. Then we will analyze the use cases to identify critical gaps in operational and error models for both human-AI interactions and integration processes. Based on the findings, we will develop and refine metrics to evaluate:

• Human-AI collaboration effectiveness (HAC metrics)
• AI fitness-for-purpose for different manufacturing tasks
• Teaming, interpretability, traceability, and trust in GenAI applications
• System integration effort, performance, and semantic correctness

And build tools to support capture and use of these metrics in both qualitative and quantitative evaluations.

Methodology Development and Pilot Studies

This task involves the development of formal schema-driven interviewing methodologies for requirement elicitation, using the metrics defined for both AI and human understanding. We will pilot the measurement methodologies in simulated (GenAI surrogate) and real-word manufacturing scenarios—starting with target applications such as production scheduling or process control.

We will also develop and validate algorithms for automated artifact (V&V diagrams, functional flows, audit trails) generation linked to requirements and operator inputs and instrument pilots to collect detailed metrics on completeness, clarity, and traceability of AI-generated manufacturing solutions.

The influence of domain-specific languages (DSLs) and metrics dashboards on the success of AI-human teaming in these pilots will be evaluated. Working with our collaborators, we will iterate and refine methods based on pilot feedback, recording lessons learned for standards work.

Comparative Benchmarking of Integration Paradigms

We will us an additive manufacturing / MES integration use case to benchmark the efficacy and efficiency of standards-based, AI-generated and hybrid integration methods. Implement integration scenarios using:

A. Manual standards-based integration (e.g., OPC UA, MQTT, REST).
B. AI-generated integration logic (with minimal human mapping).
C. Hybrid AI + standards approach leveraging both technologies.

Based on the general metrics defined earlier, we will define and apply task specific, comprehensive metrics: integration effort (time, resources, manual steps), performance (throughput, latency, error rates), semantic correctness, and scalability to collect and analyze data to produce comparative, evidence-based insights on strengths and trade-offs of each approach.

Synthesis, Reporting, and Standards Advancement

We will Synthesize results from all tasks into structured guidance, standards recommendations, and NIST AMS technical reports, including:

• Sharing metric frameworks and lessons learned to working groups (e.g., ISO/IEC JTC 1 SC42, manufacturing standards bodies).
• Disseminating recommendations and best practices to manufacturers, technology providers, and collaborating institutes.
• Developing and promoting a roadmap for technology adoption, aligned with the National AI Strategy and the White House AI Action Plan.
• Fostering ongoing ecosystem engagement to ensure frameworks and guidance remain aligned with industry needs and technological advances.