AI Benchmarking Lead, Performance Benchmarking Evaluation

ID: 5130

Type: Full-time

Category: Others

Company Name: ADCI HYD 13 SEZ - H84

Location: IND, TS, Hyderabad

Education Level: Lead

Job Description

Join our mission-critical team supporting Seller Assistant, Amazon's Gen-AI powered copilot that helps sellers navigate Amazon's complex ecosystem and grow their businesses. As a Quality Assurance Specialist, you'll play a pivotal role in ensuring the reliability and accuracy of AI model evaluations as we scale from 61% to 90%+ active seller coverage worldwide.

About Seller Assistant
Seller Assistant is a conversational AI copilot that understands the full context of a seller's business. It intelligently orchestrates backend tools to deliver actionable, drilled-down responses and can independently complete complex tasks on behalf of sellers with their permission.
Our Scale and Impact:
- Expanded to 2.44MM sellers (45x growth vs. Dec 2024)
- Currently serving 61% of active sellers worldwide across 9 international stores (CN2XX, IN, UK, DE, JP, BR, MX, AE, SA)
- Supporting four languages: English, Chinese, German, and Japanese
- 2026 Goal: Scale to 90%+ active sellers WW with 5 new store launches (France, Italy, Spain, Canada, Australia)
As a Quality Assurance Specialist/AI Benchmarking Lead, you will benchmark Seller Assistant AI models for relevancy, correctness, and completeness. Your primary responsibilities include: 1) Evaluate audits performed by the core auditing team to increase confidence in evaluation metrics, 2) Improve audit reliability and consistency through systematic measurement of auditor accuracy,3) Conduct targeted calibration to ensure quality standards across the auditing function, 4) Enforce quality standards by quality-checking audits and providing actionable feedback to team members, 5) Drive continuous improvement in audit processes and methodologies.

- You conduct quality checks on audits performed by the core auditing team.
- You identify rubric gaps and evaluation ambiguities that lead to inconsistent audit outcomes.
- You surface high-confidence product issues earlier by validating and categorizing model failures.
- You serve as point of contact for annotation tasks across ML data process areas, ensuring quality execution and delivery
- You understand dependencies across ML data workflows and articulate customer impact effectively
- You modify existing annotation methods and update SOPs.
- You document SOP changes, secure approval, share knowledge with the team, and audit adoption and execution
- You test new SOPs and tools, providing feedback on quality and improvement recommendations to support onboarding

Key job responsibilities
- You structure data collection, analyse results and share inputs for SOP changes.
- You collate, track, and report progress on key metrics agreed to with respective stakeholders (e.g., Program managers, Applied Scientist) specific to your functional area.
- You identify operational issues related to process and tooling and recommend suggestions to improve key project metrics such as productivity and quality.

Basic Qualifications

- Bachelor's degree or equivalent
- Experience in natural language data labeling, data annotation, linguistic annotation or other forms of data markup
- Technical Skills: Proficiency in MS Excel; basic understanding of SQL and Python
- Experience with Microsoft Office products and applications
- Communication Skills: Strong verbal and written communication skills in English
- Knowledge about SOA and process that deal with sellers.

Preferred Qualifications

- 1 to 3 years of equivalent experience
- Performed annotation related tasks across ML data process areas.
- Strong knowledge of process documentation, analysis knowledge
- Technical proficiency in SQL querying and Python programming for data analysis
- Strong analytical and problem-solving skills
- Ability to work independently and as part of a team

Our inclusive culture empowers Amazonians to deliver the best results for our customers. If you have a disability and need a workplace accommodation or adjustment during the application and hiring process, including support for the interview or onboarding process, please visit https://amazon.jobs/content/en/how-we-hire/accommodations for more information. If the country/region you’re applying in isn’t listed, please contact your Recruiting Partner.

Company Information

Company Name: ADCI HYD 13 SEZ - H84

Company Website: https://www.accenture.com

Company Address: Financial District, Nanakramguda Gachibowli, Hyderabad Telangana 500032 India

Accenture Development Centre India (commonly abbreviated ADCI) is the India-based research, development and delivery arm of Accenture, a global professional services company that provides consulting, technology, and outsourcing services. In public and corporate materials, ADCI refers to Accenture’s network of development centres and delivery hubs across India—of which Hyderabad is one of the largest and most strategically important locations. These India-based development centres are structured to support Accenture’s global clients by designing, building, testing and operating technology solutions across multiple industries and technology domains. Company overview Accenture (through its India development centres) operates as a technology services and solutions organization embedded within a global professional services firm. The ADCI organisation in India brings together software engineering, industry specialists, data scientists, cloud architects, security experts, automation engineers and consulting practitioners to deliver services ranging from strategy and design through implementation and managed operations. The Hyderabad ADCI presence is a major delivery site that houses engineering teams, cloud and infrastructure operations, innovation labs, and client delivery units that serve North America, Europe and Asia-Pacific clients. Core business activities ADCI’s core activities are aligned with Accenture’s broader service offerings and include: technology consulting, systems integration, application development and maintenance, cloud migration and cloud-native application engineering, digital transformation, data & analytics and AI solutions, cybersecurity and managed security services, intelligent automation and robotic process automation (RPA), testing and quality engineering, and business process outsourcing (BPO) and managed services. In Hyderabad specifically, ADCI teams often focus on large-scale implementation programs, platform engineering and continuous delivery for enterprise ERP, CRM, Industry X.0 initiatives, and industry-specific digital services (for example, for financial services, communications, healthcare, and public sector clients). Main products and services While Accenture as a whole markets a portfolio of branded platforms and solutions, the ADCI centres primarily act as engineering and delivery engines that build, customize, integrate and manage those solutions for clients. Key service areas and offerings produced or supported by ADCI teams include: - Cloud and Infrastructure Services: cloud migration, multi-cloud architecture, cloud-native application development, and platform engineering. ADCI teams help implement Accenture’s cloud assessment, migration and managed cloud services. - Digital and Application Services: end-to-end software development, modernization of legacy applications, microservices architectures, API management and DevOps/continuous delivery pipelines. - Data & AI: data engineering, analytics, artificial intelligence and machine learning model development, MLOps and decisioning platforms to help clients operationalize data-driven insights. - Security Services: cybersecurity engineering, threat detection, incident response, secure-by-design development practices, identity and access management and managed security operations. - Intelligent Automation and Operations: RPA, process mining, low-code/no-code automation, and managed services to improve efficiency and scale operations. - Industry Solutions: implementation and customization of industry-specific platforms and suites (for example, ERP/CRM transformations) and vertical digital solutions for sectors such as financial services, communications, consumer goods, health, and public services. - Innovation and Emerging Technology Labs: prototyping, proof-of-concept development and early-stage engineering work in domains such as Internet of Things (IoT), edge computing, blockchain, extended reality (XR) and quantum-ready computing research. Role within Accenture’s global footprint ADCI centres, including Hyderabad, serve as central engineering and delivery hubs that enable Accenture’s global teams to scale quickly and provide cost-effective technical talent and project delivery. These centres collaborate with Accenture’s industry groups, local client teams, global delivery centers and innovation studios to translate business strategy into implemented technology solutions. ADCI staff frequently contribute to development of Accenture-branded platforms and assets—such as tools for cloud migration, automation frameworks and analytics accelerators—that are reused across client projects. Clients and market positioning ADCI supports a diverse set of clients—ranging from large multinational enterprises to governments and startups—primarily through Accenture’s client relationships. The Hyderabad development centre is positioned as a high-capacity engineering and operations hub that provides both bespoke engineering work and repeatable platform implementations. By embedding multidisciplinary teams (consultants, designers, engineers and operations specialists) within client programs, ADCI aims to deliver end-to-end outcomes: strategy and design through build, deploy and run. Organizational mission and approach Consistent with Accenture’s publicly stated positioning, ADCI aligns to the company’s mission of combining technology and human ingenuity to help clients become high-performance businesses and governments. ADCI emphasises agile engineering practices, cloud-first architectures, secure development lifecycles and continuous innovation—leveraging global best practices and localized execution capabilities. The Hyderabad ADCI presence also focuses on talent development, partnering with academic institutions, and building specialized capability centers for emerging technologies. Note on naming and identification The string “ADCI HYD 13 SEZ - H84” is consistent with internal or site-specific designations used to identify a particular Accenture Development Centre India (ADCI) unit or facility within a Hyderabad Special Economic Zone (SEZ) campus. For public-facing information and official corporate references, Accenture’s global website and Accenture India pages provide authoritative descriptions of the company’s capabilities, service lines and India delivery footprint.

Visit company vacancy