AI Business Analyst

Job Details: Job Location: Allied Benefit Systems - CHICAGO, IL, Position Type: Full Time, Salary Range: $105000.00 - $125000.00Salary, Job Category: Operations, POSITION SUMMARY The AI Business Analyst will be responsible for evaluating whether Allied’s internal AI solutions are functioning as intended, identifying where they succeed and where they fall short, and translating those findings into clear, actionable feedback for product and engineering teams. ESSENTIAL FUNCTIONS Evaluate AI capabilities integrated into core systems (e.g., Dynamics CRM, claims adjudication platform, document repositories) to determine whether they are functioning as intended and delivering expected business outcomes Investigate edge cases, anomalies, and failure patterns by reviewing raw data, logs, and system outputs Collaborate with technical product owners and SMEs to document what is and isn’t working, and help translate findings into user stories and product requirements Participate in model and capability reviews, providing business-level validation of AI behavior and identifying gaps in functionality or alignment with intended use Contribute to the development of test scenarios and evaluation frameworks that reflect real-world usage and business expectations Recommend improvements or refinements to AI capabilities based on observed behavior, user feedback, and business context Support the AI Product Owner and QA team in defining acceptance criteria and success metrics for new AI features Help ensure that AI solutions are understandable, trustworthy, and aligned with Allied’s operational goals Other duties as assigned EDUCATION Bachelor’s degree in related field or equivalent work experience required. EXPERIENCE AND SKILLS 8–12 years of experience in business analysis, quality assurance, or systems evaluation roles, ideally within the insurance or administrative healthcare industry Deep subject matter expertise in claims, health call centers, provider operations, or health plan administration Strong analytical mindset with the ability to identify subtle patterns, inconsistencies, or edge cases in complex systems Comfortable working with structured and unstructured data, including reviewing logs, outputs, and raw files to trace issues or validate behavior Experience writing clear, structured documentation such as user stories, test cases, and issue reports Familiarity with AI/ML systems and their evaluation is a plus, but not required Excellent communication skills and the ability to collaborate across technical and non-technical teams Familiarity with tools for data inspection and analysis (e.g., Excel, SQL, Power BI) Experience with log analysis and debugging tools (e.g., Azure Monitor, Splunk) Exposure to API testing tools (e.g., Postman) and basic scripting or query languages is helpful Agile collaboration tools (e.g., Freshservice, Azure DevOps, or similar) for tracking issues and documenting requirements Play a key role in Allied’s AI delivery team, helping ensure that AI capabilities are not only technically sound but also aligned with business needs and user expectations POSITION COMPETENCIES Accountability Analytical Thinking Collaboration Communication Customer Focus Functional Expertise Initiative PHYSICAL DEMANDS This is a standard desk role requiring extended sitting and computer work. WORK ENVIRONMENT Remote Here at Allied, we believe that great talent can thrive from anywhere. Our remote friendly culture offers flexibility and the comfort of working from home; while also ensuring you are set up for success. To support a smooth and efficient remote work experience, the internet connection must be obtained through a cable broadband or fiber optic internet service provider with speeds of at least 100Mbps download/25Mbps upload. Reliable internet service is essential for staying connected and productive. The company has reviewed this job description to ensure that essential functions and basic duties have been included. It is not intended to be construed as an exhaustive list of all functions, responsibilities, skills, and abilities. Additional functions and requirements may be assigned by supervisors as deemed appropriate. Compensation is not limited to base salary. Allied values our Total Rewards, and offers a competitive Benefit Package including, but not limited to, Medical, Dental, Vision, Life and Disability Insurance, Generous Paid Time Off, Tuition Reimbursement, EAP, and a Technology Stipend. Allied reserves the right to amend, change, alter, and revise, pay ranges and benefits offerings at any time. All applicants acknowledge that by applying to the position you understand that the specific pay range is contingent upon meeting the qualification and requirements of the role, and for the successful completion of the interview selection and process. It is at the Company's discretion to determine what pay is provided to a candidate within the range associated with the role.Qualifications: