# Applied AI Engineer Intern - Advancing Agent Quality at Brightstar.AI - Company: Brightstar.AI - Location: Miami, FL - Type: Internship · Internship - Posted: 2026-09-24 - Apply: https://jobs.ashbyhq.com/brightstar-ai/a72ffc7e-38ba-4d7c-8d08-cac6b9974024/application?embed=true - Page: https://newgrad.ai/jobs/applied-ai-engineer-intern-advancing-agent-quality-d96a5b80 ## Summary by newgrad.ai What you'll do: - Design realistic agent environments and task suites for board deliberation, research, and multi-step tool use. - Develop LLM judges calibrated against human experts and benchmark agents against frontier models. - Build diagnostic tools to identify agent failure modes and feed fixes back into prompts and architecture. - Create datasets and reward signals from multi-turn interactions for model fine-tuning. - Contribute AI research perspectives to weekly Think Tank sessions and strategic discussions. What they're looking for: - Currently in Masters or PhD program in AI, ML, CS, or related field, or postdoctoral researcher. - 1 year experience with LLMs or agents; 2 years with machine learning and deep learning. - Software engineering and cloud development experience; familiarity with agent builders. - Ability to explain technical concepts to non-technical executives. - Preferred: LLM evaluation systems experience, multi-agent orchestration, research publications in AI safety or reasoning. ## Posting Role Type: External Research Affiliate / Think Tank Contributor Location: South Florida preferred Affiliation: Local university-based Masters or PhD researcher or postdoctoral AI researcher working under faculty direction Organization: Brightstar.AI ### Role Summary Brightstar.AI is seeking a full-time or recently graduated Masters or PhD-level Applied AI Engineer Intern – Advancing Agent Quality with a strong research mindset, affiliated with a leading (South Florida) university, to contribute to its AI Think Tank and to the hands-on build-out of Brightstar's proprietary AI systems. The intern will pair academic rigor with applied engineering: building the evaluation environments, automated raters, and diagnostic tooling that establish — with evidence rather than impression — whether an agent is reliable enough to put in front of operators, executives, and investors. The first concrete assignment is the AI Twin Board, Brightstar's proprietary board-simulation product, where the intern will help build the evaluation harness end to end: realistic task suites, calibrated LLM judge, failure-mode diagnostics, and the data flywheel that feeds improvements back into the agents themselves. The AI Twin Board is the starting point, not the full scope — as Brightstar's AI portfolio expands, the same quality discipline will be applied to other agentic systems, internal platforms, and portfolio-company deployments. Working in coordination with university professors and research leaders, the intern will also bring up-to-date knowledge of the fast-evolving AI landscape — especially large language models, reasoning systems, emerging model capabilities, and the societal implications of advanced AI — and help translate what current and next-generation systems may enable over the next six months to three years into Brightstar's evaluation standards and Think Tank point of view. ### Key Responsibilities - Design, build, and scale realistic agent environments and task suites that reflect how Brightstar's agents are actually used — board-level deliberation, research synthesis, diligence workflows, and multi-step tool use. - Develop agentic LLM judge, calibrate them against human expert evaluation, and benchmark Brightstar agents against leading frontier models. - Build trajectory-analysis frameworks and diagnostic tooling that root-cause agent failure modes — passivity, hallucination, persona drift, brittle tool execution — and feed fixes back into agent prompts, harnesses, and architecture. - Enable agents to generate and iterate on their own verifiers (automated test cases, checklists, ground-truth sets) to support effective exploration and iterative problem-solving. - Harvest multi-turn interaction trajectories into high-quality datasets and reward signals that support fine-tuning and post-training of the models Brightstar relies on. - Contribute to the AI Twin Board build-out as the initial focus, and extend the same evaluation approach to other Brightstar AI initiatives as the roadmap grows. - Contribute frontier AI research perspectives to the Brightstar.AI Think Tank as part of weekly review sessions and select strategic discussions. - Review relevant research papers, conference findings, and publications from leading AI institutions and labs as inputs to the Think Tank knowledge base and to Brightstar's evaluation methodology. - Help interpret academic and technical developments for a business and investment audience, including implications for industry transformation and applied AI opportunities. ### Minimum Qualifications - Currently enrolled in a Masters or PhD program — or serving as a postdoctoral or early-career academic researcher — in Artificial Intelligence, Machine Learning, Computer Science, Computational Linguistics, or a related field. - 1 years of research, project, or professional experience with LLMs or Agents. - 2 year of experience with machine learning and deep learning. - Experience in software engineering and cloud-based development. - Experience with agent builders, e.g. OpenSDK, OpenAI Agent Builder, etc. - Able to communicate complex technical concepts clearly to non-technical executive audiences. ### Preferred Qualifications - In-depth knowledge of machine learning algorithms, including supervised learning and reinforcement learning. - Hands-on experience building LLM or agent evaluation systems — autoraters, human-evaluation pipelines, benchmark design, or evaluation infrastructure. - Experience with multi-agent orchestration, retrieval-augmented generation, and tool-use / function-calling reliability. - Active research or publications in LLMs, reasoning systems, model behavior, AI safety, or the societal impact of advanced AI systems. - Strong research discipline, intellectual curiosity, and the ability to separate meaningful signal from AI hype. - Affiliated with a (South Florida) university and working under the supervision of a faculty member or professor, or recent graduate. ### Why This Role Matters Brightstar.AI's Think Tank is intended to provide systematic forward-looking intelligence on where AI may create value, where it may disrupt industries, and how firms can better anticipate what may work tomorrow rather than only what worked yesterday. Masters and PhD-level Applied AI Engineers are a critical part of that model, bringing academic depth, frontier perspective, and analytical rigor into the Think Tank's quarterly insights and longer-term point of view. Moreover, this role extends beyond research to focus on the practical application of AI in terms of AI Twin Board technology, which we see as a key differentiator in our research approach and value creation methodology. From Brightstar.AI's public posting, laid out for reading by newgrad.ai with the wording unchanged. Check the original before you apply. --- More new-grad and intern roles: https://newgrad.ai/llms.txt