Role Description The AI Safety Institute research unit is looking for exceptionally motivated and talented people to join its Safeguard Analysis Team. Interventions that secure a system from abuse by bad actors will grow in importance as AI systems become more advanced and integrated into society. The AI Safety Institute's Safeguard Analysis Team researches such interventions, which it refers to as 'safeguards', evaluating protections used to secure current frontier AI systems and considering what measures could and should be used to secure such systems in the future. The Safeguard Analysis Team takes a broad view of security threats and interventions. It's keen to hire researchers with expertise developing and analysing attacks and protections for systems based on large language models, but is also keen to hire security researchers who have historically worked outside of AI, such as in - non-exhaustively - computer security, information security, web technology policy, and hardware security. Diverse perspectives and research interests are welcomed. The Team seeks people with skillsets leaning in the direction of either or both of Research Scientist and Research Engineer, recognising that some technical staff may prefer work that spans or alternates between engineering and research responsibilities. The Team's priorities include research-oriented responsibilities - like assessing the threats to frontier systems and developing novel attacks - and engineering-oriented ones, such as building infrastructure for running evaluations. In this role, you'll receive mentorship and coaching from your manager and the technical leads on your team. You'll also regularly interact with world-famous researchers and other incredible staff, including alumni from Anthropic, DeepMind, OpenAI and ML professors from Oxford and Cambridge. In addition to Junior roles, Senior, Staff and Principal RE positions are available for candidates with the required seniority and experience. Person Specification You may be a good fit if you have some of the following skills, experience and attitudes: Experience working on machine learning, AI, AI security, computer security, information security, or some other security discipline in industry, in academia, or independently. Experience working with a world-class research team comprised of both scientists and engineers (e.g. in a top-3 lab). Red-teaming experience against any sort of system. Strong written and verbal communication skills. Comprehensive understanding of large language models (e.g. GPT-4). This includes both a broad understanding of the literature, as well as hands-on experience with things like pre-training or fine-tuning LLMs. Extensive Python experience, including understanding the intricacies of the language, the good vs. bad Pythonic ways of doing things and much of the wider ecosystem/tooling. Ability to work in a self-directed way with high agency, thriving in a constantly changing environment and a steadily growing team, while figuring out the best and most efficient ways to solve a particular problem. Bring your own voice and experience but also an eagerness to support your colleagues together with a willingness to do whatever is necessary for the team's success and find new ways of getting things done. Have a sense of mission, urgency, and responsibility for success, demonstrating problem-solving abilities and preparedness to acquire any missing knowledge necessary to get the job done. Writing production quality code. Improving technical standards across a team through mentoring and feedback. Designing, shipping, and maintaining complex tech products. Salary & Benefits We are hiring individuals at all ranges of seniority and experience within the research unit, and this advert allows you to apply for any of the roles within this range. We will discuss and calibrate with you as part of the process. The full range of salaries available is as follows: L3: £65,000 - £75,000 L4: £85,000 - £95,000 L5: £105,000 - £115,000 L6: £125,000 - £135,000 L7: £145,000 There are a range of pension options available which can be found through the Civil Service website. Selection Process In accordance with the Civil Service Commission rules, the following list contains all selection criteria for the interview process. Required Experience This job advert encompasses a range of possible research and engineering roles within the Safeguard Analysis Team. The 'required' experiences listed below should be interpreted as examples of the expertise we're looking for, as opposed to a list of everything we expect to find in one applicant: Writing production quality code Writing code efficiently Python Frontier model architecture knowledge Frontier model training knowledge Model evaluations knowledge AI safety research knowledge Security research knowledge Research problem selection Research science Written communication Verbal communication Teamwork Interpersonal skills Tackle challenging problems Learn through coaching
Jul 16, 2025
Full time
Role Description The AI Safety Institute research unit is looking for exceptionally motivated and talented people to join its Safeguard Analysis Team. Interventions that secure a system from abuse by bad actors will grow in importance as AI systems become more advanced and integrated into society. The AI Safety Institute's Safeguard Analysis Team researches such interventions, which it refers to as 'safeguards', evaluating protections used to secure current frontier AI systems and considering what measures could and should be used to secure such systems in the future. The Safeguard Analysis Team takes a broad view of security threats and interventions. It's keen to hire researchers with expertise developing and analysing attacks and protections for systems based on large language models, but is also keen to hire security researchers who have historically worked outside of AI, such as in - non-exhaustively - computer security, information security, web technology policy, and hardware security. Diverse perspectives and research interests are welcomed. The Team seeks people with skillsets leaning in the direction of either or both of Research Scientist and Research Engineer, recognising that some technical staff may prefer work that spans or alternates between engineering and research responsibilities. The Team's priorities include research-oriented responsibilities - like assessing the threats to frontier systems and developing novel attacks - and engineering-oriented ones, such as building infrastructure for running evaluations. In this role, you'll receive mentorship and coaching from your manager and the technical leads on your team. You'll also regularly interact with world-famous researchers and other incredible staff, including alumni from Anthropic, DeepMind, OpenAI and ML professors from Oxford and Cambridge. In addition to Junior roles, Senior, Staff and Principal RE positions are available for candidates with the required seniority and experience. Person Specification You may be a good fit if you have some of the following skills, experience and attitudes: Experience working on machine learning, AI, AI security, computer security, information security, or some other security discipline in industry, in academia, or independently. Experience working with a world-class research team comprised of both scientists and engineers (e.g. in a top-3 lab). Red-teaming experience against any sort of system. Strong written and verbal communication skills. Comprehensive understanding of large language models (e.g. GPT-4). This includes both a broad understanding of the literature, as well as hands-on experience with things like pre-training or fine-tuning LLMs. Extensive Python experience, including understanding the intricacies of the language, the good vs. bad Pythonic ways of doing things and much of the wider ecosystem/tooling. Ability to work in a self-directed way with high agency, thriving in a constantly changing environment and a steadily growing team, while figuring out the best and most efficient ways to solve a particular problem. Bring your own voice and experience but also an eagerness to support your colleagues together with a willingness to do whatever is necessary for the team's success and find new ways of getting things done. Have a sense of mission, urgency, and responsibility for success, demonstrating problem-solving abilities and preparedness to acquire any missing knowledge necessary to get the job done. Writing production quality code. Improving technical standards across a team through mentoring and feedback. Designing, shipping, and maintaining complex tech products. Salary & Benefits We are hiring individuals at all ranges of seniority and experience within the research unit, and this advert allows you to apply for any of the roles within this range. We will discuss and calibrate with you as part of the process. The full range of salaries available is as follows: L3: £65,000 - £75,000 L4: £85,000 - £95,000 L5: £105,000 - £115,000 L6: £125,000 - £135,000 L7: £145,000 There are a range of pension options available which can be found through the Civil Service website. Selection Process In accordance with the Civil Service Commission rules, the following list contains all selection criteria for the interview process. Required Experience This job advert encompasses a range of possible research and engineering roles within the Safeguard Analysis Team. The 'required' experiences listed below should be interpreted as examples of the expertise we're looking for, as opposed to a list of everything we expect to find in one applicant: Writing production quality code Writing code efficiently Python Frontier model architecture knowledge Frontier model training knowledge Model evaluations knowledge AI safety research knowledge Security research knowledge Research problem selection Research science Written communication Verbal communication Teamwork Interpersonal skills Tackle challenging problems Learn through coaching
Role Description The AI Safety Institute research unit is looking for exceptionally motivated and talented Research Scientists and Research Engineers to work in the Societal impacts team. Societal Impacts Societal Impacts is a multidisciplinary team that studies how advanced AI models can impact people and society. Core research topics include the use of AI for assisting with criminal activities, undermining trust in information, jeopardising psychological wellbeing, or for malicious social engineering. We are interested in both immediate and medium-term risks. In this role, you'll join a strongly collaborative technical research team led by the Societal Impacts Research Director, Professor Christopher Summerfield. You will receive mentorship, training, and opportunities for development. You'll also regularly interact with our highly talented and experienced staff across the Institute (including alumni from Anthropic, DeepMind, OpenAI and ML professors from Oxford and Cambridge), as well as other partners across government. In addition to Junior roles, Senior, Staff and Principal RE positions are available for candidates with the required seniority and experience. Person Specification Successful candidates will work with other researchers to design and run studies that answer important questions about the effect AI will have on society. For example, can AI effectively change people's political and social views? Research Scientists/Engineers have scope to use a range of research methodologies and drive the strategy of the team. This is a multidisciplinary team and we look for people with a diversity of backgrounds. We are especially excited about candidates with experience of research in one or more of these areas: Computational social science Machine learning (research engineer / research scientist) Data Science, especially including Natural Language Processing Advanced statistical modelling and experimental design. Required Skills and Experience We select based on skills and experience regarding the following areas: Writing production quality code Writing code efficiently, especially using Python Demonstrable interest in the societal impacts of AI Experimental design Demonstrable experience running research experiments involving AI models and/or human participants Strong quantitative skills Data analytics Data science methods Frontier model architecture knowledge Frontier model training knowledge Model evaluations knowledge AI safety research knowledge Research problem selection Research science Verbal communication Teamwork Interpersonal skills Desired Skills and Experience Written communication Published work related to cognitive, social or political social science. A specialization in a particular field of social or political science, economics, cognitive science, criminology, security studies, AI safety, or another relevant field. Front-end software engineering skills to build UI for studies with human participants. Salary & Benefits We are hiring individuals at all ranges of seniority and experience within the research unit, and this advert allows you to apply for any of the roles within this range. We will discuss and calibrate with you as part of the process. The full range of salaries available is as follows: L3: £65,000 - £75,000 L4: £85,000 - £95,000 L5: £105,000 - £115,000 L6: £125,000 - £135,000 L7: £145,000 There are a range of pension options available which can be found through the Civil Service website. Selection Process In accordance with the Civil Service Commission rules, the following list contains all selection criteria for the interview process.
Feb 12, 2025
Full time
Role Description The AI Safety Institute research unit is looking for exceptionally motivated and talented Research Scientists and Research Engineers to work in the Societal impacts team. Societal Impacts Societal Impacts is a multidisciplinary team that studies how advanced AI models can impact people and society. Core research topics include the use of AI for assisting with criminal activities, undermining trust in information, jeopardising psychological wellbeing, or for malicious social engineering. We are interested in both immediate and medium-term risks. In this role, you'll join a strongly collaborative technical research team led by the Societal Impacts Research Director, Professor Christopher Summerfield. You will receive mentorship, training, and opportunities for development. You'll also regularly interact with our highly talented and experienced staff across the Institute (including alumni from Anthropic, DeepMind, OpenAI and ML professors from Oxford and Cambridge), as well as other partners across government. In addition to Junior roles, Senior, Staff and Principal RE positions are available for candidates with the required seniority and experience. Person Specification Successful candidates will work with other researchers to design and run studies that answer important questions about the effect AI will have on society. For example, can AI effectively change people's political and social views? Research Scientists/Engineers have scope to use a range of research methodologies and drive the strategy of the team. This is a multidisciplinary team and we look for people with a diversity of backgrounds. We are especially excited about candidates with experience of research in one or more of these areas: Computational social science Machine learning (research engineer / research scientist) Data Science, especially including Natural Language Processing Advanced statistical modelling and experimental design. Required Skills and Experience We select based on skills and experience regarding the following areas: Writing production quality code Writing code efficiently, especially using Python Demonstrable interest in the societal impacts of AI Experimental design Demonstrable experience running research experiments involving AI models and/or human participants Strong quantitative skills Data analytics Data science methods Frontier model architecture knowledge Frontier model training knowledge Model evaluations knowledge AI safety research knowledge Research problem selection Research science Verbal communication Teamwork Interpersonal skills Desired Skills and Experience Written communication Published work related to cognitive, social or political social science. A specialization in a particular field of social or political science, economics, cognitive science, criminology, security studies, AI safety, or another relevant field. Front-end software engineering skills to build UI for studies with human participants. Salary & Benefits We are hiring individuals at all ranges of seniority and experience within the research unit, and this advert allows you to apply for any of the roles within this range. We will discuss and calibrate with you as part of the process. The full range of salaries available is as follows: L3: £65,000 - £75,000 L4: £85,000 - £95,000 L5: £105,000 - £115,000 L6: £125,000 - £135,000 L7: £145,000 There are a range of pension options available which can be found through the Civil Service website. Selection Process In accordance with the Civil Service Commission rules, the following list contains all selection criteria for the interview process.
Control Team Our team's focus is on ensuring that even if frontier AI systems are misaligned, they can be effectively controlled . To achieve this, we are attempting to advance the state of conceptual research into control protocols and corresponding safety cases. Additionally, we will conduct realistic empirical research on mock frontier AI development infrastructure, helping to identify flaws in theoretical approaches and refine them accordingly. Role Summary As the lead research scientist on Control, you'll lead the conceptual and theoretical research efforts on control. Your team will initially include 3-4 research scientists, including researchers with existing experience in the control agenda and/or experience at frontier labs. Your responsibilities will encompass setting the research direction & agenda, ambitiously advancing the state of control research, as well as managing and developing an exceptional team. The ultimate goal is to make substantial improvements in the robustness of control protocols across major labs, particularly as we progress towards AGI. The role will involve close collaboration with our research directors, including Geoffrey Irving and Yarin Gal , and work hand-in-hand with the Control empirics team. The empirics team will support your efforts by building realistic control settings that closely mimic the infrastructure and codebases used for frontier AI development, and by helping to develop empirical experiments. Research partnerships and collaborations with many of the leading frontier AI labs will also be a significant part of your role. From a compute perspective, you will have excellent access to resources from both our research platform team and the UK's Isambard supercomputer (5,000 H100s). Person Specification You may be a good fit if you have some of the following skills, experience and attitudes. Please note that you don't need to meet all of these criteria, and if you're unsure, we encourage you to apply. Experience leading a research team or group that has delivered exceptional research in deep learning or a related field. Comprehensive understanding of frontier AI development, including key processes involved in research, data collection & generation, pre-training, post-training and safety assessment. Proven track record of academic excellence, demonstrated by novel research contributions and spotlight papers at top-tier conferences (e.g., NeurIPS, ICML, ICLR). Exceptional written and verbal communication skills, with the ability to convey complex ideas clearly and effectively to diverse audiences. Extensive experience in collaborating with multi-disciplinary teams, including researchers and engineers, and leading high-impact projects. A strong desire to improve the global state of AI safety. While existing experience working on control is desired, it is not a requirement for this role. Salary & Benefits We are hiring individuals at the more senior ranges of the following scale (L5-L7). Your dedicated talent partner will work with you as you move through our assessment process to explain our internal benchmarking process. The full range of salaries are available below, salaries comprise of a base salary, technical allowance plus additional benefits as detailed on this page. Level 3 - Total Package £65,000 - £75,000 inclusive of a base salary £35,720 plus additional technical talent allowance of between £29,280 - £39,280 Level 4 - Total Package £85,000 - £95,000 inclusive of a base salary £42,495 plus additional technical talent allowance of between £42,505 - £52,505 Level 5 - Total Package £105,000 - £115,000 inclusive of a base salary £55,805 plus additional technical talent allowance of between £49,195 - £59,195 Level 6 - Total Package £125,000 - £135,000 inclusive of a base salary £68,770 plus additional technical talent allowance of between £56,230 - £66,230 Level 7 - Total Package £145,000 inclusive of a base salary £68,770 plus additional technical talent allowance of £76,230 There are a range of pension options available which can be found through the Civil Service website. Selection Process In accordance with the Civil Service Commission rules, the following list contains all selection criteria for the interview process. Required Experience We select based on skills and experience regarding the following areas: Research problem selection Research science Writing code efficiently Python Frontier model architecture knowledge Frontier model training knowledge Model evaluations knowledge AI safety research knowledge Written communication Verbal communication Teamwork Interpersonal skills Tackle challenging problems Learn through coaching Desired Experience We additionally may factor in experience with any of the areas that our work-streams specialise in: Autonomous systems Cyber security Chemistry or Biology Safeguards Safety Cases Societal Impacts
Feb 11, 2025
Full time
Control Team Our team's focus is on ensuring that even if frontier AI systems are misaligned, they can be effectively controlled . To achieve this, we are attempting to advance the state of conceptual research into control protocols and corresponding safety cases. Additionally, we will conduct realistic empirical research on mock frontier AI development infrastructure, helping to identify flaws in theoretical approaches and refine them accordingly. Role Summary As the lead research scientist on Control, you'll lead the conceptual and theoretical research efforts on control. Your team will initially include 3-4 research scientists, including researchers with existing experience in the control agenda and/or experience at frontier labs. Your responsibilities will encompass setting the research direction & agenda, ambitiously advancing the state of control research, as well as managing and developing an exceptional team. The ultimate goal is to make substantial improvements in the robustness of control protocols across major labs, particularly as we progress towards AGI. The role will involve close collaboration with our research directors, including Geoffrey Irving and Yarin Gal , and work hand-in-hand with the Control empirics team. The empirics team will support your efforts by building realistic control settings that closely mimic the infrastructure and codebases used for frontier AI development, and by helping to develop empirical experiments. Research partnerships and collaborations with many of the leading frontier AI labs will also be a significant part of your role. From a compute perspective, you will have excellent access to resources from both our research platform team and the UK's Isambard supercomputer (5,000 H100s). Person Specification You may be a good fit if you have some of the following skills, experience and attitudes. Please note that you don't need to meet all of these criteria, and if you're unsure, we encourage you to apply. Experience leading a research team or group that has delivered exceptional research in deep learning or a related field. Comprehensive understanding of frontier AI development, including key processes involved in research, data collection & generation, pre-training, post-training and safety assessment. Proven track record of academic excellence, demonstrated by novel research contributions and spotlight papers at top-tier conferences (e.g., NeurIPS, ICML, ICLR). Exceptional written and verbal communication skills, with the ability to convey complex ideas clearly and effectively to diverse audiences. Extensive experience in collaborating with multi-disciplinary teams, including researchers and engineers, and leading high-impact projects. A strong desire to improve the global state of AI safety. While existing experience working on control is desired, it is not a requirement for this role. Salary & Benefits We are hiring individuals at the more senior ranges of the following scale (L5-L7). Your dedicated talent partner will work with you as you move through our assessment process to explain our internal benchmarking process. The full range of salaries are available below, salaries comprise of a base salary, technical allowance plus additional benefits as detailed on this page. Level 3 - Total Package £65,000 - £75,000 inclusive of a base salary £35,720 plus additional technical talent allowance of between £29,280 - £39,280 Level 4 - Total Package £85,000 - £95,000 inclusive of a base salary £42,495 plus additional technical talent allowance of between £42,505 - £52,505 Level 5 - Total Package £105,000 - £115,000 inclusive of a base salary £55,805 plus additional technical talent allowance of between £49,195 - £59,195 Level 6 - Total Package £125,000 - £135,000 inclusive of a base salary £68,770 plus additional technical talent allowance of between £56,230 - £66,230 Level 7 - Total Package £145,000 inclusive of a base salary £68,770 plus additional technical talent allowance of £76,230 There are a range of pension options available which can be found through the Civil Service website. Selection Process In accordance with the Civil Service Commission rules, the following list contains all selection criteria for the interview process. Required Experience We select based on skills and experience regarding the following areas: Research problem selection Research science Writing code efficiently Python Frontier model architecture knowledge Frontier model training knowledge Model evaluations knowledge AI safety research knowledge Written communication Verbal communication Teamwork Interpersonal skills Tackle challenging problems Learn through coaching Desired Experience We additionally may factor in experience with any of the areas that our work-streams specialise in: Autonomous systems Cyber security Chemistry or Biology Safeguards Safety Cases Societal Impacts
About the Team The Post-Training Team is dedicated to optimising AI systems to achieve state-of-the-art performance across the various risk domains that AISI focuses on. This is accomplished through a combination of scaffolding, prompting, supervised and RL fine-tuning of the AI models which AISI has access to. One of the main focuses of our evaluation teams is estimating how new models might affect the capabilities of AI systems in specific domains. To improve confidence in our assessments, we make significant effort to enhance the model's performance in the domains of interest. For many of our evaluations, this means taking a model we have been given access to and embedding it as part of a wider AI system-for example, in our cybersecurity evaluations, we provide models with access to tools for interacting with the underlying operating system and repeatedly call models to act in such environment. In our evaluations which do not require agentic capabilities, we may use elicitation techniques like fine-tuning and prompt engineering to ensure assessing the model at its full capacity. About the Role As a member of this team, you will use cutting-edge machine learning techniques to improve model performance in our domains of interest. The work is split into two sub-teams: Agents and Finetuning. Our Agents sub-team focuses on developing the LLM tools and scaffolding to create highly capable LLM-based agents, while our Finetuning Team builds out finetuning pipelines to improve models on our domains of interest. The Post-Training Team is seeking strong Research Scientists to join the team. The priorities of the team include both research-oriented tasks-such as designing new techniques for scaling inference-time computation or developing methodologies for in-depth analysis of agent behaviour-and engineering-oriented tasks-like implementing new tools for our LLM agents or creating pipelines for supporting and fine-tuning large open-source models. We recognise that some technical staff may prefer to span or alternate between engineering and research responsibilities, and this versatility is something we actively look for in our hires. You'll receive mentorship and coaching from your manager and the technical leads on your team, and regularly interact with world-class researchers and other exceptional staff, including alumni from Anthropic, DeepMind and OpenAI. In addition to junior roles, we offer Senior, Staff, and Principal Research Engineer positions for candidates with the requisite seniority and experience. Person Specification You may be a good fit if you have some of the following skills, experience and attitudes: Experience conducting empirical machine learning research (e.g. PhD in a technical field and/or papers at top ML conferences), particularly on LLMs. Experience with machine learning engineering, or extensive experience as a software engineer with a strong demonstration of relevant skills/knowledge in the machine learning. An ability to work autonomously and in a self-directed way with high agency, thriving in a constantly changing environment and a steadily growing team, while figuring out the best and most efficient ways to solve a particular problem. Particularly strong candidates also have the following experience: Building LLM agents in industry or open-source collectives, particularly in areas adjacent to the main interests of one of our workstreams e.g. in-IDE coding assistants, research assistants etc. (for our Agents subteam) Leading research on improving and measuring the capabilities of LLM agents (for our Agents sub-team) Building pipelines for fine-tuning (or pretraining LLMs). Finetuning with RL techniques is particularly relevant (for our finetuning subteam). Finetuning or pretraining LLMs in a research context, particularly to achieve increased performance in specific domains (for our finetuning subteam). Salary & Benefits We are hiring individuals at all ranges of seniority and experience within the research unit, and this advert allows you to apply for any of the roles within this range. We will discuss and calibrate with you as part of the process. The full range of salaries available is as follows: L3: £65,000 - £75,000 L4: £85,000 - £95,000 L5: £105,000 - £115,000 L6: £125,000 - £135,000 L7: £145,000 There are a range of pension options available which can be found through the Civil Service website. Selection Process In accordance with the Civil Service Commission rules, the following list contains all selection criteria for the interview process. Required Experience We select based on skills and experience regarding the following areas: Research problem selection Research Engineering Writing code efficiently Python Frontier model architecture knowledge Frontier model training knowledge Model evaluations knowledge AI safety research knowledge Written communication Verbal communication Teamwork Interpersonal skills Tackle challenging problems Learn through coaching Desired Experience We additionally may factor in experience with any of the areas that our work-streams specialise in: Cyber security Chemistry or Biology Safeguards Safety Cases Societal Impacts
Feb 11, 2025
Full time
About the Team The Post-Training Team is dedicated to optimising AI systems to achieve state-of-the-art performance across the various risk domains that AISI focuses on. This is accomplished through a combination of scaffolding, prompting, supervised and RL fine-tuning of the AI models which AISI has access to. One of the main focuses of our evaluation teams is estimating how new models might affect the capabilities of AI systems in specific domains. To improve confidence in our assessments, we make significant effort to enhance the model's performance in the domains of interest. For many of our evaluations, this means taking a model we have been given access to and embedding it as part of a wider AI system-for example, in our cybersecurity evaluations, we provide models with access to tools for interacting with the underlying operating system and repeatedly call models to act in such environment. In our evaluations which do not require agentic capabilities, we may use elicitation techniques like fine-tuning and prompt engineering to ensure assessing the model at its full capacity. About the Role As a member of this team, you will use cutting-edge machine learning techniques to improve model performance in our domains of interest. The work is split into two sub-teams: Agents and Finetuning. Our Agents sub-team focuses on developing the LLM tools and scaffolding to create highly capable LLM-based agents, while our Finetuning Team builds out finetuning pipelines to improve models on our domains of interest. The Post-Training Team is seeking strong Research Scientists to join the team. The priorities of the team include both research-oriented tasks-such as designing new techniques for scaling inference-time computation or developing methodologies for in-depth analysis of agent behaviour-and engineering-oriented tasks-like implementing new tools for our LLM agents or creating pipelines for supporting and fine-tuning large open-source models. We recognise that some technical staff may prefer to span or alternate between engineering and research responsibilities, and this versatility is something we actively look for in our hires. You'll receive mentorship and coaching from your manager and the technical leads on your team, and regularly interact with world-class researchers and other exceptional staff, including alumni from Anthropic, DeepMind and OpenAI. In addition to junior roles, we offer Senior, Staff, and Principal Research Engineer positions for candidates with the requisite seniority and experience. Person Specification You may be a good fit if you have some of the following skills, experience and attitudes: Experience conducting empirical machine learning research (e.g. PhD in a technical field and/or papers at top ML conferences), particularly on LLMs. Experience with machine learning engineering, or extensive experience as a software engineer with a strong demonstration of relevant skills/knowledge in the machine learning. An ability to work autonomously and in a self-directed way with high agency, thriving in a constantly changing environment and a steadily growing team, while figuring out the best and most efficient ways to solve a particular problem. Particularly strong candidates also have the following experience: Building LLM agents in industry or open-source collectives, particularly in areas adjacent to the main interests of one of our workstreams e.g. in-IDE coding assistants, research assistants etc. (for our Agents subteam) Leading research on improving and measuring the capabilities of LLM agents (for our Agents sub-team) Building pipelines for fine-tuning (or pretraining LLMs). Finetuning with RL techniques is particularly relevant (for our finetuning subteam). Finetuning or pretraining LLMs in a research context, particularly to achieve increased performance in specific domains (for our finetuning subteam). Salary & Benefits We are hiring individuals at all ranges of seniority and experience within the research unit, and this advert allows you to apply for any of the roles within this range. We will discuss and calibrate with you as part of the process. The full range of salaries available is as follows: L3: £65,000 - £75,000 L4: £85,000 - £95,000 L5: £105,000 - £115,000 L6: £125,000 - £135,000 L7: £145,000 There are a range of pension options available which can be found through the Civil Service website. Selection Process In accordance with the Civil Service Commission rules, the following list contains all selection criteria for the interview process. Required Experience We select based on skills and experience regarding the following areas: Research problem selection Research Engineering Writing code efficiently Python Frontier model architecture knowledge Frontier model training knowledge Model evaluations knowledge AI safety research knowledge Written communication Verbal communication Teamwork Interpersonal skills Tackle challenging problems Learn through coaching Desired Experience We additionally may factor in experience with any of the areas that our work-streams specialise in: Cyber security Chemistry or Biology Safeguards Safety Cases Societal Impacts
The Science of Evaluations Team AISI's Science of Evaluations team will conduct applied and foundational research focused on two areas at the core of our mission: (i) measuring existing frontier AI system capabilities and (ii) predicting the capabilities of a system before running an evaluation. Measurement of Capabilities: The goal is to develop and apply rigorous scientific techniques for the measurement of frontier AI system capabilities, so they are accurate, robust, and useful in decision making. This is a nascent area of research which supports one of AISI's core products: conducting tests of frontier AI systems and feeding back results, insights, and recommendations to model developers and policy makers. The team will be an independent voice on the quality of our testing reports and the limitations of our evaluations. You will collaborate closely with researchers and engineers from the workstreams who develop and run our evaluations, getting into the details of their key strengths and weaknesses, proposing improvements, and developing techniques to get the most out of our results. The key challenge is increasing the confidence in our claims about system capabilities, based on solid evidence and analysis. Directions we are exploring include: Running internal red teaming of testing exercises and adversarial collaborations with the evaluations teams, and developing "sanity checks" to ensure the claims made in our reports are as strong as possible. Example checks could include: performance as a function of context length, auditing areas with surprising model performance, checking for soft refusal performance issues, and efficient comparisons of system performance between pre-deployment and post-deployment testing. Running in-depth analyses of evaluations results to understand successes and failures and using these insights to create best practices for testing exercises. Developing our approach to uncertainty quantification and significance testing, increasing statistical power (given time and token constraints). Developing methods for inferring model capabilities across given domains from task or benchmark success rates, and assigning confidence levels to claims about capabilities. Predictive Evaluations: The goal is to develop approaches to estimate the capabilities of frontier AI systems on tasks or benchmarks, before they are run. Ideally, we would be able to do this at some point early in the training process of a new model, using information about the architecture, dataset, or training compute. This research aims to provide us with advance warning of models reaching a particular level of capability, where additional safety mitigations may need to be put in place. This work is complementary to both safety cases -an AISI foundational research effort-and AISI's general evaluations work. This topic is currently an area of active research, and we believe it is poised to develop rapidly. We are particularly interested in developing predictive evaluations for complex, long-horizon agent tasks, since we believe this will be the most important type of evaluation as AI capabilities advance. You will help develop this field of research, both by direct technical work and via collaborations with external experts, partner organizations, and policy makers. Across both focus areas, there will be significant scope to contribute to the overall vision and strategy of the science of evaluations team as an early hire. You'll receive coaching from your manager and mentorship from the research directors at AISI (including Geoffrey Irving and Yarin Gal), and work closely with talented Policy / Strategy leads and Research Engineers and Research Scientists. Responsibilities This role offers the opportunity to progress deep technical work at the frontier of AI safety and governance. Your work will include: Running internal red teaming of testing exercises and adversarial collaborations with the evaluations teams, and developing "sanity checks" to ensure the claims made in our reports are as strong as possible. Conducting in-depth analysis of evaluations methodology and results, diagnosing possible sources of uncertainty or bias, to improve our confidence in estimates of AI system capabilities. Improving the statistical analysis of evaluations results (e.g. model selection, hypothesis testing, significance testing, uncertainty quantification). Developing and implementing internal best-practices and protocols for evaluations and testing exercises. Staying well informed of the details and strengths and weaknesses of evaluations across domains in AISI and the state of the art in frontier AI evaluations research more broadly. Conducting research on predictive evaluations using the latest techniques from the published literature on AISI's internal evaluations, as well as conducting novel research to improve these techniques. Writing and editing scientific reports and other materials aimed at diverse audiences, focusing on synthesizing empirical results and recommendations to key decision-makers, ensuring high standards in clarity, precision, and style. Person Specification To set you up for success, we are looking for some of the following skills, experience and attitudes, but we are flexible in shaping the role to your background and expertise. Experience working within a world-leading team in machine learning or a related field (e.g. multiple first author publications at top-tier conferences). Strong track record of academic excellence (e.g. PhD in a technical field and/or spotlight papers at top-tier conferences). Comprehensive understanding of large language models (e.g. GPT-4). This includes both a broad understanding of the literature, hands-on experience with designing and running evaluations, scaling laws, fine-tuning, scaffolding, prompting. Broad experience in empirical research methodologies, potentially in fields outside of machine learning, and statistical analysis (T-shaped: some deep knowledge, lots of shallow knowledge, in e.g. experimental design, A/B testing, Bayesian inference, model selection, hypothesis testing, significance testing). Deeply care about methodological and statistical rigor, balanced with pragmatism, and willingness to get into the weeds. Experience with data visualization and presentation. Proven track record of excellent scientific writing and communication, with the ability to understand and communicate complex technical concepts for non-technical stakeholders and synthesize scientific results into compelling narratives. Motivated to conduct technical research with an emphasis on direct policy impact rather than exploring novel ideas. Have a sense of mission, urgency, and responsibility for success, demonstrating problem-solving abilities and preparedness to acquire any missing knowledge necessary to get the job done. Ability to work autonomously and in a self-directed way with high agency, thriving in a constantly changing environment and a steadily growing team. Bring your own voice and experience but also an eagerness to support your colleagues together with a willingness to do whatever is necessary for the team's success. Salary & Benefits We are hiring individuals at all ranges of seniority and experience within the research unit, and this advert allows you to apply for any of the roles within this range. We will discuss and calibrate with you as part of the process. The full range of salaries available is as follows: L4: £85,000 - £95,000 L5: £105,000 - £115,000 L6: £125,000 - £135,000 L7: £145,000 The Department for Science, Innovation and Technology offers a competitive mix of benefits including: A culture of flexible working, such as job sharing, homeworking and compressed hours. Automatic enrolment into the Civil Service Pension Scheme , with an average employer contribution of 27%. A minimum of 25 days of paid annual leave, increasing by 1 day per year up to a maximum of 30. An extensive range of learning & professional development opportunities, which all staff are actively encouraged to pursue. Access to a range of retail, travel and lifestyle employee discounts. The Department operates a discretionary hybrid working policy, which provides for a combination of working hours from your place of work and from your home in the UK. The current expectation for staff is to attend the office or non-home based location for 40-60% of the time over the accounting period. Selection Process In accordance with the Civil Service Commission rules, the following list contains all selection criteria for the interview process. The interview process may vary candidate to candidate, however, you should expect a typical process to include some technical proficiency tests, discussions with a cross-section of our team at AISI (including non-technical staff), conversations with your workstream lead. The process will culminate in a conversation with members of the senior team here at AISI. Candidates should expect to go through some or all of the following stages once an application has been submitted: Initial interview Technical take home test Second interview and review of take home test Third interview Final interview with members of the senior team Required Experience We select based on skills and experience regarding the following areas: . click apply for full job details
Feb 11, 2025
Full time
The Science of Evaluations Team AISI's Science of Evaluations team will conduct applied and foundational research focused on two areas at the core of our mission: (i) measuring existing frontier AI system capabilities and (ii) predicting the capabilities of a system before running an evaluation. Measurement of Capabilities: The goal is to develop and apply rigorous scientific techniques for the measurement of frontier AI system capabilities, so they are accurate, robust, and useful in decision making. This is a nascent area of research which supports one of AISI's core products: conducting tests of frontier AI systems and feeding back results, insights, and recommendations to model developers and policy makers. The team will be an independent voice on the quality of our testing reports and the limitations of our evaluations. You will collaborate closely with researchers and engineers from the workstreams who develop and run our evaluations, getting into the details of their key strengths and weaknesses, proposing improvements, and developing techniques to get the most out of our results. The key challenge is increasing the confidence in our claims about system capabilities, based on solid evidence and analysis. Directions we are exploring include: Running internal red teaming of testing exercises and adversarial collaborations with the evaluations teams, and developing "sanity checks" to ensure the claims made in our reports are as strong as possible. Example checks could include: performance as a function of context length, auditing areas with surprising model performance, checking for soft refusal performance issues, and efficient comparisons of system performance between pre-deployment and post-deployment testing. Running in-depth analyses of evaluations results to understand successes and failures and using these insights to create best practices for testing exercises. Developing our approach to uncertainty quantification and significance testing, increasing statistical power (given time and token constraints). Developing methods for inferring model capabilities across given domains from task or benchmark success rates, and assigning confidence levels to claims about capabilities. Predictive Evaluations: The goal is to develop approaches to estimate the capabilities of frontier AI systems on tasks or benchmarks, before they are run. Ideally, we would be able to do this at some point early in the training process of a new model, using information about the architecture, dataset, or training compute. This research aims to provide us with advance warning of models reaching a particular level of capability, where additional safety mitigations may need to be put in place. This work is complementary to both safety cases -an AISI foundational research effort-and AISI's general evaluations work. This topic is currently an area of active research, and we believe it is poised to develop rapidly. We are particularly interested in developing predictive evaluations for complex, long-horizon agent tasks, since we believe this will be the most important type of evaluation as AI capabilities advance. You will help develop this field of research, both by direct technical work and via collaborations with external experts, partner organizations, and policy makers. Across both focus areas, there will be significant scope to contribute to the overall vision and strategy of the science of evaluations team as an early hire. You'll receive coaching from your manager and mentorship from the research directors at AISI (including Geoffrey Irving and Yarin Gal), and work closely with talented Policy / Strategy leads and Research Engineers and Research Scientists. Responsibilities This role offers the opportunity to progress deep technical work at the frontier of AI safety and governance. Your work will include: Running internal red teaming of testing exercises and adversarial collaborations with the evaluations teams, and developing "sanity checks" to ensure the claims made in our reports are as strong as possible. Conducting in-depth analysis of evaluations methodology and results, diagnosing possible sources of uncertainty or bias, to improve our confidence in estimates of AI system capabilities. Improving the statistical analysis of evaluations results (e.g. model selection, hypothesis testing, significance testing, uncertainty quantification). Developing and implementing internal best-practices and protocols for evaluations and testing exercises. Staying well informed of the details and strengths and weaknesses of evaluations across domains in AISI and the state of the art in frontier AI evaluations research more broadly. Conducting research on predictive evaluations using the latest techniques from the published literature on AISI's internal evaluations, as well as conducting novel research to improve these techniques. Writing and editing scientific reports and other materials aimed at diverse audiences, focusing on synthesizing empirical results and recommendations to key decision-makers, ensuring high standards in clarity, precision, and style. Person Specification To set you up for success, we are looking for some of the following skills, experience and attitudes, but we are flexible in shaping the role to your background and expertise. Experience working within a world-leading team in machine learning or a related field (e.g. multiple first author publications at top-tier conferences). Strong track record of academic excellence (e.g. PhD in a technical field and/or spotlight papers at top-tier conferences). Comprehensive understanding of large language models (e.g. GPT-4). This includes both a broad understanding of the literature, hands-on experience with designing and running evaluations, scaling laws, fine-tuning, scaffolding, prompting. Broad experience in empirical research methodologies, potentially in fields outside of machine learning, and statistical analysis (T-shaped: some deep knowledge, lots of shallow knowledge, in e.g. experimental design, A/B testing, Bayesian inference, model selection, hypothesis testing, significance testing). Deeply care about methodological and statistical rigor, balanced with pragmatism, and willingness to get into the weeds. Experience with data visualization and presentation. Proven track record of excellent scientific writing and communication, with the ability to understand and communicate complex technical concepts for non-technical stakeholders and synthesize scientific results into compelling narratives. Motivated to conduct technical research with an emphasis on direct policy impact rather than exploring novel ideas. Have a sense of mission, urgency, and responsibility for success, demonstrating problem-solving abilities and preparedness to acquire any missing knowledge necessary to get the job done. Ability to work autonomously and in a self-directed way with high agency, thriving in a constantly changing environment and a steadily growing team. Bring your own voice and experience but also an eagerness to support your colleagues together with a willingness to do whatever is necessary for the team's success. Salary & Benefits We are hiring individuals at all ranges of seniority and experience within the research unit, and this advert allows you to apply for any of the roles within this range. We will discuss and calibrate with you as part of the process. The full range of salaries available is as follows: L4: £85,000 - £95,000 L5: £105,000 - £115,000 L6: £125,000 - £135,000 L7: £145,000 The Department for Science, Innovation and Technology offers a competitive mix of benefits including: A culture of flexible working, such as job sharing, homeworking and compressed hours. Automatic enrolment into the Civil Service Pension Scheme , with an average employer contribution of 27%. A minimum of 25 days of paid annual leave, increasing by 1 day per year up to a maximum of 30. An extensive range of learning & professional development opportunities, which all staff are actively encouraged to pursue. Access to a range of retail, travel and lifestyle employee discounts. The Department operates a discretionary hybrid working policy, which provides for a combination of working hours from your place of work and from your home in the UK. The current expectation for staff is to attend the office or non-home based location for 40-60% of the time over the accounting period. Selection Process In accordance with the Civil Service Commission rules, the following list contains all selection criteria for the interview process. The interview process may vary candidate to candidate, however, you should expect a typical process to include some technical proficiency tests, discussions with a cross-section of our team at AISI (including non-technical staff), conversations with your workstream lead. The process will culminate in a conversation with members of the senior team here at AISI. Candidates should expect to go through some or all of the following stages once an application has been submitted: Initial interview Technical take home test Second interview and review of take home test Third interview Final interview with members of the senior team Required Experience We select based on skills and experience regarding the following areas: . click apply for full job details
About the Chem/Bio Team The mission of the Chem/Bio team is to equip policymakers with an empirical understanding of safety-relevant AI capabilities, specifically at the intersection of AI with biology and chemistry. The team studies AI capabilities including (but not limited to) providing detailed instructions and (multimodal) troubleshooting for chemical or biological lab work, designing new or modified biological agents, and autonomously completing tasks on behalf of the user. The team works closely with other teams within the AI Safety Institute and wider UK government stakeholders, and external experts, to make sure our research has real-world impacts on AI safety and chemical and biological security through policy recommendations. Job Description We are looking for a research scientist to join the human behavioural research team within the Chem/Bio workstream. Working with an interdisciplinary group of research scientists & engineers to design and deliver research studies which measure how AI tools assist both non-experts and experts in their scientific capabilities. The research program combines online research studies, in-lab randomised controlled trials, and in-organisation projects. You will work closely with other team members and collaborators from government, academia, and industry, and receive coaching and mentorship from the human studies lead, as well as our workstream lead who is an established expert in biosecurity. Your day-to-day work may involve: Managing an online panel study, where participants complete various tasks on a web interface. Collaborating with the team and partners to design a randomised controlled trial measuring improvement in a wet lab task using an AI model. Working with other team members and partner institutions to design an intervention study looking at the impact of a specific tool on research success. Creating and validating behavioural questionnaire measures of experience using AI models in a research study. Summarizing experimental findings for a non-technical audience. Collaborating with strategy and delivery teams to share insights with government colleagues. Writing statistical analysis of experimental results, including data cleaning and detection of cheating/low effort responses. Setting up an online study to use an API to allow participants to interact with a hosted AI tool. Attending talks and journal clubs to keep up to date with the rapidly evolving AI landscape. Person specification You may be a good fit if you have some of the following skills, experience, and attitudes: Relevant experience in industry, open-source collectives, or academia in a field related to behavioral science, behavioral data science, randomized controlled trials, or human-computer interaction. Experience running behavioral randomized controlled trials online or in an organizational context. Experience in writing code that is scalable, robust, and easy to maintain, ideally in Python. Strong quantitative skills and a solid understanding of statistics in the context of research design and inference. Hands-on experience illustrating and writing up empirical research findings, ideally producing a deliverable such as a blog post or paper. Strong communication skills and the ability to engage both technical and non-technical audiences. Experience working in cross-functional teams, including both scientists and engineers. Motivated to conduct research that solves concrete open questions in governance and policy making. Ability to work autonomously and in a self-directed way with high agency, thriving in a constantly changing environment and a steadily growing team, while figuring out the best and most efficient ways to solve a particular problem. Have a sense of mission, urgency, and responsibility for success, demonstrating problem-solving abilities and preparedness to acquire any missing knowledge necessary to get the job done. The following are also nice-to-have: Experience in natural sciences, e.g. previous research placement in biology or chemistry or related undergraduate degree. Experience in evaluating human usage of frontier AI models. Web development skills, especially front-end. Track record of high impact, peer-reviewed publications. Additional Information Security Successful candidates must undergo a criminal record check. Successful candidates must meet the security requirements before they can be appointed. People working with government assets must complete baseline personnel security standard checks. Nationality Requirements UK nationals only. Working for the Civil Service The Civil Service Code sets out the standards of behaviour expected of civil servants. We recruit by merit on the basis of fair and open competition. The Civil Service embraces diversity and promotes equal opportunities. We run a Disability Confident Scheme (DCS) for candidates with disabilities who meet the minimum selection criteria and offer a Redeployment Interview Scheme to civil servants at risk of redundancy. Diversity and Inclusion The Civil Service is committed to attracting, retaining, and investing in talent wherever it is found. We value diversity and strive to create an inclusive environment where everyone can contribute to our mission.
Feb 11, 2025
Full time
About the Chem/Bio Team The mission of the Chem/Bio team is to equip policymakers with an empirical understanding of safety-relevant AI capabilities, specifically at the intersection of AI with biology and chemistry. The team studies AI capabilities including (but not limited to) providing detailed instructions and (multimodal) troubleshooting for chemical or biological lab work, designing new or modified biological agents, and autonomously completing tasks on behalf of the user. The team works closely with other teams within the AI Safety Institute and wider UK government stakeholders, and external experts, to make sure our research has real-world impacts on AI safety and chemical and biological security through policy recommendations. Job Description We are looking for a research scientist to join the human behavioural research team within the Chem/Bio workstream. Working with an interdisciplinary group of research scientists & engineers to design and deliver research studies which measure how AI tools assist both non-experts and experts in their scientific capabilities. The research program combines online research studies, in-lab randomised controlled trials, and in-organisation projects. You will work closely with other team members and collaborators from government, academia, and industry, and receive coaching and mentorship from the human studies lead, as well as our workstream lead who is an established expert in biosecurity. Your day-to-day work may involve: Managing an online panel study, where participants complete various tasks on a web interface. Collaborating with the team and partners to design a randomised controlled trial measuring improvement in a wet lab task using an AI model. Working with other team members and partner institutions to design an intervention study looking at the impact of a specific tool on research success. Creating and validating behavioural questionnaire measures of experience using AI models in a research study. Summarizing experimental findings for a non-technical audience. Collaborating with strategy and delivery teams to share insights with government colleagues. Writing statistical analysis of experimental results, including data cleaning and detection of cheating/low effort responses. Setting up an online study to use an API to allow participants to interact with a hosted AI tool. Attending talks and journal clubs to keep up to date with the rapidly evolving AI landscape. Person specification You may be a good fit if you have some of the following skills, experience, and attitudes: Relevant experience in industry, open-source collectives, or academia in a field related to behavioral science, behavioral data science, randomized controlled trials, or human-computer interaction. Experience running behavioral randomized controlled trials online or in an organizational context. Experience in writing code that is scalable, robust, and easy to maintain, ideally in Python. Strong quantitative skills and a solid understanding of statistics in the context of research design and inference. Hands-on experience illustrating and writing up empirical research findings, ideally producing a deliverable such as a blog post or paper. Strong communication skills and the ability to engage both technical and non-technical audiences. Experience working in cross-functional teams, including both scientists and engineers. Motivated to conduct research that solves concrete open questions in governance and policy making. Ability to work autonomously and in a self-directed way with high agency, thriving in a constantly changing environment and a steadily growing team, while figuring out the best and most efficient ways to solve a particular problem. Have a sense of mission, urgency, and responsibility for success, demonstrating problem-solving abilities and preparedness to acquire any missing knowledge necessary to get the job done. The following are also nice-to-have: Experience in natural sciences, e.g. previous research placement in biology or chemistry or related undergraduate degree. Experience in evaluating human usage of frontier AI models. Web development skills, especially front-end. Track record of high impact, peer-reviewed publications. Additional Information Security Successful candidates must undergo a criminal record check. Successful candidates must meet the security requirements before they can be appointed. People working with government assets must complete baseline personnel security standard checks. Nationality Requirements UK nationals only. Working for the Civil Service The Civil Service Code sets out the standards of behaviour expected of civil servants. We recruit by merit on the basis of fair and open competition. The Civil Service embraces diversity and promotes equal opportunities. We run a Disability Confident Scheme (DCS) for candidates with disabilities who meet the minimum selection criteria and offer a Redeployment Interview Scheme to civil servants at risk of redundancy. Diversity and Inclusion The Civil Service is committed to attracting, retaining, and investing in talent wherever it is found. We value diversity and strive to create an inclusive environment where everyone can contribute to our mission.