AI Development LifeCycle For TPMs

To incorporate AI in your system flow, you extend the Software Development Life Cycle(SDLC) based on what is being built. The steps in SDLC are still needed, however AI-DLC (AI-Development Life Cycle) can introduce entirely new sources of uncertainty around data quality, model behavior, experimentation, probabilistic outputs, training, evaluation, inference, cost and continuous monitoring needs to detect performance degradation after launch. This understanding is helpful for technical program managers (TPMs) and engineering managers, who are kicking off AI programs or interviewing and need to understand how the SDLC has changed with the introduction of AI.

What is The AI Development Life Cycle (AI-DLC)

The AI-DLC lifecycle reuses many of the phases of the SDLC and adds to it based on the type of AI solution proposed. A practical way to understand AI development is to break it into eight phases. 

• Phase 1: Define the Problem
• Phase 2: Data Generation & Collection
• Phase 3: Data Preparation & Feature Engineering
• Phase 4: Model Development & Training
• Phase 5: Evaluation
• Phase 6: Deployment
• Phase 7: Inference & Application Integration
• Phase 8: Monitoring and Feedback Loops

TPMs still need to drive a systematic approach to prioritize, remove bottlenecks, mitigate risks while keeping cross-functional teams aligned to the goal and achieve the expected business outcome. What changed with AI is the experiment and fail-fast cycle got accelerated and in many cases can be achieved with lower engineering/specialized expert involvement to prototype, learn quickly and decide on feasibility. The barrier to prototyping has lowered, so you can make sure the product is viable before investing the engineering effort to make it production ready(scalable, extensible, hardened, secure, monitored).

The introduction of AI into the development lifecycle also introduces multiple iteration and evaluation cycles to get it to the expected level of performance as the outputs are non-deterministic. This adds a lot of changes into the project plan that TPMs have to stay on top of, assess and communicate. The AI costs and performance metrics that gate each phase exit also change based on the design and go beyond traditional SDLC.

We dive into each of the 8 phases of the AI development life cycle in this article and how the program might look different based on how you use AI. 

I came up with this mental model on AI-DLC based on how I managed diverse teams across roles in Amazon where I owned software, hardware, computer vision and machine learning teams to build AI models(neural networks, RAG) to help robots understand items that they needed to pick up and later when dealing with influencer generated content to review, classify, enhance, and recommend. This is what I learned from real-world experience deploying production AI solutions at scale.

Chatting to my colleagues in Amazon, Meta, DoorDash, Google, who do a mix of building the AI models and using AI tools, we found that it’s fairly the same flow though the division of work and boundaries between science and engineering differ per company and team. Would love to hear your feedback on how your company handles AI programs.

AI-DLC Changes Based On The Product

The AI development lifecycle phases that apply to your project will vary based on the product being built. If your product utilizes existing AI models/tools and incorporates them into your system, such as using a pre-trained model that is hosted on cloud services, then your software development lifecycle stays fairly the same if your model is not self-learning nor being retrained and redeployed often. It still needs the deployment, monitoring and the inference layer to take its output and do something useful using that result in order to meet your target outcome.

If your product is the AI model itself, the development lifecycle will heavily focus on data collection, model development, training and iterative loops to get to the point where the model performance meets the defined validation criteria to make it suitable for deployment. This type of program will also need the pieces for deployment, inference, and monitoring.

Let’s deep dive into these two AI product scenarios as they will help understanding the AI program lifecycle. Please note that one is not greater than the other, but that each one needs managers to focus on different areas and timelines differ based on this as well.

Scenario A: An application that uses an existing AI model/ tool

Imagine an e-commerce company building an internal support assistant using a hosted large language model (LLM).

The engineering team may build the user interface, add authentication, retrieve additional information, construct a prompt or a model request, call a hosted model API, then process or transform the response received from the model, apply additional business logic before returning the response to the user. They will monitor latency, cost, errors, quality etc.

The underlying foundation model may be developed and operated by another team or another company.

In this case, much of the traditional SDLC remains intact. The application still has source code, APIs, CI/CD pipelines, infrastructure, tests, deployments, observability, incident management, and release processes.

A traditional API usually has a relatively deterministic contract where the system receives an input to a service API, looks up data in its database, uses some algorithmic/logical processing and generates a response that is sent back to the user.

However, an LLM-powered application flow tends to be that a system receives an input to a service API, the system then retrieves additional context and generates a prompt, with this additional context that it passes to the AI model. The model then generates an output, which is then validated or checked against the guardrails before passing it on to another software function that processes it further with business logic to generate a response that is passed back to the user. The model is a component in a larger system.  If you are building an AI agent, then orchestration, tool use, state management, evaluation, and safety may become central engineering problems.

In this scenario, the team integrated an AI model rather than building one. The product is AI-powered, not an AI product.

The complex engineering work in this scenario typically involves data retrieval, context construction, orchestration, evaluation, latency, reliability, security, permissions, observability, and integrating model output into deterministic business workflows.

Scenario B: The AI model/tool is the product

Now consider a company building a recommendation model, fraud detection model, search-ranking model, speech recognition system, or foundation model. The lifecycle changes substantially.

The team may need to define the prediction or generation task, collect and process training data, establish data quality requirements, create labels or other training signals, design model architectures, train models on large compute clusters, evaluate model performance, analyze failures, tune the model, compare models against baselines, validate safety and robustness, package and deploy the model to integrate it with an inference service API, monitor its performance in production, audit it and retrain or update it as needed.

This creates a loop with many iterations rather than a simple linear pipeline. The key insight for technical managers is that training is not the end of AI development. Training produces a model artifact, but a production AI system requires much more. When all of this is done, the steps in Scenario A are still needed to solve real-world use cases using this model. 

Think of Scenarios A and B as two layers of the same stack, often owned by different teams or companies. When planning your project, start with this question: Are we building software that uses an AI model, or are we building and operating the AI model itself?

The answer will determine the development lifecycle, the teams involved, the infrastructure required, the metrics you track, and the risks that must be managed.

In most big tech companies, AI capabilities can be exposed as reusable platform services, allowing application teams to consume models without owning the entire model-development lifecycle.

That creates an organizational distinction between:

• Teams building models.
• Teams operating ML infrastructure.
• Teams building AI platforms.
• Teams integrating AI into customer-facing products.

The same company can therefore contain multiple AI-SDLCs.

AI Program Lifecycle

AI-DLC Phase 1: Define the Problem 

The problem is the business problem that you are solving for or a gap you are filling. This will give you your north star success metric. It could be accurately recommending a set of creators in your feed that match your interests perfectly, leading you to click on their profile and follow them or engage with their content. It could be improving efficiency of a fulfillment process by forecasting and pre-filling the inventory needed for a holiday event accurately or improving the quality of a social media video that you created by auto-removing the background noise while keeping your voice clear and suggesting a soundtrack that matches the mood of the scene in the video. The problem can be a consumer product, a process optimization or efficiency improvement. 

Ideally, AI should not be part of the business problem statement as it is the ‘how’ you will solve the problem, not ‘why’ you are working on this problem nor ‘what’ the outcome you expect to achieve. The few exceptions are if your goal is an adoption of AI scenario. For efficiency improvements, even if behind the scenes you want to use AI to reduce the labor needed, the problem statement should be framed in terms of reduced cost or human labor hours.

Establish success metrics and ROI

You must be able to measure the outcome to understand whether you moved the needle. This is your success metric(s). Did you increase click through rate(CTR), did you increase engagements in terms of views, time spent on the app etc. Metrics also help set the baseline for the ROI expected from the program, especially for AI where costs (GPU, tokens) can scale quickly.

Instead of a goal to:

 “ Use AI to improve customer support”, 

A SMART goal would be:

“Reduce average time spent by support agents searching internal documentation by 30%, while maintaining answer accuracy above an agreed threshold and ensuring responses only use information the employee is authorized to access.”

This phase is not AI specific, this is the basis of any good program. So, this is similar to the SDLC Planning phase. In both lifecycles, we state the business problem, define the goal and the success metrics in terms of revenue, adoption, accuracy, efficiency improvements, engagement and so on. The usual system performance metrics used in SDLC programs are latency, availability, throughput , uptime etc. For AI programs, there will be additional metrics such as Accuracy, Queries per Second(QPS), Precision, Recall, F1-Score, Cost per Query/Token usage etc. This is particularly important for AI because model quality is rarely represented by a single metric. 

Read the indepth blog AI metrics for Managers and check out the AI toolkit here.

Document Requirements

With AI, you now have the option to prototype, learn quickly, fail-fast to make sure the idea is feasible and fine tune your program needs based on requirements missed or constraints you had not thought of before. We also document the requirements and constraints, generating a BRD/PRD (business requirements document/product requirements document) from this step and generate a proposed high level solution at the systems-level. This matches the SDLC Requirements Gathering and Design phases. For simplicity, we club the first 3 phases of SDLC( problem definition, requirements gathering, design) all together into a single phase for AI programs.

For example, suppose a team wants to build a model that classifies product images into packaging categories to help automate fulfillment operations. The team must agree on the categories, define what counts as a correct prediction, determine the cost of classification errors, and establish a baseline against which the model will be measured.

Determine how you will use AI

What changes between AI-DLC and SDLC is the number of phases and where uncertainty is introduced in the development lifecycle. To nail that down, determine if you are building software that uses an AI model, or building and operating the AI model itself. The answer will determine the development lifecycle, the teams involved, the infrastructure required, the metrics you track, and the risks that must be managed. You should also assess whether ML is appropriate or a rules-based solution would suffice.

AI-DLC Phase 2: Data Generation & Collection 

If you are building the AI model/tool itself, this is a step in your program management lifecycle. If you are using an existing or pre-trained model, this step might be small where you are feeding it data to fine-tune your model or skipped entirely.

In my experience, this has sometimes been the most time-consuming step as we have to find data sources, collect the data, assess it to see how useful it is, fine tune our data requirements based on that assessment, collect more data and see how much data we can collect and what breadth of data we can collect. This is a basic requirement for model development that you need large volume and diverse data to train and validate your model. My experience of classification models made it clear we were missing/low on data for certain classes and had to wait a longer period of time to collect the data so that we had enough data points, or use synthetic data. 

All the options you choose in this step come with their pros and cons, so managers working in these spaces need to understand the risks, pros/cons of each option to help drive decision making and what to expect further down the line based on the decision made. This is why building your own AI model requires a much deeper understanding of machine learning, RAG(retrieval augmented generation), neural networks and related concepts than orchestrating AI agents. This experience is not common as very few companies build their own AI models like in Scenario A above.

As a TPM, you will also need to understand and oversee data collection, and the related process flows that generate the data. This could involve setting up data collection points using hardware(sensors, cameras, barcode readers, scanners, etc.), or monitoring user interactions (clicks, views, transactions, search queries), and transmitting that data to your science/engineering team. Depending on the frequency of data generation and the type of data(text, images, scans, video, audio), you will be dealing with large quantities of data. In a factory or hospital environments, the restrictions of bandwidth, upload speed, hardware installation can make this step complicated. You will need to work with your regulatory and legal teams to ensure you have permission to use the data.

In parallel, you have to work with engineering to make sure they understand and are planning for data collection to decide on how to feed production and training data to models, process the data to prepare it for consumption, decisions on model hosting, data pipelines for training, evals, audit, feedback loops, human in the loop verification, inference etc. to work on the overall system design estimates. Remember, the quality of the data used will determine the reliability of your model’s output. You will need to work with product, science, engineering to align on the definition of quality for your use-case.

AI-DLC Phase 3: Data Preparation and Feature Engineering

We usually do not feed raw data into a model. You will have to remove duplicates, check for stale data, version the data based on your labelling requirements. The collected data has to be labelled based on the input and outputs of the model. This can take a large amount of time as a lot of labelling is human based and usually requires the best of 3 labels to make sure the label is accurate. The more specific the labelling requirements are and how well defined the edge cases are will make the labelling effort more accurate. There are ways to use models for labelling as well, but there needs to be auditing to ensure it is accurate as this will form the training and validation datasets for the model you are building. Data needs to be scrubbed to ensure anonymity, privacy, security and other geographical/legal restrictions are complied with.

As part of data preparation you need to define and segregate your training datasets and your golden /validation datasets. As a TPM, look for classes, geographies, or other areas where diversity of your dataset is compromised. Understand and assess removal of outliers in your data and how that affects what you are building. To understand and achieve this you will work closely with scientists ,ML engineers and your labelling team in this step. 

Not all data signals are equal. For example, a purchase action might be a stronger signal of preference compared to a view action. A click might indicate interest, but not necessarily preference or satisfaction. This is where product and ML/Science intersect to decide on the best signals for the use-case.

In traditional machine learning, features might need to be created. Usually statistical methods are used to create features, based on domain knowledge. 

Modern deep-learning systems often learn representations directly from raw or minimally processed data. LLM systems may require tokenization, document processing, chunking, metadata extraction, embedding generation, retrieval indexes, or instruction datasets.

AI-DLC Phase 4: Model Development & Training

Training is where the model learns parameters from the data. The exact training process will depend on the type of AI system. In classical ML, training methodologies include linear regression, gradient boosted models, random forests, decision trees. In deep learning, you would train neural networks, transformers, vision models, recommendation models, speech models. For foundation models, training will involve massive datasets, specialized accelerators etc. My experience has been in classical ML and Deep Learning. 

Model development and training is usually carried out by ML engineers or scientists. With the advancement in tooling and using off the shelf models, this is changing as engineers are able to get more hands on with model development. The scientists/ML engineers turn datasets into a candidate model. They will likely experiment with model architectures, features, parameters, and training configurations to improve performance.

If you are consuming an external foundation model, you may skip training from scratch and only fine-tune or prompt-engineer it. If you are building a recommendation model, training and experimentation might dominate the time spent in development. If you are building a foundation model, distributed training and data infrastructure become enormous parts of the lifecycle.

You need to understand how to improve the model using fine-tuning, prompting, retrieval, feedback loops and so on. The phase might take multiple iterations.

The TPM and engineering manager need to watch out for costs in this step as GPU usage can add up. There are also risks like overfitting( model training metrics improve but performance on unseen data declines) and maintaining good versioning of models and datasets.

AI-DLC Phase 5: Evaluation and Validation

This is one of the biggest differences between traditional software and AI systems as it introduces a lot of variability into the program and can lead to multiple iterations. In a software architecture, we evaluate the system by testing if the system/function returns the expected value. However, in an AI system, the test is more vague as we usually test for how often a model produces an acceptable result across a representative set of inputs.

This phase determines whether the model actually solves the intended problem as established by the targets defined and whether its failure modes are acceptable. Offline evaluation usually begins with held-out datasets. Depending on the use case, validation may also involve human review, adversarial testing, simulations and online experiments.

Based on the model, the metrics used will differ. To meet the needs of this phase, you will need to create evaluation datasets and evaluation pipelines alongside the model. Evaluation establishes whether the model is fit for the purpose it was built for.

Common metrics used for different models

For a classification model:

• Precision
• Recall
• F1
• ROC-AUC

For ranking:

• NDCG
• MAP
• Precision@K
• Recall@K

For recommendation:

• Click-through rate
• Conversion rate
• Engagement
• Revenue
• Retention

For generative AI:

• Task success
• Groundedness
• Factuality
• Relevance
• Helpfulness
• Safety
• Refusal behavior
• Hallucination rate
• Human preference
• Prompt injection resilience

Thresholds must be specific to the use case. A fraud detection model may prioritize recall subject to an acceptable false-positive rate. A medical decision-support model may require stringent sensitivity and safety review. A recommendation model will likely be evaluated on relevance, engagement, and downstream business outcomes.

So for each model, decide on the metrics needed based on the use-case as well as the model type to check how well a candidate model performs on defined datasets and tasks.

For validation, we establish whether the model, with its known limitations, satisfies the requirements for its intended use. This involves confirming that evaluation results meet pre-agreed acceptance thresholds, testing critical edge cases and high-impact failure modes. You will also want to verify performance across relevant data segments, assess robustness, safety, fairness, and calibration where applicable. The required technical, product, security, compliance approvals are obtained in this phase.

Model evaluation and validation will usually heavily involve the ML engineering or applied science teams.

AI-DLC Phase 6: Deployment

Deployment moves the model-backed system into an environment where real users or production workflows can access it. It exposes the model to production traffic. This includes more than publishing a model file. It may require application changes, API contracts, infrastructure configuration, data pipelines, feature transformations, security policies, and monitoring. This is a release and infrastructure activity.

We do not usually move directly from a successful test to a 100% production rollout. There are usually phased launches planned from staging environment to production environment to reduce the blast radius, with rollbacks as needed, similar to SDLC launches. Some validation occurs after deployment to a staging environment, and production validation may continue during a canary rollout. A shadow deployment can also provide production-like evidence before users are exposed to the model’s outputs. It is important to understand that deploying a model version does not automatically authorize full production use.

This phase is where any differences between the training setup and production setup will surface, or hidden operational dependencies that can impact model performance. A model can pass validation and still fail during deployment because of configuration errors, missing dependencies, incompatible APIs, insufficient capacity, or production data differences.

AI-DLC Phase 7: Inference

Inference is when the trained model acts on new inputs to make predictions, which are then usable in the product flow. The model is packaged along with its runtime dependencies and hosted on the serving architecture. Engineering has to work to optimize latency, throughput, consider batching, caching, and other techniques. The hardware needs to autoscale to meet capacity needs alongside retries, timeouts, and graceful failure handling. Logging is also necessary at this step and integration with downstream services. The setup is stress tested to understand production traffic and estimate costs.

There will also be software components that form the entire system to create interfaces for the data output storage, rendering or additional processing to meet the demands of the specific use-case by processing the output from the model and making additional decisions, as well as feeding additional context to the model as inputs.

This phase is where a lot of the engineering issues are uncovered. For example, a model may meet its accuracy target but require too much GPU memory, take 1.5 seconds to return a result when the product requires 200 milliseconds, or cost more per request than the business can support.

AI-DLC Phase 8: Monitoring & Feedback/Audit

Previously for software application, the main metrics we focused on were availability, latency, error rate, traffic, throughput. In parallel, we care about product metrics such as engagement, conversion, clicks, user satisfaction, time taken, revenue and other metrics based on your business problem. AI systems still need all these metrics as they are integrated into a larger system and are solving a business problem, but they also need model-specific monitoring for both hosting and inference. For example, an LLM API could have 99.99% availability while the application’s answer quality has deteriorated because the underlying knowledge base changed. Similarly, a model can have excellent offline metrics but fail to improve the product. 

Monitoring tells us whether the model continues to work in the real world. Some AI based solution metrics to consider are:

Infrastructure metrics

• CPU/GPU utilization
• Memory
• Network
• Latency
• Throughput
• Availability
• Cost

Model metrics

• Accuracy
• Precision/recall
• Distribution changes
• Prediction confidence
• Evaluation scores

Generative AI metrics

• Hallucination rate
• Retrieval quality
• Tool-call success
• Safety violations
• Refusal rate
• Token consumption
• Cost per request

The model itself is continuously evaluated based on feedback to understand how it is performing. This feedback and audit loop ensures the team is alerted to model degradation or model drift. As a result of these metrics, sometimes your model will need to be retrained or retired. 

Model Lifecycle Management

A TPM needs to understand how models are versioned, evaluated, promoted, rolled back and eventually deprecated/retired. The feedback and audit loops built into production AI/ML systems are constantly assessing models to check if their performance meets the set criteria. When there is a deviation, it signals the need for further deep dive and possibly model replacement. A model lifecycle usually consists of these states: Experiment, Candidate, Validated, Staged, Production, Deprecated. All of this comes under the umbrella of MLOps in the program.

Where TPMs and Engineering Managers Add Value

AI development creates cross-functional dependencies that are often more complex than those in conventional software projects. A TPM will need to coordinate between Product, ML Engineering, Science/Research, Data Engineering , Infrastructure, Security / Privacy /Legal, Engineering, ML Operations and hardware/vendors. A TPM in the AI space needs to understand the AI lifecycle and how their program fits into it to make high judgement calls on what applies, what to measure and what the risks are. 

Models are typically built by applied scientists and ML engineers. They are deciding the number of neural network layers, parameters, weights, biases, work on the model graph etc. Machine Learning Engineer(MLE) is an overloaded term as in different companies/teams within a company the role expectation is different depending on how deep they go into model training, debugging, deployment etc. I meet many engineers who call themselves MLE, but are just building AI agents and orchestration or are working on data pipelines for models, but never touch the actual model itself. This would fall more into the MLOps realm for the latter. What I am trying to say here is that do not get misled by titles, understand what folks actually do to know how they fit into the bigger lifecycle of the program. This is useful when interviewing for a role, explaining your own role or when taking on a new team and you need to map out who does what to understand your stakeholders.

The TPM job is to ensure that each phase produces evidence that the system is ready for the next one. A model that trains successfully is not necessarily useful. A model that performs well in evaluation is not necessarily safe or economical in production. A model that deploys successfully is not necessarily operating reliably.

For TPMs, program coordination alone isn’t enough. You need a deeper understanding of the AI lifecycle, the metrics, pitfalls and experience with building and deploying models to be able to make these high judgement calls. Work with ML engineers, scientists to learn how they think, what their challenges are, what metrics they are using, how they are selecting weights, adjusting biases, parameters and slowly you will become more familiar with the terms and their application.

In an AI project you have to understand and be prepared to handle roadblocks/dependencies such as:

• “The model cannot be evaluated until the data pipeline produces the correct training set.”
• “The model cannot be deployed until inference latency meets the application’s SLA.”
• “The model cannot be released until the evaluation set demonstrates acceptable performance across critical cohorts.”
• “The application cannot launch until retrieval quality and access controls are validated.”

These dependencies are often where technical program management becomes particularly valuable if the TPM understands what each of these scenarios entails, which teams to engage and what the pros and cons of each decision made might be alongside the cost and timeline constraints. 

For technical leaders, the goal is not to become a research scientist. It is to understand where the AI lifecycle differs from traditional software, what the dependencies are, which metrics matter, where projects fail, and how data, models, infrastructure, and product engineering come together to deliver a production system.

Subscribe to our newsletter, download the AI toolkit to get familiar with AI metrics and terminology and read our other articles on AI for TPMs and Managers.