Microsoft 365 Full-Stack Developer's Recap to Ignite 2024

Conferences such as Microsoft’s Inspire, Ignite, and the recent Build Conference serve as platforms where Microsoft displays their latest and upcoming projects. The last Ignite 2024 conference in Chicago was no exception!

A current challenge is trying to keep up with all the news, let alone news relevant to your job. It’s a demanding task to filter out what’s pertinent to your role before you can even start catching up. However, if you’re a full-stack Microsoft 365 developer, this article might be helpful.

After the Build 2024 Conference in May earlier this year, I published my Microsoft 365 Full-Stack Developer’s Recap to Build 2024 article for Microsoft 365 full-stack developers (with an associated YouTube playlist of all the keynotes, breakout sessions, and pre-recorded content from the event):

Microsoft 365 Full-Stack Developer's Recap to Build 2024

I spent over 40+ hours watching 30+ sessions, keynotes, & reading announcements to gather what Microsoft 365 full-stack developers should know from Build 2024.

https://www.voitanos.io/blog/microsoft-365-fullstack-developer-recap-microsoft-build-2024/

Well, just like the Build 2024 Conference, I dedicated over 40 hours to watching more than 40 sessions and keynotes, reading announcements, and asking questions to gather what I believe full-stack developers in the Microsoft 365 space should know.

My favorite resources creating this summary are the Microsoft Build 2024 Book of News & session recordings. I’ve created a playlist on my YouTube channel of all the sessions that Microsoft 365 full-stack developers should check out: Microsoft Ignite 2024 - for Microsoft Full-Stack Developers:

This article is a summary of that information, along with links to announcements, session recordings, and documentation where relevant for further exploration.

Let’s dive in.

Microsoft 365 Copilot Agents

There wasn’t too much new stuff shared with declarative agents for developers at Ignite - not surprising given all the news in September & October. But there were a few new things we learned, specifically around capabilities with improvements in the new v1.2 agent manifest:

• WebSearch: This existing capability can now include up to 4 site filters to restrict the sites the agent is allowed to search.

• GraphicArt: Your agent can be used to create images and art based on the text input from the user using the Image generator.

• CodeInterpreter: A new capability will allow users to trigger the agent to execute code from a response. In a demo, Microsoft showed how this was used to take data from Microsoft Graph to generate a pie chart using Python.

  See the Code Interpreter in action in this 8-minute video:

• SharePoint & OneDrive: We now have details on the file types and sizes that the agent can use in its data collection. The agent has access to the file types DOCX, PDF, and PPTX up to 512 MB and DOC, TXT, PPT, and HTML files up to 150 MB.

Learn more:

• Declarative agent manifest reference - version 1.2
• Add Agent Capabilities · Image generator
• BRK167 · Developers guide to building your own agents

Microsoft Copilot Studio

Microsoft is making significant enhancements to Copilot Studio’s integration with Azure AI services. For instance:

Use Azure AI capabilities directly in Copilot Studio

Copilot Studio now enables makers to tap into custom Azure AI Search indexes as knowledge sources for custom RAG scenarios. Plus, you can now access over 1,800 prebuilt or custom-built Azure AI models right from within Copilot Studio.

New multi-modal capabilities allow agents to interact beyond text

We’re seeing a major expansion in how users can interact with agents. Organizations can now implement generative AI in their voice solutions, including IVR systems, and deploy agents that users can talk to in various applications. What’s particularly cool is that these agents aren’t limited to voice - users can also upload images and ask the agent to analyze them and answer questions about what it sees.

Use custom Azure AI capabilities directly in Copilot Studio

The integration between Copilot Studio and Azure AI Foundry is getting tighter, opening up possibilities for more sophisticated custom scenarios. This is a big win for Copilot Studio customers who’ve already invested in Azure AI Foundry’s offerings.

Here’s a practical example: many makers have substantial datasets in Azure AI that can now serve as knowledge sources directly in Copilot Studio. They can run search operations on vectorized indexes built in Azure AI Search to access unstructured data. With the ability to bring this data and custom RAG into Copilot Studio, organizations can create agents that are deeply grounded in their business data, delivering more accurate and contextually relevant responses.

Learn more:

• Copilot Studio enhancements: knowledge improvements, Azure AI integration, and more
• BRK165 · What’s new with Copilot Studio and agents

Copilot Analytics - part of the Copilot Control System

This is more of an IT Pro/adoption tool, but one I’m curious to learn more about as a developer. As an agent developer, both declarative agents & custom engine agents, I’d love to see how I can get analytics on how customers are using the agents I create.

Copilot Analytics includes…

• The Copilot dashboard, now generally available, offers an out-of-the-box solution for tracking Copilot readiness, adoption, impact, and learning.
• Microsoft 365 admin center reporting tools, now generally available, provide IT professionals with insights into adoption and usage trends along with suggested actions.
• Viva Insights, currently in public preview, will included in Microsoft 365 Copilot at no additional cost starting in early 2025, is a comprehensive toolset for measuring productivity and business outcomes, featuring customizable report templates and the new Copilot Business Impact Report for analyzing business KPIs.

Learn more:

• Introducing Copilot Analytics to measure AI impact on your business
• BRK283 · Reimagining employee experience with AI and agents
• OD805 · Measure AI adoption and value with Copilot Dashboard

Microsoft 365 Agents SDK

The Microsoft 365 Agent SDK, currently in public preview, simplifies the development of comprehensive, multichannel agents. These agents work with Microsoft 365, Microsoft Teams, Copilot Studio, Webchat, and third-party services like Facebook Messenger, Slack, and Twilio. The SDK provides essential tools for creating agents that manage user interactions, process requests, generate responses, and collaborate with other agents. It serves as a complete framework for enterprise-grade agents, integrating seamlessly with the Azure AI SDK, Semantic Kernel, and third-party AI solutions.

The SDK launches with C# support (Node.js and Python coming soon) and enables developers to build enterprise-grade, scalable, multi-channel agents using code. Developers can select their preferred AI services like Azure AI Foundry, add an orchestration layer like Semantic Kernel, and deploy across multiple channels—including web, Microsoft Teams, Microsoft 365 Copilot, third-party channels, and custom clients.

Sound familiar… like the Bot Framework SDK? It should… The Agents SDK has evolved from Bot Framework v4, which focused on topics, dialogs, and messages in Conversational AI. While the Bot Framework focused on conversational AI through topics, dialogs, and messages, today’s industry demands have shifted toward Generative AI, enterprise knowledge integration, and orchestrated actions within conversations. The Agents SDK meets these modern needs by unifying conversational agent creation, conversation management, orchestration, and connectivity to various AI services and clients, including third-party solutions.

One intriguing aspect Microsoft discussed is the ability to create agents in Copilot Studio as a power user, then extend them with developer tools using the SDK—and vice versa. This shows Microsoft has learned from past challenges. SharePoint users will remember having to choose between SharePoint Designer or Visual Studio for workflows, with no way to switch between them once development began.

The SDK’s flexible component architecture lets developers build agents tailored to their specific needs for user experience, orchestration, and intelligence. Teams can create scenarios where multiple agents—built using Azure AI Foundry, Microsoft Copilot Studio, or third-party technologies—work together to answer questions and solve problems.

🤔 Why the Agents SDK matters to full-stack developers

How will this new Microsoft 365 Agents SDK impact you as a Microsoft 365 full-stack developer?

At first, Microsoft empowered Microsoft Teams app developers to create conversational bots for Microsoft Teams using the Bot Framework SDK. This approach worked & had modest improvements for many years, but in 2023 with the spread of generative AI & LLMs, Microsoft saw an opportunity: Generative AI & LLMs could solve one of the challenges developers faced in creating bots was deducing the user’s intent from their prompt to the bot.

Let’s take a step back…

Initially developers would have to write complex code to determine what the user’s intent was to try to match it up to business logic in the bot. This was improved with language understanding (LUIS) API and associated service from Microsoft Azure (now deprecated and scheduled for retirement on October 1, 2025).

Then, last year, Microsoft introduced the Teams AI Library, built on top of the Bot Framework to make it easier to interact with LLMs hosted by OpenAI or Azure OpenAI. One of the biggest benefits this brought was the ability to (1) deduce the user’s intent with an LLM and then (2) map that intent to a skill (action) that you then handle in your code. The Teams AI Library handles the communication with the LLM to get an action plan based on skills you’ve taught your bot. When the LLM identifies an action to perform based on the user’s intent, it returns a plan back to the Teams AI Library that will call your code (ie: the business logic) defined in your bot using the Teams AI Library’s API.

But, developers were now left with two ways to create bots: use the less powerful Bot Framework SDK or the Teams AI Library.

What do you think? As a Microsoft 365 full-stack developer, do you agree/disagree or just have a different conclusion? Share it in the comments below 👇

Extend or integrate with Copilot Studio agents

The Microsoft 365 Agents SDK enhances Copilot Studio in two ways:

1. Developers can extend existing Copilot Studio agents with new features through skills—add-ons that expand agent capabilities.
2. Developers can write code that connects to Copilot Studio agents, accessing its full feature set, including over 1,000 connectors. This allows teams to work independently—some using Copilot Studio, others writing code—and combine their work seamlessly.

Learn more:

• Microsoft 365 Agents SDK GitHub repo
• Microsoft 365 Agents SDK (GitHub repo)
• Microsoft 365 Agents SDK (docs)
• BRK167 · Developers guide to building your own agents
• BRK165 · What’s new with Copilot Studio and agents
• BRK115 · Coding the future of AI
• BRK126 · Productive AI with Semantic Kernel
• BRK102 · Introducing Azure AI Foundry Agent Service to scale your AI agents

Azure AI Foundry

Announced at Ignite 2024, is a platform that enables organizations to design, customize, and manage next-generation AI applications and agents at scale by integrating existing Azure AI models and tools with new capabilities. It provides access through an SDK and portal experience, connecting with popular developer tools like GitHub, Visual Studio, and Copilot Studio, while supporting efficient, cost-effective development and management of AI safety, reliability, and performance.

🤔 Why the Azure AI Foundry & services matter to full-stack developers

You might wonder why I’ve dedicated such a big section to an Azure service, or suite of services, for Microsoft 365 developers. Good question - it’s simple!

As I covered in my Build 2024 recap & Microsoft Copilot GA articles…

Microsoft 365 Copilot Developer GA - All you need to know!

In this article, get the latest on the developer extensibility story's GA milestone with Microsoft 365 Copilot. Learn everything from GA & what's next!

https://www.voitanos.io/blog/microsoft-365-copilot-generally-available-october-2024/

Microsoft 365 Full-Stack Developer's Recap to Build 2024

I spent over 40+ hours watching 30+ sessions, keynotes, & reading announcements to gather what Microsoft 365 full-stack developers should know from Build 2024.

https://www.voitanos.io/blog/microsoft-365-fullstack-developer-recap-microsoft-build-2024/

… Microsoft gives developers two options for customizing & extending the Microsoft 365 Copilot extensibility story - create declarative agents (DA’s) or custom engine agents (CEA’s). One key important aspect is taught, whichever you choose to implement your custom agent is 100% transparent to the user - users will work with agents; a user won’t have to determine if they want to use a DA or CEA… in fact, they don’t care! Both will work in the Microsoft 365 Copilot experiences and both can be @mentioned.

To quickly recap:

• DA’s allow you to use JSON scope, provide instructions, specify knowledge sources through capabilities, and teach your agent skills using actions (formerly called plugins). But your agent will run on top of the Microsoft 365 Copilot stack, using it’s foundational model, orchestrator, etc. Learn more about DA’s in my Copilot GA recap.
• CEA’s allow you have a lot more control by creating your own tech stack, use your own models and configure them how you like, your own orchestrator, and more. Learn more about CEA’s in my Copilot GA recap.

The Azure AI Foundry & associated services are perfect for those developers who want to create CEA’s and have more control over the implementation of their agent. These services will allow you to have much more control and capabilities to create unique agent scenarios beyond what you can do with DA’s.

Therefore…

Azure AI Foundry offerings & capabilities

Azure AI Foundry includes the following offerings:

• Azure OpenAI Service: Azure OpenAI Service offers industry-leading coding and language AI models that you can fine-tune to your specific needs for a variety of use cases. Get access to OpenAI’s models including GPT-4o, GPT-4o mini, GPT-4, GPT-4 Turbo with Vision, DALLE-3, Whisper, and Embeddings models, all deployed in your Microsoft Azure subscription.
• Azure AI Search: more on this below 👇
• Azure AI Agent Service: more on this below 👇
• Azure AI Content Safety: Implement AI guardrails and filters in your generative AI applications to detect and prevent offensive and inappropriate content or risks to enhance responsible AI efforts.

These are in addition to many existing Azure AI services from Microsoft, including:

• Azure AI Document Intelligence: Apply advanced machine learning to extract text, key-value pairs, tables, and structures from documents.
• Azure AI Language: Build conversational interfaces, summarize documents, and analyze text using prebuilt AI-powered features.
• Azure AI Speech: Use industry-leading AI services such as speech-to-text, text-to-speech, speech translation, and speaker recognition.
• Azure AI Translator: Translate documents and text in real time across more than 100 languages.
• Azure AI Vision: Read text, analyze images, and detect faces with optical character recognition (OCR) and machine learning.

The Azure AI Foundry is a collection of a few different capabilities:

• The Azure AI Foundry portal, formerly known as Azure AI Studio and now in preview, provides a comprehensive visual interface for developers to discover and evaluate AI models and tools, with a management center for centralized subscription information and controls, enabling teams to manage and optimize AI applications at scale.

• The Azure AI Foundry SDK, currently in preview, offers a unified toolchain for customizing, testing, deploying, and managing AI applications with enterprise-level control, featuring an integrated library and simplified coding experience for improved productivity and scalability, accessible via tools like GitHub, Visual Studio, and Copilot Studio. Want to see it in action?

  Here’s a 7-minute overview showing how you can use the SDK:

• The upcoming Azure AI Agent Service, set to enter preview in December 2024, will allow professional developers to orchestrate, deploy, and scale enterprise-ready agents for automating business processes, featuring options like bring your own storage (BYOS) and private networking to ensure data privacy and compliance.

Learn more:

• Announcing the Azure AI Foundry SDK
• BRK103 · Azure AI Foundry unlocking the AI revolution
• BRK119 · Azure AI Foundry and the dev toolchain to infuse AI in all your apps
• BRK129 · Trustworthy AI: Future trends and best practices

Azure AI Search

The Azure AI Search (formerly known as Azure Cognitive Search) is a fully managed cloud search service that provides information retrieval over user-owned content. it provides developers the ability to leverage search search services, indexes, documents, indexers, data sources, skill sets, and synonym maps. In fact, It’s what powers search on this site!

At Ignite 2024, Microsoft announced 2 updates that step up the usefulness of Azure AI Search when used in the retrieval-augmented generation (RAG) pattern that combines info retrieval systems with LLMs to generate more accurate, relevant and up to date text.

Learn more how you can leverage the RAG pattern in Azure AI Search 👇

Retrieval Augmented Generation (RAG) in Azure AI Search

Learn how generative AI and retrieval augmented generation (RAG) patterns are used in Azure AI Search solutions.

https://learn.microsoft.com/azure/search/retrieval-augmented-generation-overview

The first thing they announced is a new capability: query rewriting. This new feature leverages a specially optimized small language model that’s built for speed. It can generate up to 10 different versions of your search query to help find more relevant results. This is particularly useful for basic keyword searches where you might not be getting enough matches - tests show it improves search accuracy by 4 points on the NDCG@3 scale. The best part? You get this feature included for free when you use semantic ranking queries.

The other update is a new semantic ranker brings several impressive improvements. At its core, it uses a cross-encoder model that reprocesses the top 50 search results to ensure better relevance. Testing across 90 datasets in 19 different languages shows remarkable results - when combined with query rewriting, it improves search accuracy by up to 22 points on the NDCG@3 scale. It’s also much faster, running 2.3 times quicker than the previous version. Microsoft has updated all the supporting models for answers, captions, and highlights as well. The best part is that users don’t need to do anything to get these benefits - the upgrade will happen automatically on November 19, 2024.

Why are these important? Let me just take it straight from the Azure AI Services Blog announcement:

Now, I’m not going to act like an expert in the performance aspects and impacts these two new upgrades have. Instead, check the article below for more details.

Learn more:

• Azure AI Search
• Raising the bar for RAG excellence: query rewriting and new semantic ranker
• Ground your AI agents with knowledge from Bing Search, Microsoft Fabric, SharePoint and more
• BRK105 · Azure AI Search: RAG for better results larger scale faster answers
• BRK104 · How AT&T delivers RAG at scale
• BRK115 · Coding the future of AI
• BRK166 · Better together: Copilot Studio and Azure AI

Azure AI Agent Service

While AI language models have made work much easier and faster, many companies still struggle with complex manual tasks that take too much time and aren’t automated. AI agents are great at automating these tasks. But getting AI agents to work reliably in real business settings has been challenging - existing tools aren’t secure enough to handle sensitive data and actions. Companies also have trouble keeping track of costs, measuring how well the agents perform, making different systems work together, and scaling up their AI solutions.

Microsoft unveiled a comprehensive solution for developers to create and deploy AI agents: the Azure AI Agent Service (public preview coming soon), a set of feature-rich, managed capabilities that brings together all the models, data, tools, and services that enterprises need to automate business processes of any complexity.

The Azure AI Agent Service is designed to scale autonomous agents across business processes with extensive built-in connectivity. Through Azure Logic Apps, developers can access over 1,400 apps and services out-of-the-box. For additional flexibility, the service supports OpenAPI specifications and Azure Functions integration, enabling custom code execution.

The service allows agents to perform a wide range of tasks autonomously, from sending email updates to troubleshooting issues. For sensitive operations like payment processing, the system includes human approval workflows. Microsoft has also integrated robust knowledge capabilities - agents can access public information through Bing, corporate documents via SharePoint, and business analytics through Microsoft Fabric. External data sources can also be connected.

Developers have the flexibility to choose their preferred AI model, whether it’s OpenAI’s GPT-4.0, open-source models like Llama 3, or custom fine-tuned models. Security is a core feature, with authentication flowing seamlessly from users to tools. The service is accessible through the Azure AI Foundry portal for prototyping, and developers can integrate it into their development environments using the Azure AI Foundry SDK.

Want to see it in action? Here’s a quick 2-minute overview video from Microsoft of what the Azure AI Agent Service can do:

Learn more:

• Azure AI Agent Service
• Introducing Azure AI Agent Service
• BRK102 · Introducing Azure AI Foundry Agent Service to scale your AI agents

Conclusion

And there you have it! That’s my recap for Microsoft 365 full-stack developers from the Microsoft Ignite 2024 conference!

What about you… what’s your favorite announcement?

Did I miss anything you think should have made the list?

Let me know by dropping a comment below!