PACHI: Power Apps x Copilot Headless Integration
Ever wondered what it would look like if Copilot could go use your Power Apps for you? Look no further!
I built PACHI to demonstrate how Copilot could open and interact with a Power Apps canvas app on a user's behalf to satisfy their request within a Copilot chat session.
Click the image below for a working demonstration
TL;DR
PACHI demonstrates how we could give Copilot the ability to interact with any Power App to fulfill a user's intent.
To support this capability, I developed methods for canvas app schema extraction and loading, canvas app simulation in a headless environment, and an LLM-prompting methodology to facilitate an LLM interacting with a Power App to fulfill the user's intent.
Use PACHI Yourself
- Ensure you have .NET 9.0 installed.
- Install the PAC CLI, log in, and select an environment.
- Enter the endpoint and API key for your Azure OpenAI deployment to keys.json in the root directory (if you have trouble, proceed and it will make the file for you and give you further instructions).
- Navigate to the src folder and run
dotnet run.
How PACHI Works
PACHI, a multi-agent system, uses a few unique capabilities strung together into one cohesive system:
Within a chat session with Copilot (in this demo, just a generic agent), these are the steps that happen:
Step 1: The User Requests Copilot Use a Power App to Accomplish Something
Much like how a user can tag a teams channel or other resources within Copilot, the user can tag a Power Apps app to ask Copilot to use the app to fulfill a request.
In the example above, the user is requesting that Copilot use the "Contact Entry" app to save a new contact with first name 'Tim', last name 'Hanewich', email 'timhanewich@email.com', and phone number '94177743221'.
This then kicks off a multi-step process in which the agent explores the application on its quest to fulfill this user's request.
Step 2: Download App Schema
Upon this being requested, a tool is called. This tool kicks off a multi-step process.
Firstly, the canvas app is downloaded. The PAC CLI is used to download the .msapp file. After it is downloaded, the PAC CLI is then again used to unpack the .msapp into a working directory.
Step 3: Load App Schema Into Headless Simulator
With the canvas app schema downloaded, it is now loaded as a CanvasApp object and then subsequently loaded into a CanvasSimulator object, a class designed to simulate interacting with a canvas app (clicking + typing) in a headless, pure-text environmnt.
Step 4: Describe to the LLM what is on the screen and ask it what to do
With the canvas app ready to be simulated, we now describe to the LLM what it is seeing on the first screen.
For the current screen the headless simulator is on, the Describe() function is used to produce a human-readable string that describes what is visually present on the screen. The task that the LLM is being charged with and a question of what to do next is appended at the bottom.
You are using an app named 'Contact Entry' and you are on a screen named 'WelcomeScreen' within the app.
This is what you see on the screen:
- A label named 'WelcomeTitle' that says 'Welcome to the app where you can enter in new Contact records!'
- A label named 'ClickBelowInstructions' that says 'Click the enter button below to start entering a new contact record'
- A button named 'EnterButton' that says 'Enter'
Your task is: Add contact record with name 'Tim Hanewich', email 'timhanewich@email.com', and phone number 9417774321.
What do you do next?