Brokered components when using TFS Build

When using TFS to build my application, it fails when I have an application with Brokered components.

Here is what I am using in my Package.appxmanifest, and it works when I build/run locally. How can I still build/deploy using TFS and keep my brokered components? Is this possible?

<Extensions>
    <Extension Category="windows.activatableClass.inProcessServer">
      <InProcessServer>
        <Path>clrhost.dll</Path>
        <ActivatableClass ActivatableClassId="BrokeredRuntimeComponent.{ClassName}" ThreadingModel="MTA">
          <ActivatableClassAttribute Name="DesktopApplicationPath"
                                     Type="string"
                                     Value="C:\Source\{AppName}\Debug\BrokeredRuntimeComponentProxy" />
        </ActivatableClass>
      </InProcessServer>
    </Extension>
  </Extensions>

When I run the build, I get the following error. I do not have access to run the regsvr32 /s on our TFS. Is this something that needs to be done on every client or can it be done on the build server and run on every device? I think it might work if it's possible to get the build service to run as administrator. Is this even possible?

C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V120\Microsoft.CppCommon.targets (1620): Failed to register output. Please try enabling Per-user Redirection or register the component from a command prompt with elevated permissions.

Please let me know if anything I've put here is unclear so I can hopefully clarify.