Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 14 additions & 11 deletions .github/skills/create-launcher-plugin/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,9 +35,12 @@ packaging/storage, not a requirement to add MQTT or Home Assistant.
Let SDK defaults derive `PackageId`, `AssemblyName` and `RootNamespace`
from the filename. Put host-independent logic and transports
in suffixed helper projects only when needed; keep helpers/tests non-packable.
4. Follow MQTT's component `Directory.Build.props` import/version-file pattern.
Use central dependency versions, initial independent full SemVer, matching
non-empty changelog, README and required license/third-party notices.
4. Add the component `Directory.Build.props` importing
`plugins\Directory.Build.props`. Shared plugin defaults load the component's
`version.txt` for `Version`/`PackageVersion`; don't duplicate that logic in
each component. Use central dependency versions, initial independent full
SemVer, matching non-empty changelog, README and required
license/third-party notices.

## Implement the launcher boundary

Expand Down Expand Up @@ -69,14 +72,14 @@ packaging/storage, not a requirement to add MQTT or Home Assistant.

## Register and validate the deliverable

Add the solution entry, registry project/tests/identities/host bounds/version and
independent tag prefix, artifact format and explicit feed eligibility. Update
all explicit allowlists/counts in `eng\PluginRelease.psm1`,
`eng\Build-Plugin.ps1`, `eng\Validate-Plugin.ps1`, `eng\Test-NuGetPackage.ps1`,
repository/selection/destination/publication tests, and CI/release matrices,
dispatch options, tag filters and component jobs. Update root navigation and
release documentation. Do not copy MQTT-specific HA/template validation into a
generic launcher component.
Add the solution entry and `eng\plugins.json` registry entry with project/tests,
identities, host bounds, version file, independent tag prefix, artifact format
and explicit feed eligibility. CI and release selection derive from this
registry; don't copy redundant allowlists or component counts into release
scripts and workflows. Inspect component-specific tests and workflow
conditions, adding only cases the registry-driven path does not cover. Update
root navigation and release documentation. Do not copy MQTT-specific
HA/template validation into a generic launcher component.

Add focused host-independent tests and source-pinned host-loader tests. Adapt
[Test-LauncherPackage.ps1](../../../eng/Test-LauncherPackage.ps1) and
Expand Down
27 changes: 14 additions & 13 deletions .github/skills/create-server-plugin/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,11 +34,12 @@ pipeline. Follow [CONTRIBUTING.md](../../../CONTRIBUTING.md),
after that root: **no `.Plugin` suffix**. Let SDK defaults derive `PackageId`,
`AssemblyName` and `RootNamespace` from the filename. Helpers/tests append descriptive suffixes
to the same root and remain non-packable.
4. Add component `Directory.Build.props` importing the root defaults and
reading `version.txt` for `Version`/`PackageVersion`, following Quake3.
Keep dependencies centrally versioned; only the entry project is packable.
Add a README, license/notices where required, initial full SemVer and a
non-empty changelog section for that exact version.
4. Add the component `Directory.Build.props` importing
`plugins\Directory.Build.props`. Shared plugin defaults load the component's
`version.txt` for `Version`/`PackageVersion`; don't duplicate that logic in
each component. Keep dependencies centrally versioned; only the entry
project is packable. Add a README, license/notices where required, initial
full SemVer and a non-empty changelog section for that exact version.

## Implement the server boundary

Expand Down Expand Up @@ -69,14 +70,14 @@ pipeline. Follow [CONTRIBUTING.md](../../../CONTRIBUTING.md),

## Integrate and verify

Update the root solution and registry with project/test paths, canonical
identities, independent version/tag prefix, host bounds, artifact format and
explicit feed eligibility. Inspect and update **all** explicit allowlists/counts:
`eng\PluginRelease.psm1`, `eng\Build-Plugin.ps1`, `eng\Validate-Plugin.ps1`,
`eng\Test-NuGetPackage.ps1`, selection/repository/destination/publication tests,
and CI/release workflow matrices, dispatch options, tag filters and component
jobs. Update README navigation and the release table; adding a project alone
does not make it selectable or releasable.
Update the root solution and `eng\plugins.json` registry with project/test paths,
canonical identities, independent version/tag prefix, host bounds, artifact
format and explicit feed eligibility. CI and release selection derive from this
registry; don't copy redundant allowlists or component counts into release
scripts and workflows. Inspect component-specific tests and workflow
conditions, adding only cases the registry-driven path does not cover. Update
README navigation and the release table; adding a project alone does not make
it selectable or releasable.

Add focused behavior/auth tests and a private-only package inventory check,
plus isolated loading of the **actual packaged entry assembly** with shared host
Expand Down
147 changes: 147 additions & 0 deletions .github/workflows/skill-evals.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
name: Skill evals

on:
pull_request:
paths:
- .github/skills/create-launcher-plugin/**
- .github/skills/create-server-plugin/**
- .github/workflows/skill-evals.yml
- .vally.yaml
- evals/**
- package.json
- package-lock.json
workflow_dispatch:

permissions:
contents: read

jobs:
lint:
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
persist-credentials: false
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
with:
node-version: 24
cache: npm
- run: npm ci --no-audit --no-fund
- run: npm run lint
env:
VALLY_TELEMETRY_OPTOUT: '1'

select:
runs-on: ubuntu-latest
timeout-minutes: 5
outputs:
eval_paths: ${{ steps.changed.outputs.eval_paths }}
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
fetch-depth: 0
persist-credentials: false
- id: changed
shell: bash
env:
EVENT_NAME: ${{ github.event_name }}
BASE_SHA: ${{ github.event.pull_request.base.sha }}
HEAD_REPOSITORY: ${{ github.event.pull_request.head.repo.full_name }}
GITHUB_REPOSITORY: ${{ github.repository }}
ENABLE_MODEL_EVALS: ${{ vars.ENABLE_SKILL_MODEL_EVALS }}
run: |
declare -A selected=()
if [[ "$EVENT_NAME" == "workflow_dispatch" ]]; then
selected[server]=1
selected[launcher]=1
else
changed_paths=$(git diff --name-only --no-renames "$BASE_SHA" HEAD)
while IFS= read -r path; do
case "$path" in
.github/skills/create-server-plugin/*|evals/create-server-plugin.eval.yaml)
selected[server]=1
;;
.github/skills/create-launcher-plugin/*|evals/create-launcher-plugin.eval.yaml)
selected[launcher]=1
;;
.vally.yaml|.github/workflows/skill-evals.yml|package.json|package-lock.json)
selected[server]=1
selected[launcher]=1
;;
esac
done <<< "$changed_paths"
fi

eval_paths=()
if [[ -n "${selected[server]:-}" ]]; then
eval_paths+=(evals/create-server-plugin.eval.yaml)
fi
if [[ -n "${selected[launcher]:-}" ]]; then
eval_paths+=(evals/create-launcher-plugin.eval.yaml)
fi
{
echo "eval_paths<<EOF"
printf '%s\n' "${eval_paths[@]}"
echo "EOF"
} >> "$GITHUB_OUTPUT"
reason=''
if [[ "$EVENT_NAME" == "pull_request" && "$HEAD_REPOSITORY" != "$GITHUB_REPOSITORY" ]]; then
reason='Model-backed evals are never run on fork PRs.'
elif [[ "$EVENT_NAME" == "pull_request" && "${ENABLE_MODEL_EVALS,,}" != "true" ]]; then
reason='Automatic model-backed evals are disabled. After the organization enables Copilot CLI billing, set the repository Actions variable ENABLE_SKILL_MODEL_EVALS to true to opt in on same-repository PRs.'
elif [[ ${#eval_paths[@]} -eq 0 ]]; then
reason='No affected eval specs were selected.'
fi
if [[ -n "$reason" ]]; then
{
echo '### Model-backed evals not run'
echo "$reason"
echo 'The lint check validates eval specs only; it does not verify skill behavior.'
} >> "$GITHUB_STEP_SUMMARY"
fi

run:
needs: [lint, select]
if: >-
needs.select.outputs.eval_paths != '' &&
(github.event_name == 'workflow_dispatch' ||
(github.event_name == 'pull_request' &&
github.event.pull_request.head.repo.full_name == github.repository &&
vars.ENABLE_SKILL_MODEL_EVALS == 'true'))
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
copilot-requests: write
env:
COPILOT_CLI_PATH: ${{ github.workspace }}/node_modules/@github/copilot/npm-loader.js
VALLY_TELEMETRY_OPTOUT: '1'
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
persist-credentials: false
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
with:
node-version: 24
cache: npm
- run: npm ci --no-audit --no-fund
- name: Explain model evaluation requirements
run: |
{
echo "Copilot-backed evals launch real model sessions and may use AI credits."
echo 'Organization-owned repositories must enable the "Allow use of Copilot CLI billed to the organization" policy.'
echo "Model-backed eval failures, including authentication and billing errors, fail this job."
} >> "$GITHUB_STEP_SUMMARY"
- name: Run selected skill evals
shell: bash
env:
EVAL_PATHS: ${{ needs.select.outputs.eval_paths }}
GITHUB_TOKEN: ${{ github.token }}
run: |
while IFS= read -r eval_path; do
[[ -n "$eval_path" ]] || continue
npm run eval -- --eval-spec "$eval_path" --require-pass --max-retries 0
done <<< "$EVAL_PATHS"
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@
**/.vs/
.cache/
artifacts/
evals/results/
node_modules/
TestResults/
*.user
*.suo
Expand Down
14 changes: 14 additions & 0 deletions .vally.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
paths:
evals:
- evals
evalFilenames:
- "*.eval.yaml"
results: evals/results

environments:
create-server-plugin:
skills:
- ../.github/skills/create-server-plugin
create-launcher-plugin:
skills:
- ../.github/skills/create-launcher-plugin
19 changes: 19 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,5 +88,24 @@ automatic comment. Commenting also depends on Actions being allowed to request
[creating launcher plugins](.github/skills/create-launcher-plugin/SKILL.md),
plugin maintenance, release preparation, and authoring more skills.

### Skill evals

The **Skill evals** workflow runs Vally spec lint on PRs that touch the
creation skills or eval configuration, including PRs from forks. Run
`npm ci` and `npm run lint` locally for the same static check. Lint validates
the eval definitions; it does not exercise skill routing or grade generated
plugin scaffolds. The workflow summary explains when model-based checks
were not run.

Model runs can use AI credits and require the organization policy
**Allow use of Copilot CLI billed to the organization**. Once enabled,
maintainers can manually dispatch **Skill evals** on a trusted ref to run
all cases (the workflow must be on the default branch to be dispatchable),
or set the repository Actions variable `ENABLE_SKILL_MODEL_EVALS` to `true`
to run affected cases automatically on same-repository PRs. Fork PRs never
receive model-request permissions. Keep the optional `run` job out of
required checks; use `lint` for static validation. An opted-in model job
fails if it cannot authenticate or complete the evals.

The community repository is independent and is not an official LANCommander
release channel.
70 changes: 70 additions & 0 deletions evals/create-launcher-plugin.eval.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
name: create-launcher-plugin
description: Checks launcher-plugin creation routing and host-integration guardrails.
type: capability

defaults:
executor: copilot-sdk
model: claude-haiku-5.5
timeout: 5m
runs: 1

agent_environment: create-launcher-plugin

stimuli:
- name: blocks-unverified-launcher-contract
prompt: >-
I want to start a new launcher plugin, but this workspace has no checked-out
repository or pinned host source. What evidence do you need before writing
any code? Do not invent host APIs, storage behavior, or compatibility limits.
tags:
control: positive
graders:
- type: skill-invocation
config:
required: [create-launcher-plugin]
- type: output-matches
config:
pattern: "[Hh]ost|[Ss]ource|contract"
- type: diff-empty

- name: derives-project-identity-from-filename
prompt: >-
Start a new launcher plugin named Example by writing only its minimal
SDK-style entry project at
plugins/Launcher.Example/src/LANCommander.Community.Launcher.Example/LANCommander.Community.Launcher.Example.csproj.
Its assembly name and root namespace must both be
LANCommander.Community.Launcher.Example. Do not implement the plugin or
access the network; this is only an identity scaffold.
tags:
control: positive
graders:
- type: skill-invocation
config:
required: [create-launcher-plugin]
- type: file-exists
config:
path: plugins/Launcher.Example/src/LANCommander.Community.Launcher.Example/LANCommander.Community.Launcher.Example.csproj
- type: file-matches
config:
path: plugins/Launcher.Example/src/LANCommander.Community.Launcher.Example/LANCommander.Community.Launcher.Example.csproj
pattern: '<Project\s+Sdk\s*=\s*["'']Microsoft\.NET\.Sdk["'']'
- type: file-matches
config:
path: plugins/Launcher.Example/src/LANCommander.Community.Launcher.Example/LANCommander.Community.Launcher.Example.csproj
pattern: '<\s*(AssemblyName|RootNamespace)\b'
negate: true

- name: does-not-handle-existing-plugin-release
prompt: >-
How should I prepare a release update for an existing launcher plugin? I
am not creating a new component; answer conceptually without editing files.
tags:
control: negative
graders:
- type: skill-invocation
config:
disallowed: [create-launcher-plugin]
- type: diff-empty

scoring:
threshold: 1.0
Loading
Loading