diff --git a/apps/docs/content/docs/search/gitlab.mdx b/apps/docs/content/docs/search/gitlab.mdx index 283d6c297e8..d2dc87b5662 100644 --- a/apps/docs/content/docs/search/gitlab.mdx +++ b/apps/docs/content/docs/search/gitlab.mdx @@ -3,7 +3,7 @@ title: GitLab description: Search self-managed projects with live administrator checks or CSV grants --- -GitLab Search uses **Service account** mode, with one self-managed project per source. Teammates do not connect personal GitLab accounts. Sim matches their verified identity to the source's permission strategy on each search and read. +GitLab Search uses **Service account** mode, with a self-managed project or group per source. Teammates do not connect personal GitLab accounts. Sim matches their verified identity to the source's permission strategy on each search and read. ## Choose a permission strategy @@ -17,18 +17,18 @@ GitLab Search uses **Service account** mode, with one self-managed project per s The administrator path requires self-managed GitLab 17.4 or later and its administrator identity/settings APIs. A project Maintainer or group Owner is not an instance administrator. Non-admin mode does not require those administrator APIs. -The instance must be reachable from the Sim server and pass outbound-request protections. Use the self-managed host, not `gitlab.com`. Each source binds its host and project; a query cannot redirect the token elsewhere. +The instance must be reachable from the Sim server and pass outbound-request protections. Use the self-managed host, not `gitlab.com`. Each source binds its host and selected project or group; a query cannot redirect the token elsewhere. ## Add a project 1. Create a read-only PAT with the chosen scopes and an appropriate expiry. -2. Open **Settings → Sources → Add source → GitLab → Add project**. +2. Open **Settings → Sources → Add source → GitLab → Add connection**. 3. Choose **Administrator token** or **Non-admin token**. 4. Enter the token, self-managed **Host**, and exact **Project** path or numeric ID. 5. Choose content and filters. Non-admin mode requires both CSV files below. 6. Save, then have a permitted teammate search and read known content. Repeat with a teammate who has no grant. -Use **Add project** again for another project. There is no initial sync. Organization membership and a matching verified email are required, but membership alone does not grant project access. +Use **Add connection** again for another project. There is no initial sync. Organization membership and a matching verified email are required, but membership alone does not grant project access. ## Resource settings @@ -43,6 +43,18 @@ Use **Add project** again for another project. There is no initial sync. Organiz Sim checks returned content against these restrictions independently of the query. Available search APIs depend on the GitLab version and enabled search capabilities. Unsupported searches are reported explicitly. +## Add a group + +Choose **Group** in the source's **Scope** field and enter the numeric group ID or full group path. Enable **Include subgroups** only if descendant projects should be included. Sim discovers up to 1,000 projects visible to the source token. Projects shared from outside the group are excluded. + +The same permission strategies apply. Group membership alone never grants access to every project: administrator mode checks each project's current permissions, and non-admin mode requires a CSV grant for each project and reader. The source token must be able to read every project you want to search. + +In CSV mode, each successful upload rediscovers the group's projects and replaces all grants atomically. Newly discovered projects become searchable only for users explicitly listed in the uploaded CSV. Updating and saving source settings also refreshes discovery. A failed discovery or stale revision preserves the previous configuration and grants. + +Search visits project pages within a bounded request budget. Continuation cursors cover remaining projects and results; targeting a project narrows the request. Renamed or moved projects are blocked until a successful refresh; projects outside the configured group remain excluded. Group setup uses the existing project search APIs, so it does not require group-wide advanced search. + +To expand an existing deployment, create a **separate group source** after the application deployment finishes. Keep the working project source and its CI variables unchanged while validating the group. Give the group job its new `SIM_SOURCE_ID`, then pause the old source after confirming the wider setup. Regular knowledge-base ingestion remains project-scoped. + ## Non-admin CSV permissions Use **Download template** beside each field, replace the examples, and upload both files. @@ -75,14 +87,16 @@ Saving the CSVs grants the access listed in them. Changing a membership in GitLa ### Export both CSVs from GitLab -Download [export-permissions.py](/static/examples/gitlab/export-permissions.py) and save it as `ci/export-gitlab-permissions.py` in your repository. It uses Python 3.10 or later with no extra packages and exports one project per run. +Download [export-permissions.py](/static/examples/gitlab/export-permissions.py) and save it as `ci/export-gitlab-permissions.py` in your repository. It uses Python 3.10 or later with no extra packages and exports a project or group per run. -Use GitLab 17.4 or later and an internal export token with `read_api` and permission to see all members of the target project, their primary emails, and confirmation status. Keep this token in a masked, protected CI variable named `GITLAB_EXPORT_TOKEN`. It stays in your CI job; Sim receives the CSVs. The source's separate read-only PAT continues to retrieve content. +Use GitLab 17.4 or later and an internal export token with `read_api` and permission to see all members of each selected project, their primary emails, and confirmation status. Keep this token in a masked, protected CI variable named `GITLAB_EXPORT_TOKEN`. It stays in your CI job; Sim receives the CSVs. The source's separate read-only PAT continues to retrieve content. The exporter makes only these GitLab API requests: | Endpoint | Purpose | | --- | --- | +| `GET /api/v4/groups/:id` | Resolve a group identity when using `--group`. | +| `GET /api/v4/groups/:id/projects` | Page through group projects, excluding shared projects; include subgroups only when requested. | | `GET /api/v4/projects/:id` | Resolve the numeric ID and full project path. | | `GET /api/v4/projects/:id/members/all` | Page through visible direct, inherited, and invited-group members, 100 at a time. | | `GET /api/v4/users/:id` | Read each eligible member's account state, email confirmation, and primary email. | @@ -98,11 +112,13 @@ python3 ci/export-gitlab-permissions.py \ --output-dir gitlab-permissions ``` +For a group, replace `--project engineering/platform` with `--group engineering`. Add `--include-subgroups` only when the Sim source has the same setting. Both CSVs cover the whole group: users appear once in `users.csv`, and `project-permissions.csv` contains a row for each project/user grant. Projects without eligible readers contribute no grants. If the entire export has no readers, pause the source instead of uploading empty files. + This creates `gitlab-permissions/users.csv` and `gitlab-permissions/project-permissions.csv`. Numeric project IDs are also accepted. The output directory must not already exist. Both files are published together after the export completes and passes validation. The export includes active, unlocked human members with Reporter access or higher. It excludes Guests, Planners, bots, and memberships expiring today or earlier in UTC. It does not infer access from public/internal visibility, administrator or Auditor privileges, or custom permissions below Reporter. GitLab policies such as SSO or external authorization are not represented by these CSVs. Review the grants against your organization's access requirements before uploading them. -Requests run serially, paced at 60 per minute by default. Set `--requests-per-minute` to change this export-job ceiling; it does not configure live Search traffic. Each run uses one project request, one request per membership page, and one profile request per candidate reader. API errors, hidden emails, identity conflicts, inconsistent pagination totals (when GitLab provides them), and oversized exports stop the job without publishing either file. Run exports after bulk membership updates finish; paginated API reads are not a point-in-time snapshot. +Requests run serially, paced at 60 per minute by default. Set `--requests-per-minute` to change this export-job ceiling; it does not configure live Search traffic. Each run reads each project's identity and membership pages, plus one profile request per distinct candidate reader. Group exports also read the group identity and project listing. The request ceiling applies across the entire export. API errors, hidden emails, identity conflicts, inconsistent pagination totals (when GitLab provides them), and oversized exports stop the job without publishing either file. Run exports after bulk membership updates finish; paginated API reads are not a point-in-time snapshot. ## Access checks @@ -114,15 +130,15 @@ Reads recheck current settings, permissions, and metadata. A reference from anot ## Update permissions or rotate the token -In project settings, replace either CSV file, retaining the other when appropriate. Sim validates the resulting pair before saving. Invalid or stale updates preserve saved grants. Removed readers lose access on the next operation. +In source settings, replace either CSV file, retaining the other when appropriate. Sim validates the resulting pair before saving. Invalid or stale updates preserve saved grants. Removed readers lose access on the next operation. -To rotate the PAT, supply its replacement in source settings. A blank token retains the saved token. Changing host, project, or permission strategy requires valid configuration for the new identity. +To rotate the PAT, supply its replacement in source settings. A blank token retains the saved token. Changing host, scope, or permission strategy requires valid configuration for the new identity. ### Automate updates with GitLab CI After an admin configures a source in **Non-admin token** mode, a job can [read its revision](/api-reference/organizations/getSearchSourcePermissions) and [replace both CSVs](/api-reference/organizations/replaceSearchSourcePermissions). Copy `SIM_ORG_ID` and `SIM_SOURCE_ID` from the source settings URL: `/o//settings/integrations/sources/`. Set `SIM_BASE_URL` to your Sim origin (for hosted Sim, `https://www.sim.ai`). -Store an organization admin's [personal Sim API key](/api-reference/authentication) with [**Full access**](/api-reference/authentication#key-access) as a [masked, protected GitLab CI variable](https://docs.gitlab.com/ci/variables/) named `SIM_API_KEY`. A personal key inherits its owner's access; a read-only key can read the revision but not replace the CSVs. This endpoint can only read the revision or replace both CSVs; it cannot change the GitLab PAT, host, or project. +Store an organization admin's [personal Sim API key](/api-reference/authentication) with [**Full access**](/api-reference/authentication#key-access) as a [masked, protected GitLab CI variable](https://docs.gitlab.com/ci/variables/) named `SIM_API_KEY`. A personal key inherits its owner's access; a read-only key can read the revision but not replace the CSVs. This endpoint can only read the revision or replace both CSVs; it cannot change the GitLab PAT, host, or configured scope. Commit [the exporter above](#export-both-csvs-from-gitlab) at `ci/export-gitlab-permissions.py`. Set `GITLAB_EXPORT_TOKEN`, `GITLAB_PROJECT` (full path or numeric ID), and the Sim variables above. This example uses `CI_SERVER_URL` for the GitLab host and runs on a schedule against a protected branch. Choose the schedule based on how quickly permission changes must reach Sim. @@ -157,7 +173,17 @@ sync_gitlab_permissions: -H 'Content-Type: application/json' --data-binary "@$export_root/update.json" "$url" ``` -A successful update returns the new revision. A `409 Conflict` means someone updated the source after the job read its revision. Rerun the job from the beginning so it exports current grants. An invalid or header-only export leaves the previous grants in place. Alert on failed jobs so revocations aren't missed. If the project should have no readers, choose **Pause search** in the source settings. You can update the grants while the source is paused, then resume search when the grants are correct. +For a group source, keep the same revision/read/upload steps and replace only the exporter command above: + +```bash +python3 ci/export-gitlab-permissions.py \ + --gitlab-url "$CI_SERVER_URL" --group "$GITLAB_GROUP" \ + --output-dir "$export_root/permissions" --requests-per-minute 60 +``` + +Set `GITLAB_GROUP` to the configured group's full path or numeric ID. Add `--include-subgroups` if enabled in Sim. The existing `--project` command, CSV headers, upload endpoint, and revision checks are unchanged. Grants for projects outside the configured group are rejected; a reused path with a different project ID requires a new connection. + +A successful update returns the new revision. A `409 Conflict` means someone updated the source after the job read its revision. Rerun the job from the beginning so it exports current grants. An invalid or header-only export leaves the previous grants in place. Alert on failed jobs so revocations aren't missed. If the source should have no readers, choose **Pause search** in the source settings. You can update the grants while the source is paused, then resume search when the grants are correct. ## Troubleshooting diff --git a/apps/docs/public/static/examples/gitlab/export-permissions.py b/apps/docs/public/static/examples/gitlab/export-permissions.py index 33a53d049c5..5f37b73b512 100644 --- a/apps/docs/public/static/examples/gitlab/export-permissions.py +++ b/apps/docs/public/static/examples/gitlab/export-permissions.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Export one GitLab project's readers to Sim's permission CSVs (Python 3.10+).""" +"""Export GitLab project or group readers to Sim's permission CSVs (Python 3.10+).""" import argparse import csv @@ -14,12 +14,13 @@ import tempfile import time from urllib.error import HTTPError, URLError -from urllib.parse import quote, urlencode, urlsplit +from urllib.parse import quote, unquote, urlencode, urlsplit from urllib.request import HTTPRedirectHandler, HTTPSHandler, Request, build_opener MAX_BYTES = 4 * 1024 * 1024 MAX_ROWS = 100_000 +MAX_PROJECTS = 1000 EMAIL = re.compile( r"[a-zA-Z0-9.!#$%&'*+/=?^_`{|}~-]+@" r"[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?" @@ -73,6 +74,17 @@ def get(self, path, query=None): raise ExportError("GitLab returned invalid JSON.") from None +def resource_id(value): + raw = str(value).strip() + try: + decoded = unquote(raw, errors="strict") + except UnicodeDecodeError: + decoded = raw + if not decoded or decoded in (".", ".."): + raise ExportError("Use a valid GitLab project or group ID or full path.") + return quote(decoded, safe="") + + def identifier(value): if type(value) is not int or not 0 < value <= 9_007_199_254_740_991: raise ExportError("GitLab returned an invalid numeric identity.") @@ -85,8 +97,29 @@ def active(record): return record["state"] == "active" and not record.get("locked", False) -def export(client, project): - record, _ = client.get("projects/" + quote(project, safe="")) +def verified_email(profile, user_id): + if not isinstance(profile, dict) or identifier(profile.get("id")) != user_id: + raise ExportError("GitLab returned a mismatched user identity.") + if not active(profile) or profile.get("bot", False): + return None + confirmed_at = profile.get("confirmed_at") + if not isinstance(confirmed_at, str): + return None + try: + datetime.fromisoformat(confirmed_at.replace("Z", "+00:00")) + except ValueError: + return None + email = profile.get("email") + if not isinstance(email, str): + raise ExportError(f"User {user_id}: primary email is unavailable. The CI token needs directory email access.") + email = email.strip().lower() + if len(email) > 254 or not EMAIL.fullmatch(email): + raise ExportError(f"User {user_id}: primary email is invalid.") + return email + + +def export(client, project, profiles): + record, _ = client.get("projects/" + resource_id(project)) if not isinstance(record, dict): raise ExportError("GitLab returned an invalid project.") project_id = identifier(record.get("id")) @@ -135,26 +168,14 @@ def export(client, project): raise ExportError("GitLab returned an invalid membership expiry.") from None if expired: continue - profile, _ = client.get(f"users/{user_id}") - if not isinstance(profile, dict) or identifier(profile.get("id")) != user_id: - raise ExportError("GitLab returned a mismatched user identity.") - if not active(profile) or profile.get("bot", False): - continue - confirmed_at = profile.get("confirmed_at") - if not isinstance(confirmed_at, str): + if user_id not in profiles: + profile, _ = client.get(f"users/{user_id}") + profiles[user_id] = verified_email(profile, user_id) + if len(profiles) > MAX_ROWS: + raise ExportError("Group identities exceed the 100,000-row limit.") + email = profiles[user_id] + if email is None: continue - try: - datetime.fromisoformat(confirmed_at.replace("Z", "+00:00")) - except ValueError: - continue - email = profile.get("email") - if not isinstance(email, str): - raise ExportError( - f"User {user_id}: primary email is unavailable. The CI token needs directory email access." - ) - email = email.strip().lower() - if len(email) > 254 or not EMAIL.fullmatch(email): - raise ExportError(f"User {user_id}: primary email is invalid.") if email in emails and emails[email] != user_id: raise ExportError("Multiple GitLab users share an email. Resolve the identity conflict first.") emails[email] = user_id @@ -171,18 +192,71 @@ def export(client, project): raise ExportError("GitLab membership pagination exceeds 1,000 pages.") if expected_total is not None and len(seen_members) != expected_total: raise ExportError("Exported membership count does not match GitLab's total. Rerun the export.") - if not users: - raise ExportError("No eligible readers. Pause search in Sim; empty CSVs cannot revoke all grants.") return project_path, users -def write_export(output, project_path, users): +def group_projects(client, group, include_subgroups): + record, _ = client.get("groups/" + resource_id(group)) + if not isinstance(record, dict): + raise ExportError("GitLab returned an invalid group.") + group_id = identifier(record.get("id")) + path = record.get("full_path") + if not isinstance(path, str) or not path: + raise ExportError("GitLab did not return the group path.") + projects, seen_ids, seen_paths = [], set(), set() + expected_total = None + for page in range(1, MAX_PROJECTS // 100 + 2): + rows, headers = client.get(f"groups/{group_id}/projects", { + "with_shared": "false", "include_subgroups": str(include_subgroups).lower(), + "per_page": 100, "page": page, "order_by": "id", "sort": "asc", + }) + if not isinstance(rows, list) or len(rows) > 100: + raise ExportError("GitLab returned an invalid group project page.") + total = headers.get("X-Total") + if total is not None: + if not re.fullmatch(r"[0-9]{1,6}", total) or int(total) > MAX_PROJECTS: + raise ExportError("GitLab group exceeds the 1,000-project limit.") + if expected_total is not None and expected_total != int(total): + raise ExportError("GitLab group project total changed during export.") + expected_total = int(total) + for row in rows: + if not isinstance(row, dict): + raise ExportError("GitLab returned an invalid group project.") + project_id = identifier(row.get("id")) + project_path = row.get("path_with_namespace") + namespace = project_path.rpartition("/")[0] if isinstance(project_path, str) else "" + if not (namespace == path or (include_subgroups and namespace.startswith(path + "/"))): + raise ExportError("GitLab returned a project outside the configured group.") + if project_id in seen_ids or project_path in seen_paths: + raise ExportError("GitLab group pagination repeated a project.") + seen_ids.add(project_id) + seen_paths.add(project_path) + projects.append((project_id, project_path)) + if len(projects) > MAX_PROJECTS: + raise ExportError("GitLab group exceeds the 1,000-project limit.") + next_page = headers.get("X-Next-Page") + if next_page is None: + next_page = str(page + 1) if len(rows) == 100 else "" + if not next_page: + break + if not next_page.isdigit() or int(next_page) != page + 1 or not rows: + raise ExportError("GitLab returned invalid group pagination.") + else: + raise ExportError("GitLab group exceeded its pagination limit.") + if expected_total is not None and len(projects) != expected_total: + raise ExportError("GitLab group project total does not match the complete listing.") + if not projects: + raise ExportError("No accessible projects in the GitLab group.") + return sorted(projects) + + +def write_export(output, grants, users): files = {} ordered = sorted(users) for name, header, rows in [ ("users.csv", ["user_id", "email"], [(user_id, users[user_id]) for user_id in ordered]), ("project-permissions.csv", ["project_path", "user_id"], - [(project_path, user_id) for user_id in ordered]), + sorted(grants)), ]: buffer = io.StringIO(newline="") writer = csv.writer(buffer, lineterminator="\n") @@ -204,10 +278,15 @@ def write_export(output, project_path, users): def main(): parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--gitlab-url", required=True, help="HTTPS instance URL, without /api/v4") - parser.add_argument("--project", required=True, help="Project ID or full namespace/project path") + scope = parser.add_mutually_exclusive_group(required=True) + scope.add_argument("--project", help="Project ID or full namespace/project path") + scope.add_argument("--group", help="Group ID or full namespace/group path") + parser.add_argument("--include-subgroups", action="store_true", help="Include descendant groups (requires --group)") parser.add_argument("--output-dir", default="gitlab-permissions", help="New directory for both CSVs") parser.add_argument("--requests-per-minute", type=int, default=60, help="Request-start ceiling (default: 60)") args = parser.parse_args() + if args.include_subgroups and not args.group: + parser.error("--include-subgroups requires --group.") token = os.environ.get("GITLAB_EXPORT_TOKEN", "") if not token or any(char.isspace() for char in token): parser.error("Set GITLAB_EXPORT_TOKEN to the CI export token.") @@ -217,8 +296,23 @@ def main(): if output.exists() or output.is_symlink(): raise ExportError("Output directory already exists. Choose a new directory for a fresh export.") client = GitLab(args.gitlab_url, token, args.requests_per_minute) - project_path, users = export(client, args.project) - write_export(output, project_path, users) + projects = group_projects(client, args.group, args.include_subgroups) if args.group else [(args.project, None)] + profiles, users, emails, grants = {}, {}, {}, [] + for project_id, expected_path in projects: + project_path, readers = export(client, str(project_id), profiles) + if expected_path is not None and expected_path != project_path: + raise ExportError("GitLab project identity changed during export. Rerun the export.") + for user_id, email in readers.items(): + if email in emails and emails[email] != user_id: + raise ExportError("Multiple GitLab users share an email. Resolve the identity conflict first.") + emails[email] = user_id + users[user_id] = email + grants.append((project_path, user_id)) + if len(grants) > MAX_ROWS: + raise ExportError("Project permissions exceed the 100,000-row limit.") + if not users: + raise ExportError("No eligible readers. Pause search in Sim; empty CSVs cannot revoke all grants.") + write_export(output, grants, users) print(f"Exported {len(users)} readers to {output}/users.csv and {output}/project-permissions.csv") diff --git a/apps/docs/scripts/fixtures/content-baseline.json b/apps/docs/scripts/fixtures/content-baseline.json index 11ba80992de..94fc142889d 100644 --- a/apps/docs/scripts/fixtures/content-baseline.json +++ b/apps/docs/scripts/fixtures/content-baseline.json @@ -177239,6 +177239,13 @@ "sha256": "29261e0a66aac3cf52e7b9164aaa44f235765bcfd2cae4e68d744d31b29b57bf", "bytes": 1441, "line": 129 + }, + { + "lang": "bash", + "meta": null, + "sha256": "b47b752aa05ada4c5cc86ae0288d5ddd6962cc494914734782a2e1af473dc702", + "bytes": 166, + "line": 178 } ], "links": [ @@ -232107,8 +232114,8 @@ }, { "path": "favicon/site.webmanifest", - "bytes": 1142, - "sha256": "9c7befab4f7f4af7df44cc29c7f7331b74cd0d916c19668c5bef0c65c1587b6d" + "bytes": 1140, + "sha256": "966e2c6a7796175ff779afff49e416d0b39ef6fb953d57c151a2bce34682d8b6" }, { "path": "favicon/web-app-manifest-192x192.png", diff --git a/apps/docs/scripts/test-gitlab-export-e2e.py b/apps/docs/scripts/test-gitlab-export-e2e.py index 8f3075e033b..55a85aa4aca 100644 --- a/apps/docs/scripts/test-gitlab-export-e2e.py +++ b/apps/docs/scripts/test-gitlab-export-e2e.py @@ -42,6 +42,39 @@ def do_GET(self): if scenario == "denied": self.reply(403, {"message": TOKEN}) return + if scenario.startswith("group"): + if endpoint in ("groups/engineering", "groups/7"): + self.reply(200, {"id": 7, "full_path": "engineering"}) + return + if endpoint == "groups/7/projects": + query = parse_qs(parsed.query) + assert query.get("with_shared") == ["false"] + assert query.get("include_subgroups") == ["true" if scenario == "group-subgroups" else "false"] + projects = [{"id": 91, "path_with_namespace": "engineering/platform"}] + if query.get("page", ["1"]) == ["1"]: + self.reply(200, projects, {"X-Next-Page": "2", "X-Total": "2"}) + elif scenario == "group-failure": + self.reply(503, {}) + else: + path = "outside/shared" if scenario == "group-outside" else ( + "engineering/sub/backend" if scenario == "group-subgroups" else "engineering/backend") + self.reply(200, [{"id": 92, "path_with_namespace": path}], {"X-Next-Page": "", "X-Total": "2"}) + return + if endpoint == "projects/92": + self.reply(200, {"id": 92, "path_with_namespace": ( + "engineering/sub/backend" if scenario == "group-subgroups" else "engineering/backend")}) + return + if endpoint in ("projects/91/members/all", "projects/92/members/all"): + if scenario == "group-member-failure" and endpoint.startswith("projects/92/"): + self.reply(403, {}) + return + members = [member(1)] if endpoint.startswith("projects/91/") else [member(1), member(2)] + if scenario == "group-duplicate-email" and endpoint.startswith("projects/92/"): + members = [member(2)] + if scenario == "group-empty-project" and endpoint.startswith("projects/92/"): + members = [] + self.reply(200, members, {"X-Next-Page": "", "X-Total": str(len(members))}) + return if endpoint in ("projects/engineering%2Fplatform", "projects/91"): if scenario == "redirect": self.reply(302, {}, {"Location": "/redirect/api/v4/trap"}) @@ -75,7 +108,7 @@ def do_GET(self): user_id = int(endpoint.split("/")[1]) email = {1: "ALICE@example.com", 2: "bob@example.com", 7: "carol@example.com"}.get( user_id, f"service-{user_id}@example.com") - if scenario == "duplicate-email" and user_id == 2: + if scenario in ("duplicate-email", "group-duplicate-email") and user_id == 2: email = "alice@example.com" profile = {"id": user_id, "state": "active", "locked": user_id == 9, "bot": user_id == 5, "email": email, "confirmed_at": "2025-01-01T00:00:00Z"} @@ -173,20 +206,19 @@ def run(): ["engineering/platform", row[0]] for row in expected]], grants parsed = subprocess.run([ "bun", "-e", """ -import { parseGitLabCsv, gitLabCsvSubjects } from './connectors/gitlab/permission-config/parser' +import { parseGitLabCsv } from './connectors/gitlab/permission-config/parser' const directory = process.argv[1] const users = parseGitLabCsv({filename: 'users.csv', content: await Bun.file(`${directory}/users.csv`).text()}, 'userMapping') const grants = parseGitLabCsv({filename: 'project-permissions.csv', content: await Bun.file(`${directory}/project-permissions.csv`).text()}, 'projectPermissions') process.stdout.write(JSON.stringify({ - subjects: gitLabCsvSubjects(users, grants, 'engineering/platform'), - otherProject: gitLabCsvSubjects(users, grants, 'engineering/other'), + users, grants, })) """, str(output), ], cwd=SIM_ROOT, capture_output=True, text=True, timeout=20) assert parsed.returncode == 0, parsed.stderr assert json.loads(parsed.stdout) == { - "subjects": sorted("u:" + row[1] for row in expected), - "otherProject": [], + "users": expected, + "grants": [["engineering/platform", row[0]] for row in expected], }, parsed.stdout if scenario == "paced": arrivals = [arrived for case, _, _, arrived in GitLabFixture.calls if case == scenario] @@ -197,6 +229,51 @@ def run(): "observedSeconds": arrivals[-1] - arrivals[0], "minimumSeconds": minimum_span} report["checks"].append({"name": scenario, "passed": True}) + for scenario in ("group", "group-subgroups", "group-empty-project", "group-outside", "group-failure", "group-member-failure", "group-duplicate-email"): + output = root / scenario + command = [sys.executable, str(SCRIPT), "--gitlab-url", + f"https://127.0.0.1:{server.server_port}/{scenario}", "--group", "engineering", + "--output-dir", str(output), "--requests-per-minute", "600000"] + if scenario == "group-subgroups": + command.append("--include-subgroups") + result = subprocess.run(command, capture_output=True, text=True, timeout=20, + env={**os.environ, "GITLAB_EXPORT_TOKEN": TOKEN, "SSL_CERT_FILE": str(cert)}) + assert TOKEN not in result.stdout + result.stderr + if scenario in ("group-outside", "group-failure", "group-member-failure", "group-duplicate-email"): + assert result.returncode != 0 and not output.exists(), (scenario, result.stderr) + error = {"group-outside": "outside", "group-failure": "503", "group-member-failure": "403", "group-duplicate-email": "email"}[scenario] + assert error in result.stderr.lower(), (scenario, result.stderr) + else: + assert result.returncode == 0, (scenario, result.stderr) + with (output / "users.csv").open() as handle: + users = list(csv.reader(handle)) + with (output / "project-permissions.csv").open() as handle: + grants = list(csv.reader(handle)) + expected_users = [["user_id", "email"], ["1", "alice@example.com"]] + expected_grants = [["project_path", "user_id"], ["engineering/platform", "1"]] + if scenario != "group-empty-project": + expected_users.append(["2", "bob@example.com"]) + path = "engineering/sub/backend" if scenario == "group-subgroups" else "engineering/backend" + expected_grants.extend([[path, "1"], [path, "2"]]) + assert users == expected_users, users + assert sorted(grants[1:]) == sorted(expected_grants[1:]), grants + profiles = [endpoint for case, endpoint, _, _ in GitLabFixture.calls + if case == scenario and endpoint.startswith("users/")] + assert len(profiles) == len(set(profiles)), "Fetched a shared user profile more than once" + report["checks"].append({"name": scenario, "passed": True}) + for index, group in enumerate((".", "..", "%2e", "%2e%2e", ".%2e", "%2e.")): + output = root / f"invalid-group-{index}" + calls = len(GitLabFixture.calls) + result = subprocess.run([ + sys.executable, str(SCRIPT), "--gitlab-url", + f"https://127.0.0.1:{server.server_port}/group", "--group", group, + "--output-dir", str(output), + ], capture_output=True, text=True, timeout=20, + env={**os.environ, "GITLAB_EXPORT_TOKEN": TOKEN, "SSL_CERT_FILE": str(cert)}) + assert result.returncode != 0 and not output.exists() + assert TOKEN not in result.stdout + result.stderr + assert len(GitLabFixture.calls) == calls, "Sent a token for an invalid group identifier" + report["checks"].append({"name": "invalid-group-identifiers", "passed": True}) assert not any(endpoint == "trap" for _, endpoint, _, _ in GitLabFixture.calls) report["passed"] = True finally: diff --git a/apps/sim/app/o/[organizationId]/settings/components/integrations/live-search-policy-modal.tsx b/apps/sim/app/o/[organizationId]/settings/components/integrations/live-search-policy-modal.tsx index 8c2aa0ea7c3..dc3823870ef 100644 --- a/apps/sim/app/o/[organizationId]/settings/components/integrations/live-search-policy-modal.tsx +++ b/apps/sim/app/o/[organizationId]/settings/components/integrations/live-search-policy-modal.tsx @@ -146,7 +146,7 @@ export function LiveSearchPolicyModal({ (provider === 'gitlab' || provider === 'github' ? ( {provider === 'github' ? 'Manage GitHub repositories' - : 'Manage GitLab projects'} + : 'Manage GitLab connections'} ) : null} diff --git a/apps/sim/app/o/[organizationId]/settings/components/integrations/live-search-settings.tsx b/apps/sim/app/o/[organizationId]/settings/components/integrations/live-search-settings.tsx index 5eb06b22474..4b4134f190e 100644 --- a/apps/sim/app/o/[organizationId]/settings/components/integrations/live-search-settings.tsx +++ b/apps/sim/app/o/[organizationId]/settings/components/integrations/live-search-settings.tsx @@ -203,7 +203,7 @@ export function LiveSearchSettings() { ))) const scope = serviceAccount ? type === 'gitlab' - ? 'Projects and permissions' + ? 'Connections and permissions' : type === 'github' ? 'GitHub App repositories' : integration.policy?.sourceId @@ -227,11 +227,7 @@ export function LiveSearchSettings() {
{serviceAccount && ( - {type === 'gitlab' - ? 'Projects' - : type === 'github' - ? 'Repositories' - : 'Connections'} + {type === 'github' ? 'Repositories' : 'Connections'} )} {type === 'slack' && ( @@ -339,7 +335,7 @@ export function LiveSearchSettings() { : type === 'github' ? 'GitHub App repositories' : type === 'gitlab' - ? 'Projects and permissions' + ? 'Connections and permissions' : access.admin ? 'Service account or member accounts' : 'Member accounts', diff --git a/apps/sim/app/o/[organizationId]/settings/integrations/providers/[connectorType]/provider-detail.tsx b/apps/sim/app/o/[organizationId]/settings/integrations/providers/[connectorType]/provider-detail.tsx index b053227406e..d6c9d2efc03 100644 --- a/apps/sim/app/o/[organizationId]/settings/integrations/providers/[connectorType]/provider-detail.tsx +++ b/apps/sim/app/o/[organizationId]/settings/integrations/providers/[connectorType]/provider-detail.tsx @@ -97,7 +97,7 @@ export function OrganizationProviderDetail({ connectorType }: OrganizationProvid ? 'Unavailable in this deployment' : provider ? connectorType === 'gitlab' - ? 'Projects and permissions' + ? 'Connections and permissions' : connectorType === 'github' ? 'GitHub App repositories' : 'Service account connections' diff --git a/apps/sim/app/workspace/[workspaceId]/knowledge/[id]/components/edit-connector-modal/use-connector-settings-form.test.tsx b/apps/sim/app/workspace/[workspaceId]/knowledge/[id]/components/edit-connector-modal/use-connector-settings-form.test.tsx index 0113ffce7b4..a3d2060facb 100644 --- a/apps/sim/app/workspace/[workspaceId]/knowledge/[id]/components/edit-connector-modal/use-connector-settings-form.test.tsx +++ b/apps/sim/app/workspace/[workspaceId]/knowledge/[id]/components/edit-connector-modal/use-connector-settings-form.test.tsx @@ -199,6 +199,50 @@ describe('shared connector settings form', () => { expect(form.dirty).toBe(false) }) + it.each([ + { + initial: { project: 'engineering/platform' }, + scope: 'group', + field: 'group', + removed: ['project'], + }, + { + initial: { scope: 'group', group: 'engineering', includeSubgroups: 'true' }, + scope: 'project', + field: 'project', + removed: ['group', 'includeSubgroups'], + }, + ])( + 'removes the old resource fields when saving $scope scope', + ({ initial, scope, field, removed }) => { + render( + connector({ + connectorType: 'gitlab', + accessMode: 'admin', + sourceConfig: { + host: 'gitlab.example.com', + ...initial, + tagSlotMapping: { title: 'tag1' }, + }, + }), + scope + ) + expect(form.dirty).toBe(false) + act(() => form.fieldsProps.onFieldChange('scope', scope)) + act(() => form.fieldsProps.onFieldChange(field, '42')) + expect(form.canSave).toBe(true) + act(() => form.save()) + const saved = mocks.update.mock.calls[0]?.[0].updates.sourceConfig + expect(saved).toMatchObject({ + host: 'gitlab.example.com', + scope, + [field]: '42', + tagSlotMapping: { title: 'tag1' }, + }) + for (const key of removed) expect(saved).not.toHaveProperty(key) + } + ) + it('preserves the GitHub repository and pending connection after an incompatible replacement is refused', () => { const sourceConfig = { repository: 'acme/platform' } render( diff --git a/apps/sim/app/workspace/[workspaceId]/knowledge/[id]/components/edit-connector-modal/use-connector-settings-form.ts b/apps/sim/app/workspace/[workspaceId]/knowledge/[id]/components/edit-connector-modal/use-connector-settings-form.ts index 59a35285c5b..7379c06172d 100644 --- a/apps/sim/app/workspace/[workspaceId]/knowledge/[id]/components/edit-connector-modal/use-connector-settings-form.ts +++ b/apps/sim/app/workspace/[workspaceId]/knowledge/[id]/components/edit-connector-modal/use-connector-settings-form.ts @@ -393,6 +393,10 @@ export function useConnectorSettingsForm({ if (Object.keys(changedEntries).length > 0 || modesChanged) { const next: Record = { ...connector.sourceConfig, ...changedEntries } + for (const field of connectorConfig?.configFields ?? []) { + const key = field.canonicalParamId ?? field.id + if (field.condition && !Object.hasOwn(resolved, key)) delete next[key] + } if (Object.keys(canonicalModes).length > 0) { next[CANONICAL_MODES_KEY] = canonicalModes } else { diff --git a/apps/sim/app/workspace/[workspaceId]/knowledge/[id]/hooks/use-connector-config-fields.ts b/apps/sim/app/workspace/[workspaceId]/knowledge/[id]/hooks/use-connector-config-fields.ts index 1308e9c7e2c..f87b5319942 100644 --- a/apps/sim/app/workspace/[workspaceId]/knowledge/[id]/hooks/use-connector-config-fields.ts +++ b/apps/sim/app/workspace/[workspaceId]/knowledge/[id]/hooks/use-connector-config-fields.ts @@ -178,6 +178,12 @@ export function useConnectorConfigFields({ const isFieldVisible = useCallback( (field: ConnectorConfigField): boolean => { + if ( + field.condition && + (sourceConfig[field.condition.field] === field.condition.value) === + Boolean(field.condition.negate) + ) + return false if (accessMode === 'members' && field.hideInMemberMode) return false if (accessMode === 'admin' && field.hideInAdminMode) return false if (accessMode !== 'admin' && field.showInAdminModeOnly) return false @@ -185,7 +191,7 @@ export function useConnectorConfigFields({ const activeMode = canonicalModes[field.canonicalParamId] ?? 'basic' return field.mode === activeMode }, - [canonicalModes, accessMode] + [canonicalModes, accessMode, sourceConfig] ) const isFieldPopulated = useCallback( @@ -249,6 +255,7 @@ export function useConnectorConfigFields({ if (!connectorConfig) return resolved for (const field of connectorConfig.configFields) { + if (field.condition && !isFieldVisible(field)) continue if (field.canonicalParamId) { if (processed.has(field.canonicalParamId)) continue processed.add(field.canonicalParamId) @@ -266,7 +273,14 @@ export function useConnectorConfigFields({ resolved[SOURCE_LABELS_KEY] = createSourceLabelMetadata(connectorConfig, resolved, selectionLabels) ?? null return resolved - }, [connectorConfig, canonicalGroups, canonicalModes, sourceConfig, selectionLabels]) + }, [ + connectorConfig, + canonicalGroups, + canonicalModes, + sourceConfig, + selectionLabels, + isFieldVisible, + ]) return { sourceConfig, diff --git a/apps/sim/connectors/gitlab/gitlab.ts b/apps/sim/connectors/gitlab/gitlab.ts index 6b791d8de97..ec3602dda11 100644 --- a/apps/sim/connectors/gitlab/gitlab.ts +++ b/apps/sim/connectors/gitlab/gitlab.ts @@ -6,6 +6,7 @@ import type { SecureFetchResponse } from '@/lib/core/security/input-validation.s import { decodeTextBuffer } from '@/lib/file-parsers/utils' import { secureFetchWithRetry } from '@/lib/knowledge/documents/secure-fetch.server' import { VALIDATE_RETRY_OPTIONS } from '@/lib/knowledge/documents/utils' +import { gitLabGroupMatchesConfig, gitLabGroupScopeSchema } from '@/connectors/gitlab/group' import { gitlabConnectorMeta } from '@/connectors/gitlab/meta' import { gitLabPermissionConfig } from '@/connectors/gitlab/permission-config/capability' import { @@ -15,6 +16,7 @@ import { import { getGitLabDocumentAcls, openGitLabDirectory, + validateGitLabGroupToken, validateGitLabPermissionToken, } from '@/connectors/gitlab/permissions' import type { ConnectorConfig, ExternalDocument, ExternalDocumentList } from '@/connectors/types' @@ -1331,6 +1333,41 @@ export const gitlabConnector: ConnectorConfig = { sourceConfig: Record, syncContext?: Record ): Promise<{ valid: boolean; error?: string }> => { + if (sourceConfig.scope === 'group' || syncContext?.gitLabGroupScope) { + if (syncContext?.isSearchIndex !== true || syncContext.mirrorsSourceAcls !== true) + return { + valid: false, + error: 'GitLab groups require an administrator-managed Search source.', + } + const scope = gitLabGroupScopeSchema.safeParse(syncContext.gitLabGroupScope) + const mode = syncContext.gitLabPermissionMode + if ( + !scope.success || + (mode !== 'csv' && mode !== 'administrator') || + !gitLabGroupMatchesConfig(scope.data, sourceConfig) + ) + return { + valid: false, + error: 'Save the GitLab group with its token and permission settings.', + } + try { + const current = await validateGitLabGroupToken(accessToken, sourceConfig, mode) + if ( + current.group.id !== scope.data.group.id || + current.group.path !== scope.data.group.path + ) + return { + valid: false, + error: 'The GitLab group identity changed. Update the source and its permissions.', + } + return { valid: true } + } catch (error) { + return { + valid: false, + error: getErrorMessage(error, 'Could not validate the GitLab group.'), + } + } + } const project = (sourceConfig.project as string)?.trim() if (!project) { return { valid: false, error: 'Project is required' } diff --git a/apps/sim/connectors/gitlab/group.ts b/apps/sim/connectors/gitlab/group.ts new file mode 100644 index 00000000000..fe68a81f0ca --- /dev/null +++ b/apps/sim/connectors/gitlab/group.ts @@ -0,0 +1,49 @@ +import { z } from 'zod' +import { encodeGitLabResourceId, normalizeGitLabHost } from '@/tools/gitlab/utils' + +export const GITLAB_GROUP_MAX_PROJECTS = 1000 + +export const gitLabGroupScopeSchema = z.object({ + host: z.string().min(1), + group: z.object({ + id: z.number().int().positive(), + path: z.string().min(1), + includeSubgroups: z.boolean(), + }), + projects: z + .array(z.object({ id: z.number().int().positive(), path: z.string().min(1) })) + .min(1) + .max(GITLAB_GROUP_MAX_PROJECTS), +}) + +export type GitLabGroupScope = z.output + +/** Namespace boundaries exclude shared projects and similarly named sibling groups. */ +export function gitLabProjectInGroup(path: string, group: GitLabGroupScope['group']): boolean { + const namespace = path.slice(0, path.lastIndexOf('/')) + return ( + namespace === group.path || (group.includeSubgroups && namespace.startsWith(`${group.path}/`)) + ) +} + +/** Bind saved permissions to the same canonical scope on every validation and live request. */ +export function gitLabGroupMatchesConfig( + scope: GitLabGroupScope, + config: Record +): boolean { + try { + return ( + config.scope === 'group' && + typeof config.group === 'string' && + (config.project === undefined || config.project === '') && + [undefined, '', 'true', 'false'].includes(config.includeSubgroups as string | undefined) && + scope.host === normalizeGitLabHost(config.host) && + [String(scope.group.id), encodeGitLabResourceId(scope.group.path)].includes( + encodeGitLabResourceId(config.group) + ) && + scope.group.includeSubgroups === (config.includeSubgroups === 'true') + ) + } catch { + return false + } +} diff --git a/apps/sim/connectors/gitlab/meta.ts b/apps/sim/connectors/gitlab/meta.ts index 6a8ded013d2..a124fe79411 100644 --- a/apps/sim/connectors/gitlab/meta.ts +++ b/apps/sim/connectors/gitlab/meta.ts @@ -39,8 +39,42 @@ export const gitlabConnectorMeta: ConnectorMeta = { requiredInAdminMode: true, description: 'Your GitLab instance host. Sim Search requires a self-managed instance.', }, + { + id: 'scope', + title: 'Scope', + type: 'dropdown', + searchOnly: true, + placeholder: 'Project', + options: [ + { label: 'Project', id: 'project' }, + { label: 'Group', id: 'group' }, + ], + }, + { + id: 'group', + title: 'Group', + type: 'short-input', + searchOnly: true, + required: true, + condition: { field: 'scope', value: 'group' }, + placeholder: 'group/subgroup or numeric ID', + description: 'Up to 1,000 projects. Shared projects are excluded.', + }, + { + id: 'includeSubgroups', + title: 'Include subgroups', + type: 'dropdown', + searchOnly: true, + condition: { field: 'scope', value: 'group' }, + placeholder: 'No', + options: [ + { label: 'No', id: 'false' }, + { label: 'Yes', id: 'true' }, + ], + }, { id: 'project', + condition: { field: 'scope', value: 'group', negate: true }, title: 'Project', type: 'short-input', placeholder: 'group/project or numeric ID', diff --git a/apps/sim/connectors/gitlab/permission-config/capability.ts b/apps/sim/connectors/gitlab/permission-config/capability.ts index 9daf8b9f232..52dc1c7973a 100644 --- a/apps/sim/connectors/gitlab/permission-config/capability.ts +++ b/apps/sim/connectors/gitlab/permission-config/capability.ts @@ -1,3 +1,8 @@ +import { getErrorMessage } from '@sim/utils/errors' +import { isPlainRecord } from '@sim/utils/object' +import { decryptApiKey } from '@/lib/api-key/crypto' +import { OutboundRoutingError } from '@/lib/core/network/routing' +import { OrchestrationError } from '@/lib/core/orchestration/types' import type { ConnectorPermissionConfigCapability } from '@/lib/knowledge/connectors/permission-config' import { writeConnectorPermissions } from '@/lib/knowledge/connectors/permission-store' import { prepareGitLabPermissionChange } from '@/connectors/gitlab/permission-config/prepare' @@ -6,21 +11,52 @@ import { prepareGitLabPermissions, readGitLabPermissionMetadata, readGitLabPermissionSummaries, - seedGitLabCsvContext, + seedGitLabPermissionContext, } from '@/connectors/gitlab/permission-config/repository' import { setGitLabCsvContext } from '@/connectors/gitlab/permission-config/types' +import { discoverGitLabGroup } from '@/connectors/gitlab/permissions' export const gitLabPermissionConfig: ConnectorPermissionConfigCapability = { grantReplacement: { supports: (metadata) => readGitLabPermissionMetadata(metadata).mode === 'csv', - prepare(snapshot, expectedRevision, grants) { + async prepare(snapshot, expectedRevision, grants, source) { const stored = { ...readGitLabPermissionMetadata(snapshot.metadata), revision: snapshot.revision, } + if (stored.revision !== expectedRevision) + throw new OrchestrationError( + 'conflict', + 'GitLab permissions changed. Reload the connection before saving.' + ) + let scope = stored + if ('group' in stored) { + if ( + !source.encryptedApiKey || + !isPlainRecord(source.sourceConfig) || + source.sourceConfig.scope !== 'group' + ) + throw new OrchestrationError('validation', 'An admin must update this GitLab connection.') + const { decrypted } = await decryptApiKey(source.encryptedApiKey) + const discovered = await discoverGitLabGroup(decrypted, source.sourceConfig, 'csv').catch( + (error: unknown) => { + if (error instanceof OutboundRoutingError) throw error + throw new OrchestrationError( + 'validation', + getErrorMessage(error, 'Unable to validate the GitLab group.') + ) + } + ) + if (discovered.host !== stored.host || discovered.group.id !== stored.group.id) + throw new OrchestrationError( + 'validation', + 'The GitLab group identity changed. Update the connection before replacing permissions.' + ) + scope = { ...stored, ...discovered } + } const prepared = prepareGitLabPermissions( { ...grants, mode: 'csv', expectedRevision }, - stored, + scope, stored, true ) @@ -40,7 +76,12 @@ export const gitLabPermissionConfig: ConnectorPermissionConfigCapability = { requiresContentSync: prepared.rewriteAccess || Boolean(change.encryptedApiKey), encryptedApiKey: change.encryptedApiKey, populateSyncContext(context, connectorId) { - if (prepared.snapshot.mode === 'csv') + if ('group' in prepared.snapshot) { + const { host, group, projects } = prepared.snapshot + context.gitLabGroupScope = { host, group, projects } + context.gitLabPermissionMode = prepared.snapshot.mode + } + if (prepared.snapshot.mode === 'csv' && !('group' in prepared.snapshot)) setGitLabCsvContext(context, { connectorId, ...prepared.snapshot }) }, async write(tx, connectorId) { @@ -68,5 +109,5 @@ export const gitLabPermissionConfig: ConnectorPermissionConfigCapability = { ]) ) }, - populateSyncContext: seedGitLabCsvContext, + populateSyncContext: seedGitLabPermissionContext, } diff --git a/apps/sim/connectors/gitlab/permission-config/parser.test.ts b/apps/sim/connectors/gitlab/permission-config/parser.test.ts index 77d0a4aa2cb..dec2a016302 100644 --- a/apps/sim/connectors/gitlab/permission-config/parser.test.ts +++ b/apps/sim/connectors/gitlab/permission-config/parser.test.ts @@ -1,5 +1,5 @@ import { describe, expect, it } from 'vitest' -import { gitLabCsvSubjects, parseGitLabCsv } from '@/connectors/gitlab/permission-config/parser' +import { parseGitLabCsv } from '@/connectors/gitlab/permission-config/parser' import { GITLAB_CSV_MAX_BYTES } from '@/connectors/gitlab/permission-config/types' describe('GitLab CSV permission uploads', () => { @@ -55,22 +55,6 @@ describe('GitLab CSV permission uploads', () => { parseGitLabCsv({ filename: '../users.csv', content: '1,a@example.com' }, 'userMapping') ).toThrow('filename') }) - it('requires an exact selected project and mapping; unrelated rows and unknown users grant nothing', () => { - expect( - gitLabCsvSubjects( - [ - ['1', 'a@example.com'], - ['2', 'b@example.com'], - ], - [ - ['group/project', '1'], - ['group/other', '2'], - ['group/project', '99'], - ], - 'group/project' - ) - ).toEqual(['u:a@example.com']) - }) it('checks UTF-8 bytes, not just string length', () => { expect(() => parseGitLabCsv( diff --git a/apps/sim/connectors/gitlab/permission-config/parser.ts b/apps/sim/connectors/gitlab/permission-config/parser.ts index c0334ee2732..e5d13e02529 100644 --- a/apps/sim/connectors/gitlab/permission-config/parser.ts +++ b/apps/sim/connectors/gitlab/permission-config/parser.ts @@ -105,19 +105,3 @@ export function parseGitLabCsv(upload: GitLabCsvUpload, kind: GitLabCsvKind): [s if (rows.length === 0) reject('The file must contain at least one data row.') return rows } - -/** Only explicitly mapped users in the selected canonical project receive grants. */ -export function gitLabCsvSubjects( - users: readonly [string, string][], - permissions: readonly [string, string][], - projectPath: string -): string[] { - const identities = new Map(users) - const subjects = new Set() - for (const [path, id] of permissions) { - if (path !== projectPath) continue - const email = identities.get(id) - if (email) subjects.add(`u:${email}`) - } - return [...subjects].sort() -} diff --git a/apps/sim/connectors/gitlab/permission-config/prepare.ts b/apps/sim/connectors/gitlab/permission-config/prepare.ts index b1ea162d7a4..2b1f53d3332 100644 --- a/apps/sim/connectors/gitlab/permission-config/prepare.ts +++ b/apps/sim/connectors/gitlab/permission-config/prepare.ts @@ -8,10 +8,11 @@ import { prepareGitLabPermissions, } from '@/connectors/gitlab/permission-config/repository' import type { GitLabPermissionInput } from '@/connectors/gitlab/permission-config/types' -import { validateGitLabCsvToken } from '@/connectors/gitlab/permissions' +import { discoverGitLabGroup, validateGitLabCsvToken } from '@/connectors/gitlab/permissions' /** Prepares private writes after the application has authorized the canonical connector owner. */ export async function prepareGitLabPermissionChange(input: { + isSearchIndex?: boolean connectorType: string accessMode: string sourceConfig: Record @@ -38,6 +39,14 @@ export async function prepareGitLabPermissionChange(input: { 'Token and CSV settings require permission-aware access.' ) } + if ( + input.sourceConfig.scope === 'group' && + (input.isSearchIndex !== true || input.accessMode !== 'admin') + ) + throw new OrchestrationError( + 'validation', + 'GitLab groups require an administrator-managed Search source.' + ) const existing = input.existing ? await loadGitLabPermissionSnapshot(input.existing.id) : null if (input.accessMode !== 'admin' && !existing) return {} if (!input.permissions && !existing && !input.apiKey) return {} @@ -59,12 +68,21 @@ export async function prepareGitLabPermissionChange(input: { const sameProject = existing && previousConfig && + !('group' in existing) && + input.sourceConfig.scope !== 'group' && previousConfig.host === input.sourceConfig.host && previousConfig.project === input.sourceConfig.project const project = - sameProject && !input.apiKey && existing.mode === permissions.mode + sameProject && + existing && + !('group' in existing) && + !input.apiKey && + existing.mode === permissions.mode ? { host: existing.host, projectId: existing.projectId, projectPath: existing.projectPath } - : await validateGitLabCsvToken( + : await (input.sourceConfig.scope === 'group' + ? (token: string, config: Record) => + discoverGitLabGroup(token, config, permissions.mode) + : validateGitLabCsvToken)( input.apiKey ?? (input.existing?.encryptedApiKey ? (await decryptApiKey(input.existing.encryptedApiKey)).decrypted diff --git a/apps/sim/connectors/gitlab/permission-config/repository.test.ts b/apps/sim/connectors/gitlab/permission-config/repository.test.ts new file mode 100644 index 00000000000..c74d7013962 --- /dev/null +++ b/apps/sim/connectors/gitlab/permission-config/repository.test.ts @@ -0,0 +1,58 @@ +import { describe, expect, it } from 'vitest' +import { prepareGitLabPermissions } from '@/connectors/gitlab/permission-config/repository' + +const files = { + mode: 'csv' as const, + userMapping: { + filename: 'users.csv', + content: 'user_id,email\n1,alice@example.com\n2,bob@example.com', + }, + projectPermissions: { + filename: 'permissions.csv', + content: 'project_path,user_id\nteam/a,1\nteam/b,2', + }, +} +const group = { + host: 'gitlab.example.com', + group: { id: 7, path: 'team', includeSubgroups: false }, + projects: [ + { id: 11, path: 'team/a' }, + { id: 12, path: 'team/b' }, + ], +} + +describe('GitLab permission snapshot boundaries', () => { + it('rejects grants outside the discovered group without creating a partial snapshot', () => { + expect(() => + prepareGitLabPermissions( + { + ...files, + projectPermissions: { + filename: 'permissions.csv', + content: 'project_path,user_id\nteam/a,1\nother/private,2', + }, + }, + group, + null, + false + ) + ).toThrow('outside') + }) + it('does not transfer grants when a project path is reused', () => { + const initial = prepareGitLabPermissions(files, group, null, false) + expect(() => + prepareGitLabPermissions( + { ...files, expectedRevision: 1 }, + { + ...group, + projects: [ + { id: 99, path: 'team/a' }, + { id: 12, path: 'team/b' }, + ], + }, + initial.snapshot, + true + ) + ).toThrow('identity') + }) +}) diff --git a/apps/sim/connectors/gitlab/permission-config/repository.ts b/apps/sim/connectors/gitlab/permission-config/repository.ts index 4cdab7b03ba..817d2e99999 100644 --- a/apps/sim/connectors/gitlab/permission-config/repository.ts +++ b/apps/sim/connectors/gitlab/permission-config/repository.ts @@ -7,9 +7,9 @@ import { loadConnectorPermissionSnapshot, readConnectorPermissionMetadata, } from '@/lib/knowledge/connectors/permission-store' +import { type GitLabGroupScope, gitLabGroupScopeSchema } from '@/connectors/gitlab/group' import { GitLabCsvValidationError, - gitLabCsvSubjects, parseGitLabCsv, } from '@/connectors/gitlab/permission-config/parser' import { @@ -19,13 +19,15 @@ import { setGitLabCsvContext, } from '@/connectors/gitlab/permission-config/types' -const metadataSchema = gitLabPermissionSummarySchema - .omit({ provider: true, revision: true }) - .extend({ +const summarySchema = gitLabPermissionSummarySchema.omit({ provider: true, revision: true }) +const metadataSchema = z.union([ + summarySchema.merge(gitLabGroupScopeSchema), + summarySchema.extend({ host: z.string().min(1), projectId: z.number().int().positive(), projectPath: z.string().min(1), - }) + }), +]) type RevisionMetadata = z.output & { revision: number } interface StoredRows { userMappingRows: [string, string][] | null @@ -36,7 +38,7 @@ type Snapshot = RevisionMetadata & StoredRows export interface PreparedGitLabPermissions { expectedRevision: number snapshot: Snapshot - subjects: string[] + groups: ConnectorPermissionData['groups'] rewriteAccess: boolean } @@ -95,7 +97,7 @@ export async function loadGitLabPermissionSnapshot(connectorId: string): Promise export function prepareGitLabPermissions( input: GitLabPermissionInput, - project: { host: string; projectId: number; projectPath: string }, + project: { host: string; projectId: number; projectPath: string } | GitLabGroupScope, existing: (RevisionMetadata & Partial) | null, replacing: boolean ): PreparedGitLabPermissions { @@ -135,12 +137,41 @@ export function prepareGitLabPermissions( if (input.mode === 'csv' && (!userMapping || !projectPermissions)) { throw new OrchestrationError('validation', 'Non-admin token mode requires both CSV files.') } + if ('group' in project && userMapping && projectPermissions) { + const paths = new Set(project.projects.map((entry) => entry.path)) + if (projectPermissions.rows.some(([path]) => !paths.has(path))) + throw new OrchestrationError( + 'validation', + 'The CSV contains projects outside the configured GitLab group.' + ) + if (existing && 'group' in existing) { + const previousIds = new Map(existing.projects.map((entry) => [entry.path, entry.id])) + if ( + project.projects.some( + (entry) => previousIds.has(entry.path) && previousIds.get(entry.path) !== entry.id + ) + ) + throw new OrchestrationError( + 'validation', + 'A GitLab project identity changed. Create a new connection before granting access to a replacement project.' + ) + } + } + const paths = + 'group' in project ? project.projects.map((entry) => entry.path) : [project.projectPath] + const grants = new Map(paths.map((path) => [path, new Set()])) + if (userMapping && projectPermissions) { + const identities = new Map(userMapping.rows) + for (const [path, id] of projectPermissions.rows) { + const email = identities.get(id) + if (email) grants.get(path)?.add(`u:${email}`) + } + } + const subjects = (path: string) => [...(grants.get(path) ?? [])].sort() return { expectedRevision: revision, snapshot: { - host: project.host, - projectId: project.projectId, - projectPath: project.projectPath, + ...project, mode: input.mode, revision: revision + 1, userMapping: userMapping @@ -160,15 +191,22 @@ export function prepareGitLabPermissions( userMappingRows: userMapping?.rows ?? null, projectPermissionRows: projectPermissions?.rows ?? null, }, - subjects: - userMapping && projectPermissions - ? gitLabCsvSubjects(userMapping.rows, projectPermissions.rows, project.projectPath) - : [], + groups: + 'group' in project + ? project.projects.map((entry) => ({ + groupKey: `project:${entry.id}`, + subjects: subjects(entry.path), + })) + : [{ groupKey: 'project', subjects: subjects(project.projectPath) }], rewriteAccess: replacing && ((existing?.mode ?? 'administrator') !== input.mode || (existing !== null && - (existing.host !== project.host || existing.projectId !== project.projectId))), + (existing.host !== project.host || + ('group' in existing || 'group' in project + ? JSON.stringify('group' in existing ? existing.group : null) !== + JSON.stringify('group' in project ? project.group : null) + : existing.projectId !== project.projectId)))), } } @@ -183,7 +221,7 @@ export function gitLabPermissionData(prepared: PreparedGitLabPermissions): Conne return { metadata, payload: { userMappingRows, projectPermissionRows }, - groups: [{ groupKey: 'project', subjects: prepared.subjects }], + groups: prepared.groups, } } @@ -200,12 +238,16 @@ export async function readGitLabPermissionSummaries( } /** Workers load canonical mode without loading the potentially large CSV snapshot. */ -export async function seedGitLabCsvContext( +export async function seedGitLabPermissionContext( connectorId: string, context: Record ): Promise { const [row] = await readConnectorPermissionMetadata([connectorId]) if (!row) return - const { mode, host, projectId, projectPath } = readGitLabPermissionMetadata(row.metadata) - if (mode === 'csv') setGitLabCsvContext(context, { connectorId, host, projectId, projectPath }) + const metadata = readGitLabPermissionMetadata(row.metadata) + if ('group' in metadata) { + const { host, group, projects } = metadata + context.gitLabGroupScope = { host, group, projects } + context.gitLabPermissionMode = metadata.mode + } else if (metadata.mode === 'csv') setGitLabCsvContext(context, { connectorId, ...metadata }) } diff --git a/apps/sim/connectors/gitlab/permissions.ts b/apps/sim/connectors/gitlab/permissions.ts index 6020342e6a3..bb84aa26cd7 100644 --- a/apps/sim/connectors/gitlab/permissions.ts +++ b/apps/sim/connectors/gitlab/permissions.ts @@ -3,6 +3,11 @@ import { normalizeEmail } from '@sim/utils/string' import { groupToken, sortAccessTokens, userToken } from '@/lib/knowledge/access/tokens' import { secureFetchWithRetry } from '@/lib/knowledge/documents/secure-fetch.server' import { VALIDATE_RETRY_OPTIONS } from '@/lib/knowledge/documents/utils' +import { + GITLAB_GROUP_MAX_PROJECTS, + type GitLabGroupScope, + gitLabProjectInGroup, +} from '@/connectors/gitlab/group' import { getGitLabCsvContext, gitLabCsvGroupToken, @@ -12,7 +17,7 @@ import { type GitLabSourcePolicy, } from '@/connectors/gitlab/permission-policy' import type { ConnectorDirectory, ExternalDocument } from '@/connectors/types' -import { normalizeGitLabHost } from '@/tools/gitlab/utils' +import { encodeGitLabResourceId, normalizeGitLabHost } from '@/tools/gitlab/utils' const logger = createLogger('GitLabPermissions') @@ -63,6 +68,92 @@ export async function validateGitLabCsvToken( } } +/** Validate the current token and resolve the canonical group without enumerating its projects. */ +export async function validateGitLabGroupToken( + token: string, + config: Record, + mode: 'csv' | 'administrator', + signal?: AbortSignal +): Promise> { + const host = normalizeGitLabHost(config.host) + const requested = typeof config.group === 'string' ? config.group.trim() : '' + if (!requested || (config.project !== undefined && config.project !== '')) + throw new Error('Provide a GitLab group ID or full path without a project.') + if (![undefined, '', 'true', 'false'].includes(config.includeSubgroups as string | undefined)) + throw new Error('Invalid GitLab subgroup setting') + const encodedGroup = encodeGitLabResourceId(requested) + const base = `https://${host}/api/v4` + const viewer = await read(`${base}/user`, token, 'account', true, signal) + if ( + viewer.state !== 'active' || + viewer.locked === true || + !Number.isSafeInteger(viewer.id) || + (mode === 'administrator' && viewer.is_admin !== true) + ) + throw new Error( + mode === 'administrator' + ? 'Use an active GitLab administrator token, or choose non-admin CSV permissions.' + : 'An active GitLab identity with read_api access is required' + ) + const record = await read<{ id: number; full_path: string }>( + `${base}/groups/${encodedGroup}`, + token, + 'group', + true, + signal + ) + if ( + !Number.isSafeInteger(record.id) || + record.id <= 0 || + typeof record.full_path !== 'string' || + !record.full_path + ) + throw new Error('GitLab did not return the group identity') + const group = { + id: record.id, + path: record.full_path, + includeSubgroups: config.includeSubgroups === 'true', + } + return { host, group } +} + +/** Resolve a bounded, complete group scope before any permission snapshot can be replaced. */ +export async function discoverGitLabGroup( + token: string, + config: Record, + mode: 'csv' | 'administrator', + signal?: AbortSignal +): Promise { + const { host, group } = await validateGitLabGroupToken(token, config, mode, signal) + const base = `https://${host}/api/v4` + const rows = await listAll( + `${base}/groups/${group.id}/projects?with_shared=false&include_subgroups=${group.includeSubgroups}&order_by=id&sort=asc`, + token, + signal, + GITLAB_GROUP_MAX_PROJECTS + ) + const projects: GitLabGroupScope['projects'] = [] + const ids = new Set() + const paths = new Set() + for (const row of rows) { + if ( + !Number.isSafeInteger(row.id) || + row.id <= 0 || + typeof row.path_with_namespace !== 'string' || + !gitLabProjectInGroup(row.path_with_namespace, group) || + ids.has(row.id) || + paths.has(row.path_with_namespace) + ) + throw new Error('GitLab returned an inconsistent group project listing. Retry the update.') + ids.add(row.id) + paths.add(row.path_with_namespace) + projects.push({ id: row.id, path: row.path_with_namespace }) + } + if (!projects.length) throw new Error('The GitLab token cannot read any projects in this group.') + projects.sort((left, right) => left.id - right.id) + return { host, group, projects } +} + export interface GitLabPermissionUser { id: number email?: string @@ -117,7 +208,7 @@ function sourceAddress(sourceConfig: Record) { async function read( url: string, token: string, - operation: 'account' | 'project', + operation: 'account' | 'project' | 'group', validating = false, signal?: AbortSignal ): Promise { @@ -149,9 +240,11 @@ async function read( } if (response.status === 404) { throw new Error( - operation === 'project' - ? 'Could not access this GitLab project. Check the project ID or full path, token access, and IP restrictions.' - : 'Could not verify the GitLab account. Check the host and API access.' + operation === 'group' + ? 'Could not access this GitLab group. Check the group ID or full path, token access, and IP restrictions.' + : operation === 'project' + ? 'Could not access this GitLab project. Check the project ID or full path, token access, and IP restrictions.' + : 'Could not verify the GitLab account. Check the host and API access.' ) } throw new Error( @@ -192,13 +285,19 @@ async function requireAdministrator( } /** Bounded complete enumeration. Never accept a partial directory as authoritative membership. */ -async function listAll(url: string, token: string, signal?: AbortSignal): Promise { +async function listAll( + url: string, + token: string, + signal?: AbortSignal, + maxItems?: number +): Promise { const initial = new URL(url) initial.searchParams.set('per_page', String(PAGE_SIZE)) let next: URL | null = initial const seen = new Set() const items: T[] = [] let retainedBytes = 0 + let expectedTotal: number | undefined for (let page = 0; next && page < MAX_DIRECTORY_PAGES; page += 1) { if (seen.has(next.href)) throw new Error('GitLab permission pagination repeated a page') seen.add(next.href) @@ -219,6 +318,21 @@ async function listAll(url: string, token: string, signal?: AbortSignal): Pro throw new Error('GitLab permission directory exceeded its response budget') } items.push(...body) + if (maxItems !== undefined) { + const totalHeader = response.headers.get('x-total') + if (totalHeader !== null) { + const total = Number(totalHeader) + if ( + !Number.isSafeInteger(total) || + total < 0 || + total > maxItems || + (expectedTotal !== undefined && total !== expectedTotal) + ) + throw new Error('GitLab group project total changed or exceeded its limit') + expectedTotal = total + } + if (items.length > maxItems) throw new Error('GitLab group exceeded its project limit') + } const link = response.headers .get('link') ?.split(',') @@ -230,15 +344,31 @@ async function listAll(url: string, token: string, signal?: AbortSignal): Pro if (candidate.origin !== initial.origin || candidate.pathname !== initial.pathname) { throw new Error('GitLab permission pagination changed its source') } + if ( + maxItems !== undefined && + [...initial.searchParams].some( + ([key, value]) => key !== 'page' && candidate.searchParams.get(key) !== value + ) + ) + throw new Error('GitLab group pagination changed its scope') next = candidate } else if (nextPage) { if (!/^\d+$/.test(nextPage)) throw new Error('Invalid GitLab permission page') next = new URL(initial) next.searchParams.set('page', nextPage) + } else if ( + maxItems !== undefined && + body.length === PAGE_SIZE && + response.headers.get('x-next-page') === null + ) { + next = new URL(initial) + next.searchParams.set('page', String(page + 2)) } else { next = null } } + if (expectedTotal !== undefined && items.length !== expectedTotal) + throw new Error('GitLab group project listing was incomplete') if (next) throw new Error('GitLab permission directory exceeded its page limit') return items } diff --git a/apps/sim/connectors/types.ts b/apps/sim/connectors/types.ts index 22cb3c18dc1..ad71bb3257f 100644 --- a/apps/sim/connectors/types.ts +++ b/apps/sim/connectors/types.ts @@ -325,6 +325,10 @@ export interface SyncResult { * Config field for source-specific settings (rendered in the add-connector UI). */ export interface ConnectorConfigField { + /** Excluded from knowledge base ingestion setup. */ + searchOnly?: true + /** Show and serialize the field only when the selected configuration value matches. */ + condition?: { field: string; value: string; negate?: boolean } id: string title: string /** Label when configuring a central crawl that mirrors source permissions. */ diff --git a/apps/sim/lib/catalog/application/list-registries.test.ts b/apps/sim/lib/catalog/application/list-registries.test.ts index d5701c740f6..baf507b805a 100644 --- a/apps/sim/lib/catalog/application/list-registries.test.ts +++ b/apps/sim/lib/catalog/application/list-registries.test.ts @@ -36,6 +36,21 @@ describe('connector-type catalog', () => { const fullPage = { detail: 'full' as const, limit: 100, offset: 0 } + it('keeps Search-only scopes out of the knowledge-base setup contract', async () => { + const { entries } = await listCatalogConnectorTypes.execute({ + principal: session, + input: { workspaceId: WORKSPACE_ID, search: 'GitLab', ...fullPage }, + }) + const gitlab = entries.find((entry) => entry.connectorType === 'gitlab') + if (!gitlab || !('configFields' in gitlab)) throw new Error('Missing GitLab setup contract') + expect(gitlab.configFields.find((field) => field.id === 'project')?.required).toBe(true) + expect( + gitlab.configFields.filter((field) => + ['scope', 'group', 'includeSubgroups'].includes(field.id) + ) + ).toEqual([]) + }) + it('publishes the multi and canonical-pair config properties a caller cannot infer', async () => { const { entries } = await listCatalogConnectorTypes.execute({ principal: session, diff --git a/apps/sim/lib/catalog/projection/connector-type.ts b/apps/sim/lib/catalog/projection/connector-type.ts index 1f65fc62c60..a5d7b5295bf 100644 --- a/apps/sim/lib/catalog/projection/connector-type.ts +++ b/apps/sim/lib/catalog/projection/connector-type.ts @@ -156,7 +156,7 @@ export function projectConnectorType( description: meta.description, version: meta.version, auth: projectAuth(meta.auth), - configFields: meta.configFields.map(projectConfigField), + configFields: meta.configFields.filter((field) => !field.searchOnly).map(projectConfigField), supportsIncrementalSync: meta.supportsIncrementalSync === true, tagDefinitions: (meta.tagDefinitions ?? []).map(projectTagDefinition), } diff --git a/apps/sim/lib/knowledge/__integration__/gitlab-group.integration.ts b/apps/sim/lib/knowledge/__integration__/gitlab-group.integration.ts new file mode 100644 index 00000000000..29873b741f1 --- /dev/null +++ b/apps/sim/lib/knowledge/__integration__/gitlab-group.integration.ts @@ -0,0 +1,541 @@ +/** Real authorization, PostgreSQL, encryption, revision locks and HTTP provider responses. */ + +import { writeFile } from 'node:fs/promises' +import { createServer } from 'node:http' +import type { Principal } from '@sim/auth/principal' +import { db } from '@sim/db' +import { + knowledgeBase, + knowledgeConnector, + member, + organization, + organizationSearchIntegration, + user, + workspace, +} from '@sim/db/schema' +import { deleteWorkspaceFixture } from '@sim/db/testing/workspace-fixtures' +import { generateId } from '@sim/utils/id' +import { eq, inArray } from 'drizzle-orm' +import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest' + +const transport = vi.hoisted(() => ({ origin: '', paths: [] as string[] })) +vi.mock('@/lib/core/security/input-validation.server', async (original) => { + const module = await original() + return { + ...module, + secureFetchWithValidation: async (url: string, options: RequestInit) => { + const target = new URL(url) + if (target.hostname !== 'gitlab.fixture.test') + throw new Error('Unexpected fixture destination') + transport.paths.push(target.pathname) + return fetch(`${transport.origin}${target.pathname}${target.search}`, options) + }, + } +}) + +import { seedKnowledgeAclFixture } from '@/lib/knowledge/__integration__/seed-source-access-fixture' +import { updateKnowledgeConnectorAccess } from '@/lib/knowledge/application/connector-access' +import { createKnowledgeConnector } from '@/lib/knowledge/application/connectors' +import { + readSearchSourcePermissions, + replaceSearchSourcePermissions, +} from '@/lib/knowledge/application/source-permissions' +import { readConnectorPermissionGrants } from '@/lib/knowledge/connectors/permission-store' +import { readLiveDocument, searchLiveKnowledge } from '@/lib/sim-search/live/application' +import { decodeLiveReference } from '@/lib/sim-search/live/reference' +import { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry' + +describe('GitLab group Search permission boundary', () => { + let ids: Awaited> + let sourceId: string + let legacyId: string + let moved = false + let failedDiscovery = false + let added = false + let confidential = false + let providerState = 'healthy' + const checks: string[] = [] + const token = 'disposable-group-test-token' + const principal = (userId: string): Principal => ({ + kind: 'personal_api_key', + userId, + keyId: 'group-test', + access: 'write', + }) + const config = { + host: 'gitlab.fixture.test', + scope: 'group', + group: 'team', + contentTypes: 'issues', + } + const permissions = (revoke = false) => ({ + provider: 'gitlab' as const, + userMapping: { + filename: 'users.csv', + content: `user_id,email\n1,${ids.aliceId}@fixture.test\n2,${ids.bobId}@fixture.test`, + }, + projectPermissions: { + filename: 'permissions.csv', + content: `project_path,user_id\nteam/a,${revoke ? 2 : 1}\nteam/b,2${added ? '\nteam/c,1' : ''}`, + }, + }) + const server = createServer((request, response) => { + const url = new URL(request.url ?? '/', 'http://fixture') + const path = decodeURIComponent(url.pathname) + const reply = (status: number, body: unknown, headers = {}) => { + response.writeHead(status, { 'content-type': 'application/json', ...headers }) + response.end(JSON.stringify(body)) + } + if ( + request.headers['private-token'] !== token && + request.headers.authorization !== `Bearer ${token}` + ) + return reply(401, {}) + if (providerState === 'revoked') return reply(401, {}) + if (path === '/api/v4/user') return reply(200, { id: 50, state: 'active', locked: false }) + if (path === '/api/v4/groups/team' || path === '/api/v4/groups/7') + return reply(200, { id: 7, full_path: providerState === 'renamed' ? 'renamed-team' : 'team' }) + if (path === '/api/v4/groups/7/projects') { + if (url.searchParams.get('with_shared') !== 'false') return reply(400, {}) + const projects = [ + { id: 11, path_with_namespace: 'team/a' }, + { id: 12, path_with_namespace: 'team/b' }, + ] + if (added) projects.push({ id: 13, path_with_namespace: 'team/c' }) + const page = url.searchParams.get('page') ?? '1' + if (page === '1') + return reply(200, projects.slice(0, 1), { + 'x-total': String(projects.length), + 'x-next-page': '2', + }) + if (failedDiscovery) return reply(403, {}) + return reply(200, projects.slice(1), { + 'x-total': String(projects.length), + 'x-next-page': '', + }) + } + const match = path.match(/^\/api\/v4\/projects\/(11|12|13)(.*)$/) + if (!match) return reply(404, {}) + const id = Number(match[1]) + const projectPath = + id === 11 ? (moved ? 'outside/a' : 'team/a') : id === 12 ? 'team/b' : 'team/c' + const issue = (iid: number) => ({ + project_id: id, + iid, + title: `Atlas ${id} ${iid}`, + description: `Atlas content ${id} ${iid}`, + confidential, + state: 'opened', + labels: [], + web_url: `https://gitlab.fixture.test/${projectPath}/-/issues/${iid}`, + }) + if (!match[2]) + return reply(200, { + id, + path_with_namespace: projectPath, + web_url: `https://gitlab.fixture.test/${projectPath}`, + }) + if (match[2] === '/search') { + if (providerState === 'search-token-expired') return reply(401, {}) + if (providerState === 'search-rate-limited') return reply(429, {}, { 'retry-after': '60' }) + const page = Number(url.searchParams.get('page') ?? 1) + const size = Number(url.searchParams.get('per_page') ?? 20) + return reply(200, [issue(1), issue(2), issue(3)].slice((page - 1) * size, page * size)) + } + const issueMatch = match[2]?.match(/^\/issues\/(\d+)$/) + if (issueMatch) return reply(200, issue(Number(issueMatch[1]))) + if (match[2]?.endsWith('/notes')) return reply(200, []) + return reply(404, {}) + }) + + const search = ( + userId: string, + project?: string, + cursor?: string, + topK = 20, + accountId = sourceId + ) => + searchLiveKnowledge.execute({ + principal: principal(userId), + input: { + organizationId: ids.organizationId, + query: 'Atlas', + topK, + filters: { source: 'gitlab' }, + nativeQueries: [ + { + provider: 'gitlab', + query: 'Atlas', + kind: 'issues', + accountId: `gitlab-source:${accountId}`, + ...(project ? { project } : {}), + ...(cursor ? { cursor } : {}), + }, + ], + resultSecretRegistry: new ResolvedSecretTraceRegistry(), + }, + }) + const read = (userId: string, documentId: string) => + readLiveDocument.execute({ + principal: principal(userId), + input: { + organizationId: ids.organizationId, + documentId, + limit: 3, + resultSecretRegistry: new ResolvedSecretTraceRegistry(), + }, + }) + const replace = (revision: number, revoke = false) => + replaceSearchSourcePermissions.execute({ + principal: principal(ids.aliceId), + input: { + organizationId: ids.organizationId, + sourceId, + expectedRevision: revision, + permissions: permissions(revoke), + }, + }) + + beforeAll(async () => { + await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve)) + const address = server.address() + if (!address || typeof address === 'string') throw new Error('Missing fixture address') + transport.origin = `http://127.0.0.1:${address.port}` + ids = await seedKnowledgeAclFixture() + await db.delete(knowledgeConnector).where(eq(knowledgeConnector.id, ids.connectorId)) + await db + .update(knowledgeBase) + .set({ workspaceId: null, organizationId: ids.organizationId, isSearchIndex: true }) + .where(eq(knowledgeBase.id, ids.knowledgeBaseId)) + await db.insert(member).values( + [ids.aliceId, ids.bobId].map((id) => ({ + id: generateId(), + organizationId: ids.organizationId, + userId: id, + role: id === ids.aliceId ? 'owner' : 'member', + createdAt: new Date(), + })) + ) + await db + .insert(organizationSearchIntegration) + .values({ organizationId: ids.organizationId, connectorType: 'gitlab', approved: true }) + const create = (sourceConfig: Record) => + createKnowledgeConnector.execute({ + principal: principal(ids.aliceId), + input: { + knowledgeBaseId: ids.knowledgeBaseId, + assertedOrganizationId: ids.organizationId, + connectorType: 'gitlab', + accessMode: 'admin', + apiKey: token, + sourceConfig, + permissionConfig: { ...permissions(), mode: 'csv' }, + syncIntervalMinutes: 60, + }, + }) + legacyId = (await create({ host: config.host, project: '11', contentTypes: 'issues' })) + .connector.id + sourceId = (await create(config)).connector.id + }, 60000) + + afterAll(async () => { + if (process.env.GITLAB_GROUP_REPORT_PATH) + await writeFile( + process.env.GITLAB_GROUP_REPORT_PATH, + JSON.stringify( + { + boundary: + 'Application use cases, real PostgreSQL and HTTP GitLab API fixture; TLS/SSRF transport substituted', + checks, + passed: checks.length === 8, + }, + null, + 2 + ) + ) + server.closeAllConnections() + await new Promise((resolve) => server.close(() => resolve())) + if (ids) { + await deleteWorkspaceFixture(db, eq(workspace.id, ids.workspaceId)) + await db.delete(organization).where(eq(organization.id, ids.organizationId)) + await db.delete(user).where(inArray(user.id, [ids.aliceId, ids.bobId])) + } + await db.$client.end() + }) + + it('keeps connection settings usable without scheduling ingestion or bypassing permission scope', async () => { + const update = (sourceConfig?: Record) => + updateKnowledgeConnectorAccess.execute({ + principal: { kind: 'session', userId: ids.aliceId, sessionId: 'fixture-session' }, + input: { + knowledgeBaseId: ids.knowledgeBaseId, + connectorId: sourceId, + assertedOrganizationId: ids.organizationId, + accessMode: 'admin', + ...(sourceConfig ? { sourceConfig } : {}), + }, + }) + expect((await update()).changed).toBe(false) + await update({ ...config, group: '%74eam', issueState: 'opened' }) + const [saved] = await db + .select() + .from(knowledgeConnector) + .where(eq(knowledgeConnector.id, sourceId)) + expect(saved?.nextSyncAt).toBeNull() + expect(saved?.nextMemberSyncAt).toBeNull() + for (const state of ['revoked', 'renamed']) { + providerState = state + try { + await expect(update()).rejects.toMatchObject({ code: 'validation' }) + } finally { + providerState = 'healthy' + } + } + + for (const changedScope of [ + { ...config, group: 'another-group' }, + { ...config, includeSubgroups: 'true' }, + { host: config.host, scope: 'project', project: '11' }, + ]) { + await expect(update(changedScope)).rejects.toMatchObject({ code: 'validation' }) + } + const [unchanged] = await db + .select() + .from(knowledgeConnector) + .where(eq(knowledgeConnector.id, sourceId)) + expect(unchanged).toEqual(saved) + expect( + await readSearchSourcePermissions.execute({ + principal: principal(ids.aliceId), + input: { organizationId: ids.organizationId, sourceId }, + }) + ).toEqual({ revision: 1 }) + expect((await search(ids.aliceId)).results.length).toBeGreaterThan(0) + checks.push('Connection settings preserve live-only scheduling and canonical permission scope') + }) + + it('rejects dot-segment group identifiers before sending the token', async () => { + for (const group of ['.', '..', '%2e', '%2e%2e', '.%2e', '%2e.']) { + const requests = transport.paths.length + await expect( + createKnowledgeConnector.execute({ + principal: principal(ids.aliceId), + input: { + knowledgeBaseId: ids.knowledgeBaseId, + assertedOrganizationId: ids.organizationId, + connectorType: 'gitlab', + accessMode: 'admin', + apiKey: token, + sourceConfig: { ...config, group }, + permissionConfig: { ...permissions(), mode: 'csv' }, + syncIntervalMinutes: 60, + }, + }) + ).rejects.toMatchObject({ code: 'validation' }) + expect(transport.paths).toHaveLength(requests) + } + checks.push('Invalid group IDs fail before provider requests') + }) + + it('separates readers by project and rejects cross-project reads', async () => { + const alice = await search(ids.aliceId) + const bob = await search(ids.bobId) + expect(alice.results.length).toBeGreaterThan(0) + expect(bob.results.length).toBeGreaterThan(0) + expect( + new Set(alice.results.map((row) => decodeLiveReference(row.documentId).container)) + ).toEqual(new Set(['11'])) + expect( + new Set(bob.results.map((row) => decodeLiveReference(row.documentId).container)) + ).toEqual(new Set(['12'])) + const first = alice.results[0] + if (!first) throw new Error('Missing allowed fixture result') + expect((await read(ids.aliceId, first.documentId)).chunks.length).toBeGreaterThan(0) + await expect(read(ids.bobId, first.documentId)).rejects.toMatchObject({ code: 'not_found' }) + expect((await search(ids.aliceId, '12')).results).toEqual([]) + expect((await search(ids.aliceId, '99')).results).toEqual([]) + confidential = true + expect((await search(ids.aliceId)).results).toEqual([]) + await expect(read(ids.aliceId, first.documentId)).rejects.toMatchObject({ code: 'not_found' }) + confidential = false + moved = true + expect((await search(ids.aliceId)).results).toEqual([]) + moved = false + for (const [state, status] of [ + ['search-token-expired', 'reconnect'], + ['search-rate-limited', 'rate_limited'], + ]) { + providerState = state + const requests = transport.paths.length + try { + const failed = await search(ids.aliceId) + expect(failed.results).toEqual([]) + expect(failed.live?.accounts[0]?.status).toBe(status) + expect( + transport.paths.slice(requests).filter((path) => path.endsWith('/search')) + ).toHaveLength(1) + } finally { + providerState = 'healthy' + } + } + checks.push('Per-project identity, reference, confidentiality and moved-project isolation') + }) + + it('atomically discovers new projects and enforces revocation without touching the existing source', async () => { + const initial = await search(ids.aliceId) + const first = initial.results[0] + if (!first) throw new Error('Missing allowed fixture result') + const [legacy] = await db + .select() + .from(knowledgeConnector) + .where(eq(knowledgeConnector.id, legacyId)) + added = true + expect(await replace(1, true)).toEqual({ revision: 2 }) + expect( + await readConnectorPermissionGrants( + sourceId, + 1, + ['project:11'], + `u:${ids.bobId}@fixture.test` + ) + ).toEqual(new Set()) + expect( + await readConnectorPermissionGrants( + sourceId, + 2, + ['project:11'], + `u:${ids.bobId}@fixture.test` + ) + ).toEqual(new Set(['project:11'])) + expect((await search(ids.aliceId, '13')).results.length).toBeGreaterThan(0) + expect((await search(ids.aliceId, '11')).results).toEqual([]) + await expect(read(ids.aliceId, first.documentId)).rejects.toMatchObject({ code: 'not_found' }) + expect((await search(ids.bobId, '11')).results.length).toBeGreaterThan(0) + const [unchanged] = await db + .select() + .from(knowledgeConnector) + .where(eq(knowledgeConnector.id, legacyId)) + expect(unchanged).toEqual(legacy) + const legacyRevision = await readSearchSourcePermissions.execute({ + principal: principal(ids.aliceId), + input: { organizationId: ids.organizationId, sourceId: legacyId }, + }) + expect(legacyRevision).toEqual({ revision: 1 }) + expect( + (await search(ids.aliceId, '11', undefined, 20, legacyId)).results.length + ).toBeGreaterThan(0) + await replaceSearchSourcePermissions.execute({ + principal: principal(ids.aliceId), + input: { + organizationId: ids.organizationId, + sourceId: legacyId, + expectedRevision: 1, + permissions: permissions(true), + }, + }) + expect((await search(ids.aliceId, '11', undefined, 20, legacyId)).results).toEqual([]) + expect((await search(ids.bobId, '11', undefined, 20, legacyId)).results.length).toBeGreaterThan( + 0 + ) + expect( + await readSearchSourcePermissions.execute({ + principal: principal(ids.aliceId), + input: { + organizationId: ids.organizationId, + sourceId, + }, + }) + ).toEqual({ revision: 2 }) + checks.push('Atomic group refresh, revocation and existing source preservation') + }) + + it('preserves all grants on discovery failure and admits only one concurrent revision', async () => { + failedDiscovery = true + await expect(replace(2)).rejects.toMatchObject({ code: 'validation' }) + failedDiscovery = false + expect((await search(ids.bobId, '11')).results.length).toBeGreaterThan(0) + const results = await Promise.allSettled([replace(2), replace(2)]) + expect(results.filter((row) => row.status === 'fulfilled')).toHaveLength(1) + expect(results.filter((row) => row.status === 'rejected')).toHaveLength(1) + expect(results.find((row) => row.status === 'rejected')).toMatchObject({ + reason: { code: 'conflict' }, + }) + expect(results.find((row) => row.status === 'fulfilled')).toMatchObject({ + value: { revision: 3 }, + }) + expect( + await readSearchSourcePermissions.execute({ + principal: principal(ids.aliceId), + input: { + organizationId: ids.organizationId, + sourceId, + }, + }) + ).toEqual({ revision: 3 }) + checks.push('Failure atomicity and concurrent revision protection') + }) + + it('continues through all permitted project pages without gaps and rejects stale cursors', async () => { + const documents: string[] = [] + let cursor: string | undefined + let firstCursor: string | undefined + for (let page = 0; page < 10; page += 1) { + const result = await search(ids.aliceId, undefined, cursor, 2) + for (const row of result.results) { + const ref = decodeLiveReference(row.documentId) + documents.push(`${ref.container}:${ref.id}`) + } + cursor = result.live?.accounts[0]?.nextCursor + firstCursor ??= cursor + if (!cursor) break + } + expect(documents.sort()).toEqual(['11:1', '11:2', '11:3', '13:1', '13:2', '13:3']) + expect(firstCursor).toBeTruthy() + await replace(3) + const stale = await search(ids.aliceId, undefined, firstCursor, 2) + expect(stale.results).toEqual([]) + expect(stale.live?.accounts[0]?.status).not.toBe('ok') + checks.push('Complete bounded pagination and stale continuation rejection') + }) + + it('rejects group scope for ordinary knowledge base ingestion', async () => { + const knowledgeBaseId = generateId() + await db.insert(knowledgeBase).values({ + id: knowledgeBaseId, + userId: ids.aliceId, + workspaceId: ids.workspaceId, + name: 'Ordinary KB fixture', + }) + await expect( + createKnowledgeConnector.execute({ + principal: principal(ids.aliceId), + input: { + knowledgeBaseId, + assertedWorkspaceId: ids.workspaceId, + connectorType: 'gitlab', + accessMode: 'admin', + apiKey: token, + sourceConfig: config, + permissionConfig: { ...permissions(), mode: 'csv' }, + syncIntervalMinutes: 60, + }, + }) + ).rejects.toMatchObject({ code: 'validation' }) + checks.push('Search-only group scope and unchanged knowledge base ingestion') + }) + + it('keeps group updates restricted to organization admins', async () => { + await expect( + replaceSearchSourcePermissions.execute({ + principal: principal(ids.bobId), + input: { + organizationId: ids.organizationId, + sourceId, + expectedRevision: 4, + permissions: permissions(), + }, + }) + ).rejects.toMatchObject({ code: 'forbidden' }) + checks.push('Organization administration boundary') + }) +}) diff --git a/apps/sim/lib/knowledge/application/connector-access.ts b/apps/sim/lib/knowledge/application/connector-access.ts index 55d76f03c1c..13ab39ccb0b 100644 --- a/apps/sim/lib/knowledge/application/connector-access.ts +++ b/apps/sim/lib/knowledge/application/connector-access.ts @@ -292,6 +292,7 @@ export const updateKnowledgeConnectorAccess = defineAuthorizedKnowledgeUseCase({ accessMode: 'members', }) const rejection = await validateConnectorSourceConfig({ + isSearchIndex: context.knowledgeBase.isSearchIndex === true, principal, connector: { ...connector, accessMode: 'members', credentialId }, sourceConfig, @@ -324,6 +325,7 @@ export const updateKnowledgeConnectorAccess = defineAuthorizedKnowledgeUseCase({ }), } const rejection = await validateConnectorSourceConfig({ + isSearchIndex: context.knowledgeBase.isSearchIndex === true, principal, connector: { ...connector, @@ -347,7 +349,12 @@ export const updateKnowledgeConnectorAccess = defineAuthorizedKnowledgeUseCase({ } const outcome = await performUpdateKnowledgeConnectorAccess({ - knowledgeBase: { id: context.knowledgeBaseId, name: context.knowledgeBase.name, ...owner }, + knowledgeBase: { + id: context.knowledgeBaseId, + name: context.knowledgeBase.name, + isSearchIndex: context.knowledgeBase.isSearchIndex === true, + ...owner, + }, connectorId: context.connectorId, target, sourceConfig: input.sourceConfig === undefined ? undefined : sourceConfig, diff --git a/apps/sim/lib/knowledge/application/connectors.test.ts b/apps/sim/lib/knowledge/application/connectors.test.ts index ed8162baf23..050860e7813 100644 --- a/apps/sim/lib/knowledge/application/connectors.test.ts +++ b/apps/sim/lib/knowledge/application/connectors.test.ts @@ -456,6 +456,7 @@ describe('knowledge connector application use cases', () => { } as Parameters[0]['connector'] await expect( validateConnectorSourceConfig({ + isSearchIndex: false, principal: createSessionPrincipal({ userId: 'admin', sessionId: 'session' }), connector, sourceConfig: { adminEmail: 'admin@corp.com' }, @@ -519,6 +520,7 @@ describe('knowledge connector application use cases', () => { mocks.validateConnectorConfig.mockResolvedValueOnce({ valid: false, error: message }) } const result = validateConnectorSourceConfig({ + isSearchIndex: false, principal: patPrincipal, requestId: 'request', workspaceId: 'workspace-b', @@ -1122,6 +1124,7 @@ describe('organization connector credential authorization', () => { mocks.validateConnectorConfig.mockResolvedValueOnce({ valid: false, error: message }) } const result = validateConnectorSourceConfig({ + isSearchIndex: false, principal, requestId: 'request', organizationId: 'org', diff --git a/apps/sim/lib/knowledge/application/connectors.ts b/apps/sim/lib/knowledge/application/connectors.ts index d98fb353444..2257159bad0 100644 --- a/apps/sim/lib/knowledge/application/connectors.ts +++ b/apps/sim/lib/knowledge/application/connectors.ts @@ -392,6 +392,7 @@ export async function validateConnectorSourceConfig(input: { actingUserId: string requestId: string permissionChange?: PreparedConnectorPermissions + isSearchIndex: boolean }): Promise { const accessMode = input.connector.accessMode if (!isConnectorAccessMode(accessMode)) { @@ -485,6 +486,7 @@ export async function validateConnectorSourceConfig(input: { const validationContext = { ...syncContextForToken(resolved), + isSearchIndex: input.isSearchIndex === true, mirrorsSourceAcls: mirrorsSourceAcls(input.connector.accessMode), ...(input.connector.accessMode === 'members' ? PER_MEMBER_LISTING_CONTEXT : {}), } @@ -829,6 +831,7 @@ async function executeCreateKnowledgeConnector({ const apiKey = await resolveConnectorApiKey(input.apiKey, principal, workspaceId) const permissionChange = input.permissionConfig ? await prepareConnectorPermissions(input.connectorType, { + isSearchIndex: context.knowledgeBase.isSearchIndex === true, accessMode: input.accessMode ?? 'workspace', sourceConfig, permissionConfig: input.permissionConfig, @@ -965,6 +968,7 @@ export const updateKnowledgeConnector = defineAuthorizedKnowledgeUseCase({ if (!connector) throw new OrchestrationError('not_found', 'Connector not found') expectedUpdatedAt = connector.updatedAt permissionChange = await prepareConnectorPermissions(connector.connectorType, { + isSearchIndex: context.knowledgeBase.isSearchIndex === true, accessMode: connector.accessMode, sourceConfig: updates.sourceConfig ?? (connector.sourceConfig as Record), permissionConfig, @@ -1018,6 +1022,7 @@ export const updateKnowledgeConnector = defineAuthorizedKnowledgeUseCase({ validateSourceConfig: (connector, sourceConfig) => { const owner = resourceScopeFields(resourceScopeFromOwner(context)) return validateConnectorSourceConfig({ + isSearchIndex: context.knowledgeBase.isSearchIndex === true, principal, connector: permissionChange?.encryptedApiKey ? { ...connector, encryptedApiKey: permissionChange.encryptedApiKey } diff --git a/apps/sim/lib/knowledge/application/source-permissions.ts b/apps/sim/lib/knowledge/application/source-permissions.ts index 4c82b5031e7..002e3d138f9 100644 --- a/apps/sim/lib/knowledge/application/source-permissions.ts +++ b/apps/sim/lib/knowledge/application/source-permissions.ts @@ -66,6 +66,8 @@ async function readPermissionSource({ organizationId, sourceId }: SourceInput) { connectorType: knowledgeConnector.connectorType, revision: snapshot.revision, metadata: snapshot.metadata, + sourceConfig: knowledgeConnector.sourceConfig, + encryptedApiKey: knowledgeConnector.encryptedApiKey, }) .from(knowledgeConnector) .innerJoin(knowledgeBase, eq(knowledgeBase.id, knowledgeConnector.knowledgeBaseId)) @@ -100,7 +102,7 @@ export const replaceSearchSourcePermissions = defineAuthorizedOrganizationUseCas 'The permissions must match the configured source provider.' ) } - const prepared = source.replacement.prepare(source, expectedRevision, permissions) + const prepared = await source.replacement.prepare(source, expectedRevision, permissions, source) const revision = await db.transaction(async (tx) => { await lockOrganizationSearchApproval(tx, organizationId) await authorizeOrganizationOperation( diff --git a/apps/sim/lib/knowledge/connectors/permission-config.ts b/apps/sim/lib/knowledge/connectors/permission-config.ts index d38cd24b318..214b5516ae0 100644 --- a/apps/sim/lib/knowledge/connectors/permission-config.ts +++ b/apps/sim/lib/knowledge/connectors/permission-config.ts @@ -14,6 +14,7 @@ export type ConnectorPermissionConfig = { provider: 'gitlab' } & GitLabPermissio export type ConnectorPermissionSummary = { provider: 'gitlab' } & GitLabPermissionSummary export interface PrepareConnectorPermissionsInput { + isSearchIndex?: boolean accessMode: string sourceConfig: Record permissionConfig?: ConnectorPermissionConfig @@ -41,8 +42,9 @@ export interface ConnectorPermissionConfigCapability { prepare( snapshot: Pick, expectedRevision: number, - permissions: ConnectorPermissionGrants - ): ConnectorPermissionData + permissions: ConnectorPermissionGrants, + source: { sourceConfig: unknown; encryptedApiKey: string | null } + ): ConnectorPermissionData | Promise } prepare( input: PrepareConnectorPermissionsInput diff --git a/apps/sim/lib/knowledge/connectors/permission-store.ts b/apps/sim/lib/knowledge/connectors/permission-store.ts index 3979a617dc3..88978b611ad 100644 --- a/apps/sim/lib/knowledge/connectors/permission-store.ts +++ b/apps/sim/lib/knowledge/connectors/permission-store.ts @@ -37,6 +37,34 @@ export async function hasConnectorPermissionGrant( return Boolean(grant) } +/** Returns grants only for the bounded, canonical collections requested by the authorized caller. */ +export async function readConnectorPermissionGrants( + connectorId: string, + expectedRevision: number, + groupKeys: readonly string[], + subjectToken: string +): Promise> { + const table = knowledgeConnectorPermissionGrant + const snapshot = knowledgeConnectorPermissionSnapshot + const keys = new Set() + for (let offset = 0; offset < groupKeys.length; offset += 1000) { + const rows = await db + .select({ groupKey: table.groupKey }) + .from(table) + .innerJoin(snapshot, eq(snapshot.connectorId, table.connectorId)) + .where( + and( + eq(table.connectorId, connectorId), + eq(snapshot.revision, expectedRevision), + eq(table.subjectToken, subjectToken), + inArray(table.groupKey, groupKeys.slice(offset, offset + 1000)) + ) + ) + for (const row of rows) keys.add(row.groupKey) + } + return keys +} + /** Call only after authorizing the connector; this materializes its private normalized input. */ export async function loadConnectorPermissionSnapshot(connectorId: string) { const [row] = await db @@ -98,17 +126,17 @@ export async function writeConnectorPermissions( await tx .delete(knowledgeConnectorPermissionGrant) .where(eq(knowledgeConnectorPermissionGrant.connectorId, connectorId)) + let batch: Array = [] for (const group of input.groups) { - for (let offset = 0; offset < group.subjects.length; offset += 1000) { - await tx - .insert(knowledgeConnectorPermissionGrant) - .values( - group.subjects - .slice(offset, offset + 1000) - .map((subjectToken) => ({ connectorId, groupKey: group.groupKey, subjectToken })) - ) + for (const subjectToken of group.subjects) { + batch.push({ connectorId, groupKey: group.groupKey, subjectToken }) + if (batch.length === 1000) { + await tx.insert(knowledgeConnectorPermissionGrant).values(batch) + batch = [] + } } } + if (batch.length) await tx.insert(knowledgeConnectorPermissionGrant).values(batch) } catch { /** Query exceptions may contain bind parameters from the private payload. */ throw new OrchestrationError( diff --git a/apps/sim/lib/knowledge/orchestration/connector-access.ts b/apps/sim/lib/knowledge/orchestration/connector-access.ts index 0ad4ff6a248..5bc48fe2946 100644 --- a/apps/sim/lib/knowledge/orchestration/connector-access.ts +++ b/apps/sim/lib/knowledge/orchestration/connector-access.ts @@ -192,7 +192,13 @@ export type ConnectorAccessTarget = | { accessMode: ContentEngineAccessMode; credentialId: string | null } export interface PerformUpdateKnowledgeConnectorAccessParams extends KnowledgeOperationContext { - knowledgeBase: { id: string; name: string; workspaceId?: string; organizationId?: string } + knowledgeBase: { + id: string + name: string + workspaceId?: string + organizationId?: string + isSearchIndex?: boolean + } connectorId: string target: ConnectorAccessTarget sourceConfig?: Record diff --git a/apps/sim/lib/knowledge/orchestration/connectors.test.ts b/apps/sim/lib/knowledge/orchestration/connectors.test.ts index cd36618dd2b..2c1a68159c3 100644 --- a/apps/sim/lib/knowledge/orchestration/connectors.test.ts +++ b/apps/sim/lib/knowledge/orchestration/connectors.test.ts @@ -251,9 +251,13 @@ describe('performCreateKnowledgeConnector', () => { success: true, connector: { connectorType: 'github', credentialId: null }, }) - expect(mockValidateGitHub).toHaveBeenCalledWith('personal-token', sourceConfig, { - mirrorsSourceAcls: false, - }) + expect(mockValidateGitHub).toHaveBeenCalledWith( + 'personal-token', + sourceConfig, + expect.objectContaining({ + mirrorsSourceAcls: false, + }) + ) expect(mockEncryptApiKey).toHaveBeenCalledWith('personal-token') expect(createParams.resolveAccessToken).not.toHaveBeenCalled() expect(dbChainMockFns.values).toHaveBeenCalledWith( diff --git a/apps/sim/lib/knowledge/orchestration/connectors.ts b/apps/sim/lib/knowledge/orchestration/connectors.ts index a8c0e49e30b..1602c4cfe66 100644 --- a/apps/sim/lib/knowledge/orchestration/connectors.ts +++ b/apps/sim/lib/knowledge/orchestration/connectors.ts @@ -363,6 +363,7 @@ export async function performCreateKnowledgeConnector( if (accessToken !== null) { const validationContext = { ...tokenContext, + isSearchIndex: kb.isSearchIndex === true, mirrorsSourceAcls: accessMode === 'admin', ...(accessMode === 'members' ? PER_MEMBER_LISTING_CONTEXT : {}), } diff --git a/apps/sim/lib/sim-search/connection-labels.ts b/apps/sim/lib/sim-search/connection-labels.ts index da7ee159a52..8edecb96218 100644 --- a/apps/sim/lib/sim-search/connection-labels.ts +++ b/apps/sim/lib/sim-search/connection-labels.ts @@ -25,10 +25,10 @@ const CONNECTION_LABELS: Record = { searchPlaceholder: 'Search GitHub repositories...', }, gitlab: { - add: 'Add project', - title: 'Add GitLab project', - empty: 'No GitLab projects added.', - searchPlaceholder: 'Search GitLab projects...', + add: 'Add connection', + title: 'Connect GitLab', + empty: 'No GitLab connections added.', + searchPlaceholder: 'Search GitLab connections...', }, slack: { add: 'Add channels or DMs', diff --git a/apps/sim/lib/sim-search/live/gitlab-admin.test.ts b/apps/sim/lib/sim-search/live/gitlab-admin.test.ts index 5cbcb7d9674..0c3f73b1760 100644 --- a/apps/sim/lib/sim-search/live/gitlab-admin.test.ts +++ b/apps/sim/lib/sim-search/live/gitlab-admin.test.ts @@ -22,7 +22,7 @@ const mocks = vi.hoisted(() => ({ })) vi.mock('@/lib/knowledge/access/scope', () => knowledgeAccessScopeMock) vi.mock('@/connectors/gitlab/permission-config/repository', () => ({ - seedGitLabCsvContext: mocks.seed, + seedGitLabPermissionContext: mocks.seed, })) vi.mock('@/lib/knowledge/connectors/permission-store', () => ({ hasConnectorPermissionGrant: mocks.grant, diff --git a/apps/sim/lib/sim-search/live/gitlab-admin.ts b/apps/sim/lib/sim-search/live/gitlab-admin.ts index 5f9b8075314..8b95c69b2a6 100644 --- a/apps/sim/lib/sim-search/live/gitlab-admin.ts +++ b/apps/sim/lib/sim-search/live/gitlab-admin.ts @@ -6,21 +6,36 @@ import { type ResourceOwner, resourceScopeFromOwner } from '@/lib/core/resource- import { resourceScopeCondition } from '@/lib/core/resource-scope.server' import { createUserKnowledgeAccessProvider } from '@/lib/knowledge/access/scope' import { groupToken } from '@/lib/knowledge/access/tokens' -import { hasConnectorPermissionGrant } from '@/lib/knowledge/connectors/permission-store' +import { + hasConnectorPermissionGrant, + readConnectorPermissionGrants, + readConnectorPermissionMetadata, +} from '@/lib/knowledge/connectors/permission-store' import { searchIntegrationAccessCondition } from '@/lib/knowledge/search/integration-policy' import { readGitLab, searchGitLab } from '@/lib/sim-search/live/gitlab' import { array, NativeSearchError, object, segment, string } from '@/lib/sim-search/live/http' -import { collectNativePages } from '@/lib/sim-search/live/pages' +import { collectNativePages, joinMessages } from '@/lib/sim-search/live/pages' import type { LiveAccount, NativeClient, NativeDocument, + NativePage, NativeSearchInput, } from '@/lib/sim-search/live/types' -import { seedGitLabCsvContext } from '@/connectors/gitlab/permission-config/repository' import { + type GitLabGroupScope, + gitLabGroupMatchesConfig, + gitLabProjectInGroup, +} from '@/connectors/gitlab/group' +import { + readGitLabPermissionMetadata, + seedGitLabPermissionContext, +} from '@/connectors/gitlab/permission-config/repository' +import { + type GitLabCsvSyncContext, getGitLabCsvContext, gitLabCsvGroupToken, + setGitLabCsvContext, } from '@/connectors/gitlab/permission-config/types' import { getGitLabDocumentAcls, @@ -104,16 +119,17 @@ export function gitLabSourceKinds(config: Record): string[] { } /** Fresh source ACL evidence is request-local. CSV grants remain connector-local and administrator maintained. */ -export async function createAdminGitLabSession(input: { +interface GitLabSessionInput { owner: ResourceOwner userId: string source: AdminSource token: string client: NativeClient signal: AbortSignal -}) { - const { owner, userId, source, token, client, signal } = input - const { config } = source +} + +export async function createAdminGitLabSession(input: GitLabSessionInput) { + const { owner, userId } = input const access = await createUserKnowledgeAccessProvider(userId, { workspaceId: owner.workspaceId ?? undefined, organizationId: owner.organizationId ?? undefined, @@ -129,15 +145,33 @@ export async function createAdminGitLabSession(input: { 'unavailable', 'GitLab search requires a verified organization identity.' ) + if (input.source.config.scope === 'group') return createGroupSession(input, own) + return createProjectSession(input, own) +} + +async function createProjectSession( + input: GitLabSessionInput, + own: string, + groupProject?: { identity: GitLabCsvSyncContext; csv: boolean } +) { + const { source, token, client, signal } = input + const { config } = source const context = {} - await seedGitLabCsvContext(source.id, context) + if (groupProject?.csv) setGitLabCsvContext(context, groupProject.identity) + else if (!groupProject) await seedGitLabPermissionContext(source.id, context) const csv = getGitLabCsvContext(context) const tokens = new Set([own]) let project: string let projectPath: string if (csv) { /** Indexed ACL rewrites do not govern current CSV grants for provider-backed reads. */ - if (!(await hasConnectorPermissionGrant(source.id, 'project', own))) + if ( + !(await hasConnectorPermissionGrant( + source.id, + groupProject ? `project:${groupProject.identity.projectId}` : 'project', + own + )) + ) throw new NativeSearchError( 'unavailable', 'This GitLab source is not available to your account.' @@ -175,6 +209,15 @@ export async function createAdminGitLabSession(input: { project = string(current.id) projectPath = string(current.path_with_namespace) } + if ( + groupProject && + (project !== String(groupProject.identity.projectId) || + projectPath !== groupProject.identity.projectPath) + ) + throw new NativeSearchError( + 'unavailable', + 'An admin must refresh permissions for this GitLab project.' + ) if (!project || !projectPath) throw new NativeSearchError('unavailable', 'GitLab project identity could not be verified.') signal.throwIfAborted() @@ -325,3 +368,178 @@ export async function createAdminGitLabSession(input: { }, } } + +/** Traverse bounded project pages; every continuation and document read rechecks current grants. */ +async function createGroupSession(input: GitLabSessionInput, own: string) { + const { source, client } = input + const [stored] = await readConnectorPermissionMetadata([source.id]) + const metadata = stored && readGitLabPermissionMetadata(stored.metadata) + if (!metadata || !('group' in metadata) || !gitLabGroupMatchesConfig(metadata, source.config)) + throw new NativeSearchError('unavailable', 'An admin must update this GitLab group connection.') + const grants = + metadata.mode === 'csv' + ? await readConnectorPermissionGrants( + source.id, + stored.revision, + metadata.projects.map((project) => `project:${project.id}`), + own + ) + : null + const projects = metadata.projects.filter( + (project) => !grants || grants.has(`project:${project.id}`) + ) + const group = object(await client.json(`/api/v4/groups/${metadata.group.id}`, { memo: true })) + if ( + string(group.id) !== String(metadata.group.id) || + string(group.full_path) !== metadata.group.path + ) + throw new NativeSearchError('unavailable', 'An admin must refresh the GitLab group connection.') + const sessions = new Map>() + const session = (project: GitLabGroupScope['projects'][number]) => { + if (!gitLabProjectInGroup(project.path, metadata.group)) + throw new NativeSearchError( + 'unavailable', + 'This project is outside the configured GitLab group.' + ) + let result = sessions.get(project.id) + if (!result) { + result = createProjectSession( + { + ...input, + source: { + ...source, + config: { + ...source.config, + scope: undefined, + group: undefined, + project: String(project.id), + }, + }, + }, + own, + { + csv: metadata.mode === 'csv', + identity: { + connectorId: source.id, + host: metadata.host, + projectId: project.id, + projectPath: project.path, + }, + } + ) + sessions.set(project.id, result) + } + return result + } + const referencedProject = (container?: string) => + projects.find((project) => container === String(project.id) || container === project.path) + return { + async verify(reference: Reference, evidence?: Record) { + const project = referencedProject(reference.container) + return project ? (await session(project)).verify(reference, evidence) : false + }, + async read(reference: Reference) { + const project = referencedProject(reference.container) + if (!project) + throw new NativeSearchError( + 'unavailable', + 'This GitLab document is not available to your account.' + ) + return (await session(project)).read(reference) + }, + async search(search: NativeSearchInput): Promise { + if (search.native?.project) { + const project = referencedProject(search.native.project) + return project ? (await session(project)).search(search) : { documents: [] } + } + const kinds = gitLabSourceKinds(source.config).filter( + (kind) => !search.native?.kind || kind === search.native.kind + ) + const raw = search.native?.cursor ?? `${stored.revision}:${search.limit}:0:0:1` + if (!/^\d{1,10}:\d{1,3}:\d{1,4}:\d:\d{1,4}$/.test(raw)) + throw new NativeSearchError('unavailable', 'Invalid GitLab group page.') + const [revision, pageSize, ...position] = raw.split(':').map(Number) + if (revision !== stored.revision || pageSize !== search.limit) + throw new NativeSearchError( + 'unavailable', + 'GitLab permissions or page size changed. Restart the search.' + ) + let [projectIndex = 0, kindIndex = 0, page = 1] = position + if (page < 1 || projectIndex > projects.length || (kinds.length && kindIndex >= kinds.length)) + throw new NativeSearchError('unavailable', 'Invalid GitLab group page.') + const documents: NativeDocument[] = [] + const failures: string[] = [] + let partial = false + let collections = 0 + while ( + projectIndex < projects.length && + kinds.length && + collections < 4 && + documents.length === 0 + ) { + input.signal.throwIfAborted() + const project = projects[projectIndex] + const kind = kinds[kindIndex] + if (!project || !kind) break + try { + const current = await (await session(project)).search({ + ...search, + limit: search.limit, + native: { + provider: 'gitlab', + query: search.query, + ...search.native, + project: String(project.id), + kind: kind as 'code' | 'wiki' | 'issues' | 'merge_requests', + cursor: String(page), + }, + }) + documents.push(...current.documents) + partial ||= Boolean(current.partial) + if (current.nextCursor) page = Number(current.nextCursor) + else { + page = 1 + kindIndex += 1 + if (kindIndex >= kinds.length) { + kindIndex = 0 + projectIndex += 1 + } + } + } catch (error) { + input.signal.throwIfAborted() + if ( + error instanceof NativeSearchError && + ['reconnect', 'rate_limited'].includes(error.status) + ) + throw error + partial = true + failures.push( + error instanceof NativeSearchError + ? error.message + : 'One GitLab collection could not be searched.' + ) + page = 1 + kindIndex += 1 + if (kindIndex >= kinds.length) { + kindIndex = 0 + projectIndex += 1 + } + } + collections += 1 + } + return { + documents, + partial, + ...(projectIndex < projects.length && kinds.length + ? { + nextCursor: `${stored.revision}:${search.limit}:${projectIndex}:${kindIndex}:${page}`, + } + : {}), + message: joinMessages([ + 'Searched configured GitLab group projects with per-project permissions. Continue with the cursor for remaining projects and results.', + ...failures, + ]), + } + }, + } +} diff --git a/apps/sim/lib/sim-search/live/source-settings.ts b/apps/sim/lib/sim-search/live/source-settings.ts index 7cdbee599cb..9c32fdba249 100644 --- a/apps/sim/lib/sim-search/live/source-settings.ts +++ b/apps/sim/lib/sim-search/live/source-settings.ts @@ -26,6 +26,9 @@ const LIVE_SOURCE_FIELDS: Record = { github: ['repository', 'pathPrefix', 'extensions'], gitlab: [ 'host', + 'scope', + 'group', + 'includeSubgroups', 'project', 'contentTypes', 'ref', @@ -100,7 +103,10 @@ export function liveSearchSourceMeta( ), } } - if (!enabled) return meta + if (!enabled) + return meta.configFields.some((field) => field.searchOnly) + ? { ...meta, configFields: meta.configFields.filter((field) => !field.searchOnly) } + : meta const fields = LIVE_SOURCE_FIELDS[meta.id] if (!fields) return meta const descriptions = LIVE_FIELD_DESCRIPTIONS[meta.id] ?? {} diff --git a/apps/sim/lib/sim-search/source-identity.ts b/apps/sim/lib/sim-search/source-identity.ts index 2cba485f134..3e4651f4a2f 100644 --- a/apps/sim/lib/sim-search/source-identity.ts +++ b/apps/sim/lib/sim-search/source-identity.ts @@ -34,6 +34,7 @@ const SOURCE_ADDRESS_FIELDS = new Set([ 'host', 'domain', 'project', + 'group', 'projectKey', 'spaceKey', 'folderId', @@ -79,6 +80,7 @@ const OPAQUE_SOURCE_FIELDS = new Map([ const NAMED_SOURCE_FIELDS = new Map([ ['project', 'project'], + ['group', 'group'], ['projectKey', 'project'], ['spaceKey', 'space'], ['repository', 'repository'],