Close must-have gaps in explorer command - #16

Expand TestFullE2E from 23 to 36 progressive steps:
- Add init command test (create .kbagent/ in sub-directory)
- Add storage upload-table --incremental (verify 8 rows after appending 3)
- Add storage load-file (upload CSV as file, load into table)
- Add config update --merge (deep-merge partial JSON, verify preservation)
- Add config new scaffold generation (keboola.ex-http)
- Add full workspace lifecycle (create/list/detail/password/load/query/delete)
- Add branch merge command (verify URL response, then delete)
- Add permissions list/show/check commands
- Add job list --component-id filter and job detail

Add TestE2ESyncWorkflow: sync init/pull/status/diff/push --dry-run in temp git repo
Add TestE2EToolCommands: tool list + tool call get_buckets (skip if no MCP)

Update CLAUDE.md: add convention #16 requiring E2E tests for new commands,
add test_e2e.py to project structure.

Total: 12 tests, ~2.5 min runtime against real API.

* test: comprehensive E2E test suite for full CLI coverage

Add tests/test_e2e.py with 9 tests covering 23 progressive steps:
- Project CRUD (add/list/status/edit/remove)
- Storage CRUD (bucket/table create, upload CSV, download round-trip, unload, delete)
- Config operations (list/detail/search/update --set/--dry-run/--configuration/delete)
- Component discovery (list with --type filter, detail via AI Service)
- File lifecycle (upload/list/detail/download/tag/delete with dry-run)
- Encrypt values (KBC::ProjectSecure verification)
- Branch lifecycle (list/create/use/reset/delete)
- Sharing list + lineage show (read-only)
- Error handling (invalid token, nonexistent resources, correct exit codes)
- JSON consistency (all read commands return valid JSON, token never leaked)

Verbose output shows exact CLI commands and abbreviated JSON responses.
Resources use e2e-{timestamp} prefix with guaranteed cleanup on failure.

Also: add make test-e2e target, register @pytest.mark.e2e, fix help regex
to support digits in target names, exclude e2e from default make test.

* test: expand E2E coverage to all CLI commands (36 steps)

Expand TestFullE2E from 23 to 36 progressive steps:
- Add init command test (create .kbagent/ in sub-directory)
- Add storage upload-table --incremental (verify 8 rows after appending 3)
- Add storage load-file (upload CSV as file, load into table)
- Add config update --merge (deep-merge partial JSON, verify preservation)
- Add config new scaffold generation (keboola.ex-http)
- Add full workspace lifecycle (create/list/detail/password/load/query/delete)
- Add branch merge command (verify URL response, then delete)
- Add permissions list/show/check commands
- Add job list --component-id filter and job detail

Add TestE2ESyncWorkflow: sync init/pull/status/diff/push --dry-run in temp git repo
Add TestE2EToolCommands: tool list + tool call get_buckets (skip if no MCP)

Update CLAUDE.md: add convention #16 requiring E2E tests for new commands,
add test_e2e.py to project structure.

Total: 12 tests, ~2.5 min runtime against real API.

* docs: add E2E test scenarios reference (docs/e2e-scenarios.md)

Documents all 12 E2E tests (36 main steps) with exact commands,
assertions, and coverage. Includes table of commands NOT covered
with reasons (manage token, interactive prompts, side effects).

* test: add Snowflake transformation job run E2E (steps 28-32)

Full transformation lifecycle:
- Create output bucket + Snowflake SQL transformation config via API
- SQL: SELECT id, name, CAST(value AS INT), CAST(value AS INT)*2 AS doubled_value
- Run transformation with job run --wait --timeout 300
- Verify job detail: status=success, isFinished=true
- Download output table, verify doubled_value == value * 2 for all rows
- Clean up transformation config and output bucket via CLI

Brings main E2E flow from 36 to 41 steps. All 12 tests pass (~3.5 min).

Adds `--new-alias NEW` to `kbagent project edit` so users can rename a
project alias without going through `project remove` + `project add`
(which forces token re-entry). Adds `--dry-run` for read-only preview.
Mirrors the `kbagent config rename` precedent for the on-disk part of
the cascade.

Cascading scope:
- config.json `projects` dict key (`pop(old)` + insert under `new`)
- config.json `default_project` field (when it matched the old alias)
- nested-layout sync directory `<cwd>/<old-alias>/` -> `<cwd>/<new-alias>/`
  (with -2 collision suffix, git-mv-with-shutil-fallback)
- WARNS on `*.lineage.json` -- caches embed alias FQNs and are NOT
  auto-rewritten (partial rewrites are worse than no rewrite)

`--dry-run` (PR keboola#266 review NIT, addressed) previews collision detection,
planned disk-rename method (`git_mv` vs `shutil_move`), and the lineage-
cache warning without mutating any state. Validation errors raise the
same `ConfigError` exit-5 codes as the live path -- callers can rely on
`--dry-run` as a 1:1 pre-flight.

Combined with `--url` and/or `--token` in one call, those mutations
target the new alias post-rename: `kbagent project edit --project foo
--new-alias bar --token NEW` is one atomic operation.

Service / Command:
- `commands/project.py` -- new `--new-alias` and `--dry-run` Typer
  options; human formatter branches on `dry_run` / `old_alias`
- `services/project_service.py` -- `edit_project` accepts `new_alias`,
  `search_root`, `dry_run`; new helpers `_rename_project_alias`,
  `_validate_alias_format`, `_rename_nested_sync_dir`, `_move_directory`,
  `_detect_lineage_cache_warning` for the live path; `_plan_project_alias_rename`
  and `_plan_nested_sync_dir` for the read-only dry-run path

ConfigStore:
- `config_store.py` -- new `rename_project(old, new)` method (atomic
  dict-key swap + `default_project` cascade in one save() call)

Security hardening (from review iter 2):
- Validator regex `[A-Za-z0-9_][A-Za-z0-9_.-]*` plus explicit `..`
  rejection; rejects path traversal, NUL bytes, leading dot/dash,
  whitespace, and characters outside the slug alphabet. Stricter than
  `project add`'s no-op check; rationale is the rename's filesystem
  interaction (alias becomes a directory name).
- `search_root` resolved via `Path.resolve()` once before the disk
  rename to collapse symlinks; closes a malicious-cwd vector.
- Disk rename failures (`OSError`) trigger a config rollback so config
  and disk never end up out of sync; rollback's own failure is
  suppressed via `contextlib.suppress` so the original error wins.
- Lineage cache scan depth-capped at 2 levels (top + `*/` + `*/*/`)
  to bound cost when search_root is a deep tree.

Tests: 38 new (32 service + 6 CLI). Pin alias-key swap, collision
rejection, default_project cascade, sync-dir disk rename, no-sync-dir
no-op, sync-dir collision -2 suffix, combined edit-and-rename, no-op
on same-alias-only, parametrized 9-input path-traversal validator,
legal slug shapes accepted, OS failure rolls config back, rollback
failure surfaces original error, symlink target collision triggers
suffix bump, dry-run no-mutation happy path, dry-run collision still
raises, dry-run format validation still raises, dry-run predicts disk
method without touching disk, dry-run human DRY-RUN label, dry-run
JSON planned-block shape.

E2E: `tests/test_e2e.py::_test_project_edit_and_remove` extended with
a `--dry-run` preview (planned-block assertion) followed by a live
`--new-alias` round-trip (rename + reverse-rename to baseline) before
the existing `--url` step. Pinned by Padak's PR keboola#266 review BLOCKING --
every CLI command must have E2E coverage per CONTRIBUTING.md /
convention keboola#16.

Sync map updates:
- AGENT_CONTEXT (commands/context.py) -- new flags mentioned
- CLAUDE.md `## All CLI Commands` -- same wording
- commands-reference.md -- expanded with cascade scope + dry-run
- gotchas.md -- new `(since v0.30.7)` entry on lineage cache rebuild
- keboola-expert.md -- VERSION GATE clause + tool selection matrix row

Live-validated against project 1143 (`99_Playground_Max`,
europe-west3.gcp.keboola.com): rename to `playground` + reverse rename
to baseline; nested sync dir moved on disk; default_project cascaded;
all error paths produce correct ConfigError exit-5 messages.

Three review iterations on the original PR: self -> independent ->
convergence; zero material findings on the convergence pass. Padak's
post-merge review found 1 BLOCKING (E2E coverage) + 1 NIT (`--dry-run`);
both addressed in this iteration.

Rebased onto upstream/main after Padak shipped 0.30.4 (keboola#268),
0.30.5 (keboola#272), and 0.30.6 (keboola#273) since the original PR opened.
This PR ships as 0.30.7.

…un (#266)

Adds `--new-alias NEW` to `kbagent project edit` so users can rename a
project alias without going through `project remove` + `project add`
(which forces token re-entry). Adds `--dry-run` for read-only preview.
Mirrors the `kbagent config rename` precedent for the on-disk part of
the cascade.

Cascading scope:
- config.json `projects` dict key (`pop(old)` + insert under `new`)
- config.json `default_project` field (when it matched the old alias)
- nested-layout sync directory `<cwd>/<old-alias>/` -> `<cwd>/<new-alias>/`
  (with -2 collision suffix, git-mv-with-shutil-fallback)
- WARNS on `*.lineage.json` -- caches embed alias FQNs and are NOT
  auto-rewritten (partial rewrites are worse than no rewrite)

`--dry-run` (PR #266 review NIT, addressed) previews collision detection,
planned disk-rename method (`git_mv` vs `shutil_move`), and the lineage-
cache warning without mutating any state. Validation errors raise the
same `ConfigError` exit-5 codes as the live path -- callers can rely on
`--dry-run` as a 1:1 pre-flight.

Combined with `--url` and/or `--token` in one call, those mutations
target the new alias post-rename: `kbagent project edit --project foo
--new-alias bar --token NEW` is one atomic operation.

Service / Command:
- `commands/project.py` -- new `--new-alias` and `--dry-run` Typer
  options; human formatter branches on `dry_run` / `old_alias`
- `services/project_service.py` -- `edit_project` accepts `new_alias`,
  `search_root`, `dry_run`; new helpers `_rename_project_alias`,
  `_validate_alias_format`, `_rename_nested_sync_dir`, `_move_directory`,
  `_detect_lineage_cache_warning` for the live path; `_plan_project_alias_rename`
  and `_plan_nested_sync_dir` for the read-only dry-run path

ConfigStore:
- `config_store.py` -- new `rename_project(old, new)` method (atomic
  dict-key swap + `default_project` cascade in one save() call)

Security hardening (from review iter 2):
- Validator regex `[A-Za-z0-9_][A-Za-z0-9_.-]*` plus explicit `..`
  rejection; rejects path traversal, NUL bytes, leading dot/dash,
  whitespace, and characters outside the slug alphabet. Stricter than
  `project add`'s no-op check; rationale is the rename's filesystem
  interaction (alias becomes a directory name).
- `search_root` resolved via `Path.resolve()` once before the disk
  rename to collapse symlinks; closes a malicious-cwd vector.
- Disk rename failures (`OSError`) trigger a config rollback so config
  and disk never end up out of sync; rollback's own failure is
  suppressed via `contextlib.suppress` so the original error wins.
- Lineage cache scan depth-capped at 2 levels (top + `*/` + `*/*/`)
  to bound cost when search_root is a deep tree.

Tests: 38 new (32 service + 6 CLI). Pin alias-key swap, collision
rejection, default_project cascade, sync-dir disk rename, no-sync-dir
no-op, sync-dir collision -2 suffix, combined edit-and-rename, no-op
on same-alias-only, parametrized 9-input path-traversal validator,
legal slug shapes accepted, OS failure rolls config back, rollback
failure surfaces original error, symlink target collision triggers
suffix bump, dry-run no-mutation happy path, dry-run collision still
raises, dry-run format validation still raises, dry-run predicts disk
method without touching disk, dry-run human DRY-RUN label, dry-run
JSON planned-block shape.

E2E: `tests/test_e2e.py::_test_project_edit_and_remove` extended with
a `--dry-run` preview (planned-block assertion) followed by a live
`--new-alias` round-trip (rename + reverse-rename to baseline) before
the existing `--url` step. Pinned by Padak's PR #266 review BLOCKING --
every CLI command must have E2E coverage per CONTRIBUTING.md /
convention #16.

Sync map updates:
- AGENT_CONTEXT (commands/context.py) -- new flags mentioned
- CLAUDE.md `## All CLI Commands` -- same wording
- commands-reference.md -- expanded with cascade scope + dry-run
- gotchas.md -- new `(since v0.30.7)` entry on lineage cache rebuild
- keboola-expert.md -- VERSION GATE clause + tool selection matrix row

Live-validated against project 1143 (`99_Playground_Max`,
europe-west3.gcp.keboola.com): rename to `playground` + reverse rename
to baseline; nested sync dir moved on disk; default_project cascaded;
all error paths produce correct ConfigError exit-5 messages.

Three review iterations on the original PR: self -> independent ->
convergence; zero material findings on the convergence pass. Padak's
post-merge review found 1 BLOCKING (E2E coverage) + 1 NIT (`--dry-run`);
both addressed in this iteration.

Rebased onto upstream/main after Padak shipped 0.30.4 (#268),
0.30.5 (#272), and 0.30.6 (#273) since the original PR opened.
This PR ships as 0.30.7.

…s in the metastore)

Teach kbagent the new `semantic-reference-data` metastore type — a
per-dimension member store (one record per dimension, members in a
`members[]` array). Driving use case: hold a Chart of Accounts (the account
list + all attributes) in the semantic layer instead of a hardcoded Storage
table. Pairs with keboola/go-monorepo#533 (the metastore schema side).

CLI sub-app `kbagent semantic-layer reference-data` (alias `kbagent sl`):
- `list`   — dimension summaries (id, dimension, member_count)        [read]
- `get`    — one record + members, by --id or --model+--dimension     [read]
- `set`    — create-or-replace from a JSON members file; idempotent on
             (model, dimension): existing -> PUT/revision++, else POST [write]
- `delete` — remove by UUID (--yes gated)                       [destructive]

Implementation:
- metastore_client: register the type in SemanticType / SEMANTIC_TYPES;
  add a `put_item` verb so `set` uses the metastore's revisioned PUT
  (preserves history) instead of DELETE+POST.
- Logic in services/_semantic_layer_reference_data.py (run_* helpers,
  open_client thunk); the service exposes thin delegators, keeping
  semantic_layer_service.py under the 1500-LOC ceiling.
- serve REST router: GET /reference-data, GET /reference-data/{id},
  PUT /reference-data, DELETE /reference-data/{id} (1:1 CLI->HTTP).
- permissions + hints registered for all four leaves.

Deliberately self-contained: reference-data is NOT AI-generated, so it is
kept OUT of build/export/diff/cascade/PUSH_ORDER — zero blast radius on the
existing model flows.

Docs synced (Plugin synchronization map): context.py AGENT_CONTEXT,
CLAUDE.md command list, commands-reference.md, gotchas.md, SKILL.md
(triggers + auto-generated table). keboola-expert.md intentionally NOT
extended — it is at its enforced 62 KB prompt-budget ceiling.

Tests: service CRUD + permission registry + CLI + metastore_client
TestPutItem (httpx_mock) + serve-router parity (mocked registry). **E2E**
(convention keboola#16): CLI lifecycle hop in test_e2e.py
(TestE2ESemanticLayerLifecycle.test_semantic_layer_reference_data_roundtrip)
and 5 HTTP-route hops in test_server_semantic_layer_routes_e2e.py — both
run green live against project 1143 (99_Playground_Max), set/list/get/
replace/delete with model bootstrap + cleanup + residue scan. Full unit
suite green (3871 passed); ruff/format/ty clean for changed files.

Version bump + changelog entry are release-time per CONTRIBUTING.md.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

…s in the metastore)

Teach kbagent the new `semantic-reference-data` metastore type — a
per-dimension member store (one record per dimension, members in a
`members[]` array). Driving use case: hold a Chart of Accounts (the account
list + all attributes) in the semantic layer instead of a hardcoded Storage
table. Pairs with keboola/go-monorepo#533 (the metastore schema side).

CLI sub-app `kbagent semantic-layer reference-data` (alias `kbagent sl`):
- `list`   — dimension summaries (id, dimension, member_count)        [read]
- `get`    — one record + members, by --id or --dimension             [read]
- `set`    — create-or-replace from a JSON members file; idempotent on
             the dimension (project-wide lookup): existing -> PUT/
             revision++, else POST                                    [write]
- `delete` — remove by UUID (--yes gated)                       [destructive]

The dimension name is the metastore envelope `name`, unique per project per
type, so it is the record's project-unique key: `set`/`get` resolve by
dimension project-wide and `set` stays idempotent regardless of which
`--model` is passed (the resolved model is stored on the record, never the
lookup key). `get` rejects `--id` and `--dimension` together (exit 2).

Implementation:
- metastore_client: register the type in SemanticType / SEMANTIC_TYPES;
  add a `put_item` verb so `set` uses the metastore's revisioned PUT
  (preserves history) instead of DELETE+POST.
- Logic in services/_semantic_layer_reference_data.py (run_* helpers,
  open_client thunk); the service exposes thin delegators, keeping
  semantic_layer_service.py under the 1500-LOC ceiling.
- serve REST router: GET /reference-data, GET /reference-data/{id},
  PUT /reference-data, DELETE /reference-data/{id} (1:1 CLI->HTTP).
- permissions registered for all four leaves. No `--hint` definitions:
  hints are DEPRECATED per CONTRIBUTING.md (use `kbagent serve`), so the
  programmatic surface is the serve REST router only.

Deliberately self-contained: reference-data is NOT AI-generated, so it is
kept OUT of build/export/diff/cascade/PUSH_ORDER — zero blast radius on the
existing model flows.

Docs synced (Plugin synchronization map): context.py AGENT_CONTEXT,
CLAUDE.md command list, commands-reference.md, gotchas.md, SKILL.md
(triggers + auto-generated table), all tagged since v0.55.0 to avoid the
0.54.0 version collision with the OAuth login PR. keboola-expert.md
intentionally NOT extended — it is at its enforced 62 KB prompt-budget
ceiling.

Tests: service CRUD + cross-model idempotency regression + permission
registry + CLI (incl. --id/--dimension mutual-exclusion) + metastore_client
TestPutItem (httpx_mock) + serve-router parity (mocked registry). E2E
(convention keboola#16): CLI lifecycle hop in test_e2e.py and 5 HTTP-route hops in
test_server_semantic_layer_routes_e2e.py — both run green live against
project 1143 (99_Playground_Max), set/list/get/replace/delete with model
bootstrap + cleanup + residue scan. Full unit suite green; ruff/format/ty
clean.

Version bump + changelog entry are release-time per CONTRIBUTING.md.

…s in the metastore)

Teach kbagent the new `semantic-reference-data` metastore type — a
per-dimension member store (one record per dimension, members in a
`members[]` array). Driving use case: hold a Chart of Accounts (the account
list + all attributes) in the semantic layer instead of a hardcoded Storage
table. Pairs with keboola/go-monorepo#533 (the metastore schema side).

CLI sub-app `kbagent semantic-layer reference-data` (alias `kbagent sl`):
- `list`   — dimension summaries (id, dimension, member_count)        [read]
- `get`    — one record + members, by --id or --dimension             [read]
- `set`    — create-or-replace from a JSON members file; idempotent on
             the dimension (project-wide lookup): existing -> PUT/
             revision++, else POST                                    [write]
- `delete` — remove by UUID (--yes gated)                       [destructive]

The dimension name is the metastore envelope `name`, unique per project per
type, so it is the record's project-unique key: `set`/`get` resolve by
dimension project-wide and `set` stays idempotent regardless of which
`--model` is passed (the resolved model is stored on the record, never the
lookup key). `get` rejects `--id` and `--dimension` together (exit 2).

Implementation:
- metastore_client: register the type in SemanticType / SEMANTIC_TYPES;
  add a `put_item` verb so `set` uses the metastore's revisioned PUT
  (preserves history) instead of DELETE+POST.
- Logic in services/_semantic_layer_reference_data.py (run_* helpers,
  open_client thunk); the service exposes thin delegators, keeping
  semantic_layer_service.py under the 1500-LOC ceiling.
- serve REST router: GET /reference-data, GET /reference-data/{id},
  PUT /reference-data, DELETE /reference-data/{id} (1:1 CLI->HTTP).
- permissions registered for all four leaves. No `--hint` definitions:
  hints are DEPRECATED per CONTRIBUTING.md (use `kbagent serve`), so the
  programmatic surface is the serve REST router only.

Deliberately self-contained: reference-data is NOT AI-generated, so it is
kept OUT of build/export/diff/cascade/PUSH_ORDER — zero blast radius on the
existing model flows.

Docs synced (Plugin synchronization map): context.py AGENT_CONTEXT,
CLAUDE.md command list, commands-reference.md, gotchas.md, SKILL.md
(triggers + auto-generated table), all tagged since v0.55.0 to avoid the
0.54.0 version collision with the OAuth login PR. keboola-expert.md
intentionally NOT extended — it is at its enforced 62 KB prompt-budget
ceiling.

Tests: service CRUD + cross-model idempotency regression + permission
registry + CLI (incl. --id/--dimension mutual-exclusion) + metastore_client
TestPutItem (httpx_mock) + serve-router parity (mocked registry). E2E
(convention keboola#16): CLI lifecycle hop in test_e2e.py and 5 HTTP-route hops in
test_server_semantic_layer_routes_e2e.py — both run green live against
project 1143 (99_Playground_Max), set/list/get/replace/delete with model
bootstrap + cleanup + residue scan. Full unit suite green; ruff/format/ty
clean.

Version bump + changelog entry are release-time per CONTRIBUTING.md.

…s in the metastore) (#379)

Teach kbagent the new `semantic-reference-data` metastore type — a
per-dimension member store (one record per dimension, members in a
`members[]` array). Driving use case: hold a Chart of Accounts (the account
list + all attributes) in the semantic layer instead of a hardcoded Storage
table. Pairs with keboola/go-monorepo#533 (the metastore schema side).

CLI sub-app `kbagent semantic-layer reference-data` (alias `kbagent sl`):
- `list`   — dimension summaries (id, dimension, member_count)        [read]
- `get`    — one record + members, by --id or --dimension             [read]
- `set`    — create-or-replace from a JSON members file; idempotent on
             the dimension (project-wide lookup): existing -> PUT/
             revision++, else POST                                    [write]
- `delete` — remove by UUID (--yes gated)                       [destructive]

The dimension name is the metastore envelope `name`, unique per project per
type, so it is the record's project-unique key: `set`/`get` resolve by
dimension project-wide and `set` stays idempotent regardless of which
`--model` is passed (the resolved model is stored on the record, never the
lookup key). `get` rejects `--id` and `--dimension` together (exit 2).

Implementation:
- metastore_client: register the type in SemanticType / SEMANTIC_TYPES;
  add a `put_item` verb so `set` uses the metastore's revisioned PUT
  (preserves history) instead of DELETE+POST.
- Logic in services/_semantic_layer_reference_data.py (run_* helpers,
  open_client thunk); the service exposes thin delegators, keeping
  semantic_layer_service.py under the 1500-LOC ceiling.
- serve REST router: GET /reference-data, GET /reference-data/{id},
  PUT /reference-data, DELETE /reference-data/{id} (1:1 CLI->HTTP).
- permissions registered for all four leaves. No `--hint` definitions:
  hints are DEPRECATED per CONTRIBUTING.md (use `kbagent serve`), so the
  programmatic surface is the serve REST router only.

Deliberately self-contained: reference-data is NOT AI-generated, so it is
kept OUT of build/export/diff/cascade/PUSH_ORDER — zero blast radius on the
existing model flows.

Docs synced (Plugin synchronization map): context.py AGENT_CONTEXT,
CLAUDE.md command list, commands-reference.md, gotchas.md, SKILL.md
(triggers + auto-generated table), all tagged since v0.55.0 to avoid the
0.54.0 version collision with the OAuth login PR. keboola-expert.md
intentionally NOT extended — it is at its enforced 62 KB prompt-budget
ceiling.

Tests: service CRUD + cross-model idempotency regression + permission
registry + CLI (incl. --id/--dimension mutual-exclusion) + metastore_client
TestPutItem (httpx_mock) + serve-router parity (mocked registry). E2E
(convention #16): CLI lifecycle hop in test_e2e.py and 5 HTTP-route hops in
test_server_semantic_layer_routes_e2e.py — both run green live against
project 1143 (99_Playground_Max), set/list/get/replace/delete with model
bootstrap + cleanup + residue scan. Full unit suite green; ruff/format/ty
clean.

Version bump + changelog entry are release-time per CONTRIBUTING.md.

Co-authored-by: Max Ottomansky <[email protected]>