webapi-managed datasets + cache_id alignment#2459
Open
p-hoffmann wants to merge 18 commits into
Open
Conversation
p-hoffmann
requested review from
KangJingA,
SantanM,
Zhimin-arya,
alicia-koh,
csafreen,
hengxian-jiang and
suwarnoong
as code owners
Contributor
There was a problem hiding this comment.
Pull request overview
Introduces “WebAPI-managed” datasets and aligns cacheId usage end-to-end (portal API, UI actions, demo tooling, and Prefect flows) so analytics/flows target the correct TREX/DuckDB catalog.
Changes:
- Adds a System Admin UI action + dialog to “Convert to WebAPI-managed” and wires it to a new backend endpoint that merges a snapshot/cache row into the source row.
- Aligns dataset metadata and flow execution to use
cacheIdas the catalog/alias for qualified reads (DQD/DC/Achilles + TREX DAO JDBC connection). - Updates demo/dataset creation paths to reflect the new model (visibility defaulting, cache-step removal, response shape adjustments), plus an e2e check that the new UI bundle is served.
Reviewed changes
Copilot reviewed 26 out of 26 changed files in this pull request and generated 15 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/e2e/tests/05-datasets/verify-new-ui-served.spec.ts | Adds an e2e check that the portal serves the new bundle and includes the new action/dialog strings. |
| plugins/ui/apps/portal/src/types/study.ts | Extends dataset/action typings with webapi dataset type and transform-to-webapi action. |
| plugins/ui/apps/portal/src/plugins/SystemAdmin/StudyOverview/TransformToWebApiDialog/TransformToWebApiDialog.tsx | New confirmation dialog that calls the backend transform endpoint and shows feedback. |
| plugins/ui/apps/portal/src/plugins/SystemAdmin/StudyOverview/TransformToWebApiDialog/TransformToWebApiDialog.scss | Styling for the new transform dialog. |
| plugins/ui/apps/portal/src/plugins/SystemAdmin/StudyOverview/StudyOverview.tsx | Wires the dialog into StudyOverview and includes webapi in dataset grouping logic. |
| plugins/ui/apps/portal/src/plugins/SystemAdmin/StudyOverview/ActionSelector/ActionSelector.tsx | Adds the new action to the selector and hooks it to a handler. |
| plugins/ui/apps/portal/src/contexts/app-context/states/translation-state.ts | Adds default i18n strings for the new action and dialog copy. |
| plugins/ui/apps/portal/src/constant.ts | Adds action/tab/feature mapping for webapi datasets and the new transform action. |
| plugins/ui/apps/portal/src/axios/system-portal.ts | Adds systemPortal.transformToWebApi() client call. |
| plugins/ui/apps/portal/public/translations/zh.json | Adds translation keys for the new action/dialog (currently English text). |
| plugins/ui/apps/portal/public/translations/de.json | Adds translation keys for the new action/dialog (currently English text). |
| plugins/functions/portal/src/dataset/repository/dataset.repository.ts | Adds updateRowShape() helper used during transformation. |
| plugins/functions/portal/src/dataset/query/dataset-query.service.ts | Includes dataset.cacheId in dataset list/query selection. |
| plugins/functions/portal/src/dataset/dataset.controller.ts | Adds POST :id/transform-to-webapi endpoint. |
| plugins/functions/portal/src/dataset/command/dataset-command.service.ts | Implements transformToWebApi() merge/delete logic; aligns snapshot cacheId with source. |
| plugins/functions/portal/src/dataset/command/dataset-command.service.spec.ts | Adds unit tests covering the transform command. |
| plugins/functions/package.json | Updates demo dataset visibility default. |
| plugins/functions/demo/services/DemoService.ts | Adjusts demo dataset lookup and downstream flows to use id (not cacheId) as dataset identifier. |
| plugins/functions/demo/controllers/DemoController.ts | Removes demo “create cache” steps from the demo workflow. |
| plugins/functions/dataset/index.ts | Changes dataset creation to produce webapi type datasets and removes cache dataset creation/return fields. |
| plugins/flows/base/dqd_plugin/flow.py | Qualifies DQD schema reads against cacheId (catalog) instead of databaseCode. |
| plugins/flows/base/data_characterization_plugin/utils.py | Expands schema-name validation and qualifies TREX reads against cacheId. |
| plugins/flows/base/data_characterization_plugin/flow.py | Qualifies schema for TREX reads; forwards cacheId into Achilles execution. |
| plugins/flows/base/data_characterization_plugin/execute_achilles.R | Forces pooled JDBC connections onto the cache catalog via USE "<cacheId>". |
| plugins/flows/_shared_flow_utils/dao/trexdao.py | Uses cache_id as JDBC dbname to match Python-side connection behavior. |
| plugins/flows/_shared_flow_utils/create_dataset_tasks.py | Allows underscore-leading identifiers and optional catalog.schema patterns for safety checks. |
p-hoffmann
force-pushed
the
p-hoffmann/cache2
branch
from
a4d194a to
d696f1e
Compare
p-hoffmann
requested review from
LSriragavan,
khairul-syazwan and
maggie-li-yd
as code owners
…nto p-hoffmann/cache2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merge Checklist
Please cross check this list if additions / modifications needs to be done on top of your core changes and tick them off. Reviewer can as well glance through and help the developer if something is missed out.
developbranch)