[KYLIN-6087] Support built-in auto scheduled segment build with model-level config and scheduler by Mrhs121 · Pull Request #2347 · apache/kylin

Mrhs121 · 2026-02-09T12:26:19Z

No description provided.

Mrhs121 · 2026-02-10T01:38:40Z

retest this please

Mrhs121 · 2026-02-10T10:54:56Z

@jlfsdtc PTAL, Thanks

jlfsdtc · 2026-02-12T08:44:10Z

Please add "dev design" and create an Apache issue at the same time. @Mrhs121

fishcus · 2026-07-23T05:55:43Z

+        }
+        val parts = key.split("/", 2);
+        if (parts.length != 2) {
+            return;


add error log, identify the abnormal key

fishcus · 2026-07-23T06:00:19Z

+    @Getter
+    private final AtomicInteger schedulerModelCount = new AtomicInteger(0);
+
+    @Scheduled(cron = "*/30 * * * * ?")


Why is it necessary to schedule this operation every 30 minutes? or this crontab expression should be configurable.

fishcus · 2026-07-23T06:09:26Z

+            val projectName = project.getName();
+            val modelManager = NDataModelManager.getInstance(KylinConfig.readSystemKylinConfig(), projectName);
+            for (NDataModel model : modelManager.listAllModels()) {
+                val segmentConfig = model.getSegmentConfig();


skip the offline model or broken model？

fishcus · 2026-07-23T06:12:20Z

+
+    private void startCron(String key, String cron) {
+        stopCron(key);
+        checkSchedulerThreadPoolSize();


check every time of startCron or every time of schedule？

fishcus · 2026-07-23T06:24:05Z

+        if (autoSegmentBuild == null || !autoSegmentBuild.isEnabled()) {
+            return;
+        }
+        if (model.isStreaming() || model.isMultiPartitionModel()


model‘s status has been checked when walking through the model list in function reconcile，combine these into one.

fishcus · 2026-07-23T06:28:39Z

+        JobTypeEnum[] buildJobTypes = JobTypeEnum.getJobTypeByCategory(JobTypeEnum.Category.BUILD)
+                .toArray(new JobTypeEnum[0]);
+        List<AbstractExecutable> jobs = executableManager.listExecByModelAndStatus(modelId,
+                ExecutableState::isProgressing, buildJobTypes);


hasRunningBuildJob of the same segment or any segment？

…el config and scheduler

Copilot

Pull request overview

Adds model-level configuration and a built-in scheduler to automatically build segments on a daily schedule, and exposes the configuration through REST + Kystudio UI so users can configure/observe auto scheduled builds per model.

Changes:

Introduces AutoSegmentBuildConfig and wires it into SegmentConfig, project defaults, model config request/response, and open API mapping.
Adds AutoBuildSegmentScheduler to periodically dispatch scheduled segment build jobs using project time zones.
Extends ModelBuildService to support scheduler-triggered incremental segment builds with a configurable submitter, plus unit tests for validation and scheduler behavior.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/modeling-service/src/test/java/org/apache/kylin/rest/service/ModelServiceTest.java	Adds tests for auto-segment-build config persistence and parameter validation.
src/modeling-service/src/main/java/org/apache/kylin/rest/service/ModelService.java	Adds validation for `auto_segment_build` and returns/persists it in model config APIs.
src/modeling-service/src/main/java/org/apache/kylin/rest/service/ModelBuildService.java	Adds scheduler entrypoint and propagates `submitter` into job params.
src/modeling-service/src/main/java/org/apache/kylin/rest/response/ModelConfigResponse.java	Exposes `auto_segment_build` in REST response.
src/modeling-service/src/main/java/org/apache/kylin/rest/request/ModelConfigRequest.java	Accepts `auto_segment_build` in REST request.
src/metadata-server/src/main/java/org/apache/kylin/rest/controller/open/OpenModelController.java	Maps `auto_segment_build` between open API model config response/request.
src/data-loading-service/src/test/java/org/apache/kylin/rest/scheduler/AutoBuildSegmentSchedulerTest.java	Adds scheduler unit tests (timezone handling, once-per-window dispatch, skip conditions).
src/data-loading-service/src/main/java/org/apache/kylin/rest/scheduler/AutoBuildSegmentScheduler.java	New scheduler component that dispatches auto segment builds per project/model.
src/core-metadata/src/test/java/org/apache/kylin/metadata/cube/model/NSegmentConfigHelperTest.java	Updates constructor usage due to new `SegmentConfig` field.
src/core-metadata/src/main/java/org/apache/kylin/metadata/project/ProjectInstance.java	Adds project default `auto_segment_build` config inside default `SegmentConfig`.
src/core-metadata/src/main/java/org/apache/kylin/metadata/model/SegmentConfig.java	Adds `auto_segment_build` field to segment config.
src/core-metadata/src/main/java/org/apache/kylin/metadata/model/AutoSegmentBuildConfig.java	New config model for auto scheduled segment builds.
src/core-common/src/main/resources/kylin-defaults0.properties	Adds default cron property for scheduler dispatcher.
kystudio/src/components/setting/SettingModel/SettingModel.vue	Adds UI to edit/delete/display auto segment build config.
kystudio/src/components/setting/SettingModel/locales.js	Adds UI strings for auto segment build.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+          <el-form-item :label="$t('autoSegmentBuildRangeEnd')">
+            <el-time-picker
+              style="width: 180px;"
+              size="small"
+              format="HH:mm:ss"
+              value-format="HH:mm:ss"
+              v-model="modelSettingForm.autoSegmentBuild.data_range_end_time">
+            </el-time-picker>


+import java.time.Duration;
+import java.time.Instant;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+import java.time.LocalTime;


+        if (StringUtils.isBlank(autoSegmentBuild.getTriggerTime())) {
+            log.warn("Skip auto build segment because trigger_time is blank, project: {}, model: {}", project,
+                    model.getUuid());
+            return;
+        }


+    private JobInfoResponse incrementBuildSegmentsInternal(IncrementBuildSegmentParams params, String submitter,
+            boolean checkPermission) throws Exception {
        String project = params.getProject();
-        aclEvaluate.checkProjectOperationPermission(project);
-        checkModelPermission(project, params.getModelId());
+        if (checkPermission) {
+            aclEvaluate.checkProjectOperationPermission(project);
+            checkModelPermission(project, params.getModelId());
+        }
        val modelManager = getManager(NDataModelManager.class, project);


+                || StringUtils.isBlank(autoSegmentBuild.getDataRangeStartTime())
+                || StringUtils.isBlank(autoSegmentBuild.getDataRangeEndTime())
+                || autoSegmentBuild.getLogicalDateOffsetDays() == null) {
+            throw new KylinException(INVALID_PARAMETER, "Invalid auto_segment_build config.");


jlfsdtc · 2026-08-14T06:04:36Z

+    private final AtomicBoolean dispatching = new AtomicBoolean(false);
+    private final AtomicReference<Instant> lastDispatchTime = new AtomicReference<>();
+
+    @Scheduled(cron = "${kylin.model.auto-segment-build.dispatcher-cron:*/30 * * * * ?}")


This indicates that kylin.model.auto-segment-build.dispatcher-cron is a system-level parameter that requires a system restart after each modification. Is it possible to adjust this parameter to the project level or model level?

jlfsdtc · 2026-08-14T06:10:14Z

+        return autoSegmentBuild != null && autoSegmentBuild.isEnabled() ? autoSegmentBuild : null;
+    }
+
+    @VisibleForTesting


This annotation is only added to the code used for unit testing (UT), and no annotation is required for the methods employed in production code.

jlfsdtc · 2026-08-14T06:10:45Z

+        return scheduledTime;
+    }
+
+    @VisibleForTesting


This annotation is only added to the code used for unit testing (UT), and no annotation is required for the methods employed in production code.

jlfsdtc · 2026-08-14T06:11:06Z

+        }
+    }
+
+    @VisibleForTesting


This annotation is only added to the code used for unit testing (UT), and no annotation is required for the methods employed in production code.

jlfsdtc · 2026-08-14T06:11:42Z

+        return previousTime;
+    }
+
+    @VisibleForTesting


This annotation is only added to the code used for unit testing (UT), and no annotation is required for the methods employed in production code.

Mrhs121 changed the title ~~Support built-in auto scheduled segment build with model-level config and scheduler~~ [KYLIN-6087] Support built-in auto scheduled segment build with model-level config and scheduler Feb 26, 2026

fishcus reviewed Jul 23, 2026

View reviewed changes

Mrhs121 added 4 commits August 12, 2026 19:56

feat(model): add built-in auto scheduled segment build with model-lev…

2d071b3

…el config and scheduler

feat(model): add built-in auto scheduled segment build with model-lev…

cb8b95c

…el config and scheduler

fix forbidden check

61522d4

refactor(model): use dispatcher for automatic segment builds

09f230d

Mrhs121 force-pushed the scs branch from 4d94c3c to 09f230d Compare August 12, 2026 13:12

jlfsdtc requested a lite review from Copilot August 14, 2026 06:02

Copilot started reviewing on behalf of jlfsdtc August 14, 2026 06:02 View session

Copilot AI reviewed Aug 14, 2026

View reviewed changes

jlfsdtc requested changes Aug 14, 2026

View reviewed changes

Uh oh!

Conversation

Mrhs121 commented Feb 9, 2026

Uh oh!

Mrhs121 commented Feb 10, 2026

Uh oh!

Mrhs121 commented Feb 10, 2026

Uh oh!

jlfsdtc commented Feb 12, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fishcus Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

fishcus Jul 23, 2026 •

edited

Loading