Configure OpenLineage if present in Spark instrumentation#8541
Merged
Conversation
Contributor
|
Hi! 👋 Thanks for your pull request! 🎉 To help us review it, please make sure to:
If you need help, please check our contributing guidelines. |
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 65 metrics, 6 unstable metrics. Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.48.0-SNAPSHOT~83cf093182, baseline=1.48.0-SNAPSHOT~32046a39ed
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.057 s) : 0, 1056860
Total [baseline] (8.756 s) : 0, 8756395
Agent [candidate] (1.052 s) : 0, 1052036
Total [candidate] (8.669 s) : 0, 8668550
section iast
Agent [baseline] (1.179 s) : 0, 1179159
Total [baseline] (9.25 s) : 0, 9249768
Agent [candidate] (1.18 s) : 0, 1180295
Total [candidate] (9.241 s) : 0, 9240989
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.18 s) : 0, 1179974
Total [baseline] (9.193 s) : 0, 9193198
Agent [candidate] (1.189 s) : 0, 1189252
Total [candidate] (9.257 s) : 0, 9257106
section iast_TELEMETRY_OFF
Agent [baseline] (1.175 s) : 0, 1175267
Total [baseline] (9.225 s) : 0, 9224944
Agent [candidate] (1.18 s) : 0, 1180245
Total [candidate] (9.235 s) : 0, 9234735
gantt
title insecure-bank - break down per module: candidate=1.48.0-SNAPSHOT~83cf093182, baseline=1.48.0-SNAPSHOT~32046a39ed
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (722.977 ms) : 0, 722977
BytebuddyAgent [candidate] (719.423 ms) : 0, 719423
GlobalTracer [baseline] (241.072 ms) : 0, 241072
GlobalTracer [candidate] (239.602 ms) : 0, 239602
AppSec [baseline] (54.787 ms) : 0, 54787
AppSec [candidate] (54.277 ms) : 0, 54277
Debugger [baseline] (4.466 ms) : 0, 4466
Debugger [candidate] (4.411 ms) : 0, 4411
Remote Config [baseline] (695.648 µs) : 0, 696
Remote Config [candidate] (698.178 µs) : 0, 698
Telemetry [baseline] (12.221 ms) : 0, 12221
Telemetry [candidate] (13.046 ms) : 0, 13046
section iast
BytebuddyAgent [baseline] (837.265 ms) : 0, 837265
BytebuddyAgent [candidate] (838.061 ms) : 0, 838061
GlobalTracer [baseline] (229.65 ms) : 0, 229650
GlobalTracer [candidate] (229.073 ms) : 0, 229073
IAST [baseline] (22.626 ms) : 0, 22626
IAST [candidate] (23.535 ms) : 0, 23535
AppSec [baseline] (55.637 ms) : 0, 55637
AppSec [candidate] (55.685 ms) : 0, 55685
Debugger [baseline] (4.134 ms) : 0, 4134
Debugger [candidate] (4.173 ms) : 0, 4173
Remote Config [baseline] (609.876 µs) : 0, 610
Remote Config [candidate] (610.992 µs) : 0, 611
Telemetry [baseline] (8.752 ms) : 0, 8752
Telemetry [candidate] (8.808 ms) : 0, 8808
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (837.669 ms) : 0, 837669
BytebuddyAgent [candidate] (845.656 ms) : 0, 845656
GlobalTracer [baseline] (229.522 ms) : 0, 229522
GlobalTracer [candidate] (230.526 ms) : 0, 230526
IAST [baseline] (22.735 ms) : 0, 22735
IAST [candidate] (22.921 ms) : 0, 22921
AppSec [baseline] (56.071 ms) : 0, 56071
AppSec [candidate] (56.053 ms) : 0, 56053
Debugger [baseline] (4.159 ms) : 0, 4159
Debugger [candidate] (4.134 ms) : 0, 4134
Remote Config [baseline] (619.782 µs) : 0, 620
Remote Config [candidate] (608.445 µs) : 0, 608
Telemetry [baseline] (8.746 ms) : 0, 8746
Telemetry [candidate] (8.735 ms) : 0, 8735
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (833.838 ms) : 0, 833838
BytebuddyAgent [candidate] (839.013 ms) : 0, 839013
GlobalTracer [baseline] (229.238 ms) : 0, 229238
GlobalTracer [candidate] (229.025 ms) : 0, 229025
IAST [baseline] (22.287 ms) : 0, 22287
IAST [candidate] (22.331 ms) : 0, 22331
AppSec [baseline] (56.115 ms) : 0, 56115
AppSec [candidate] (55.92 ms) : 0, 55920
Debugger [baseline] (4.188 ms) : 0, 4188
Debugger [candidate] (4.163 ms) : 0, 4163
Remote Config [baseline] (616.429 µs) : 0, 616
Remote Config [candidate] (617.675 µs) : 0, 618
Telemetry [baseline] (8.65 ms) : 0, 8650
Telemetry [candidate] (8.664 ms) : 0, 8664
Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.48.0-SNAPSHOT~83cf093182, baseline=1.48.0-SNAPSHOT~32046a39ed
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.049 s) : 0, 1049442
Total [baseline] (10.53 s) : 0, 10530093
Agent [candidate] (1.052 s) : 0, 1052209
Total [candidate] (10.461 s) : 0, 10461334
section appsec
Agent [baseline] (1.194 s) : 0, 1194103
Total [baseline] (10.722 s) : 0, 10722065
Agent [candidate] (1.194 s) : 0, 1193507
Total [candidate] (10.727 s) : 0, 10726572
section iast
Agent [baseline] (1.188 s) : 0, 1187776
Total [baseline] (11.07 s) : 0, 11070306
Agent [candidate] (1.18 s) : 0, 1179747
Total [candidate] (11.041 s) : 0, 11040520
section profiling
Agent [baseline] (1.273 s) : 0, 1272804
Total [baseline] (10.863 s) : 0, 10862747
Agent [candidate] (1.283 s) : 0, 1283341
Total [candidate] (10.905 s) : 0, 10905045
gantt
title petclinic - break down per module: candidate=1.48.0-SNAPSHOT~83cf093182, baseline=1.48.0-SNAPSHOT~32046a39ed
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (717.637 ms) : 0, 717637
BytebuddyAgent [candidate] (718.733 ms) : 0, 718733
GlobalTracer [baseline] (238.937 ms) : 0, 238937
GlobalTracer [candidate] (239.567 ms) : 0, 239567
AppSec [baseline] (54.411 ms) : 0, 54411
AppSec [candidate] (54.482 ms) : 0, 54482
Debugger [baseline] (5.153 ms) : 0, 5153
Debugger [candidate] (4.433 ms) : 0, 4433
Remote Config [baseline] (685.957 µs) : 0, 686
Remote Config [candidate] (701.371 µs) : 0, 701
Telemetry [baseline] (12.132 ms) : 0, 12132
Telemetry [candidate] (13.817 ms) : 0, 13817
section appsec
BytebuddyAgent [baseline] (738.003 ms) : 0, 738003
BytebuddyAgent [candidate] (737.564 ms) : 0, 737564
GlobalTracer [baseline] (235.92 ms) : 0, 235920
GlobalTracer [candidate] (235.483 ms) : 0, 235483
AppSec [baseline] (175.841 ms) : 0, 175841
AppSec [candidate] (175.629 ms) : 0, 175629
Debugger [baseline] (4.279 ms) : 0, 4279
Debugger [candidate] (4.294 ms) : 0, 4294
Remote Config [baseline] (640.17 µs) : 0, 640
Remote Config [candidate] (640.948 µs) : 0, 641
Telemetry [baseline] (8.204 ms) : 0, 8204
Telemetry [candidate] (8.541 ms) : 0, 8541
IAST [baseline] (21.415 ms) : 0, 21415
IAST [candidate] (21.708 ms) : 0, 21708
section iast
BytebuddyAgent [baseline] (844.286 ms) : 0, 844286
BytebuddyAgent [candidate] (837.964 ms) : 0, 837964
GlobalTracer [baseline] (230.656 ms) : 0, 230656
GlobalTracer [candidate] (229.547 ms) : 0, 229547
AppSec [baseline] (55.641 ms) : 0, 55641
AppSec [candidate] (55.889 ms) : 0, 55889
Debugger [baseline] (4.162 ms) : 0, 4162
Debugger [candidate] (4.109 ms) : 0, 4109
Remote Config [baseline] (622.265 µs) : 0, 622
Remote Config [candidate] (615.666 µs) : 0, 616
Telemetry [baseline] (8.795 ms) : 0, 8795
Telemetry [candidate] (8.625 ms) : 0, 8625
IAST [baseline] (22.99 ms) : 0, 22990
IAST [candidate] (22.555 ms) : 0, 22555
section profiling
ProfilingAgent [baseline] (102.23 ms) : 0, 102230
ProfilingAgent [candidate] (103.317 ms) : 0, 103317
BytebuddyAgent [baseline] (709.029 ms) : 0, 709029
BytebuddyAgent [candidate] (715.025 ms) : 0, 715025
GlobalTracer [baseline] (348.306 ms) : 0, 348306
GlobalTracer [candidate] (351.262 ms) : 0, 351262
AppSec [baseline] (53.761 ms) : 0, 53761
AppSec [candidate] (53.734 ms) : 0, 53734
Debugger [baseline] (4.252 ms) : 0, 4252
Debugger [candidate] (4.297 ms) : 0, 4297
Remote Config [baseline] (694.487 µs) : 0, 694
Remote Config [candidate] (707.246 µs) : 0, 707
Telemetry [baseline] (8.93 ms) : 0, 8930
Telemetry [candidate] (9.033 ms) : 0, 9033
Profiling [baseline] (102.256 ms) : 0, 102256
Profiling [candidate] (103.342 ms) : 0, 103342
LoadParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 18 unstable metrics. Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.48.0-SNAPSHOT~83cf093182, baseline=1.48.0-SNAPSHOT~32046a39ed
dateFormat X
axisFormat %s
section baseline
no_agent (386.813 µs) : 367, 406
. : milestone, 387,
iast (516.4 µs) : 495, 538
. : milestone, 516,
iast_FULL (735.829 µs) : 714, 758
. : milestone, 736,
iast_GLOBAL (573.642 µs) : 550, 597
. : milestone, 574,
iast_HARDCODED_SECRET_DISABLED (511.929 µs) : 490, 533
. : milestone, 512,
iast_INACTIVE (467.099 µs) : 445, 489
. : milestone, 467,
iast_TELEMETRY_OFF (508.127 µs) : 486, 530
. : milestone, 508,
tracing (470.344 µs) : 447, 493
. : milestone, 470,
section candidate
no_agent (376.262 µs) : 357, 396
. : milestone, 376,
iast (514.304 µs) : 493, 536
. : milestone, 514,
iast_FULL (737.11 µs) : 715, 759
. : milestone, 737,
iast_GLOBAL (560.891 µs) : 539, 583
. : milestone, 561,
iast_HARDCODED_SECRET_DISABLED (515.283 µs) : 493, 537
. : milestone, 515,
iast_INACTIVE (468.495 µs) : 447, 490
. : milestone, 468,
iast_TELEMETRY_OFF (500.898 µs) : 479, 522
. : milestone, 501,
tracing (462.803 µs) : 442, 484
. : milestone, 463,
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.48.0-SNAPSHOT~83cf093182, baseline=1.48.0-SNAPSHOT~32046a39ed
dateFormat X
axisFormat %s
section baseline
no_agent (1.359 ms) : 1339, 1379
. : milestone, 1359,
appsec (1.743 ms) : 1719, 1767
. : milestone, 1743,
appsec_no_iast (1.748 ms) : 1725, 1772
. : milestone, 1748,
code_origins (1.698 ms) : 1672, 1724
. : milestone, 1698,
iast (1.524 ms) : 1499, 1549
. : milestone, 1524,
profiling (1.548 ms) : 1523, 1573
. : milestone, 1548,
tracing (1.508 ms) : 1483, 1534
. : milestone, 1508,
section candidate
no_agent (1.346 ms) : 1326, 1366
. : milestone, 1346,
appsec (1.736 ms) : 1712, 1760
. : milestone, 1736,
appsec_no_iast (1.757 ms) : 1732, 1782
. : milestone, 1757,
code_origins (1.685 ms) : 1657, 1712
. : milestone, 1685,
iast (1.525 ms) : 1501, 1550
. : milestone, 1525,
profiling (1.565 ms) : 1539, 1590
. : milestone, 1565,
tracing (1.508 ms) : 1483, 1533
. : milestone, 1508,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics. Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.48.0-SNAPSHOT~83cf093182, baseline=1.48.0-SNAPSHOT~32046a39ed
dateFormat X
axisFormat %s
section baseline
no_agent (15.454 s) : 15454000, 15454000
. : milestone, 15454000,
appsec (15.176 s) : 15176000, 15176000
. : milestone, 15176000,
iast (18.992 s) : 18992000, 18992000
. : milestone, 18992000,
iast_GLOBAL (17.605 s) : 17605000, 17605000
. : milestone, 17605000,
profiling (15.549 s) : 15549000, 15549000
. : milestone, 15549000,
tracing (14.945 s) : 14945000, 14945000
. : milestone, 14945000,
section candidate
no_agent (15.421 s) : 15421000, 15421000
. : milestone, 15421000,
appsec (14.938 s) : 14938000, 14938000
. : milestone, 14938000,
iast (18.812 s) : 18812000, 18812000
. : milestone, 18812000,
iast_GLOBAL (18.259 s) : 18259000, 18259000
. : milestone, 18259000,
profiling (15.206 s) : 15206000, 15206000
. : milestone, 15206000,
tracing (14.78 s) : 14780000, 14780000
. : milestone, 14780000,
Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.48.0-SNAPSHOT~83cf093182, baseline=1.48.0-SNAPSHOT~32046a39ed
dateFormat X
axisFormat %s
section baseline
no_agent (1.472 ms) : 1460, 1483
. : milestone, 1472,
appsec (2.371 ms) : 2324, 2418
. : milestone, 2371,
iast (2.166 ms) : 2106, 2226
. : milestone, 2166,
iast_GLOBAL (2.201 ms) : 2141, 2260
. : milestone, 2201,
profiling (2.046 ms) : 1998, 2095
. : milestone, 2046,
tracing (1.981 ms) : 1935, 2027
. : milestone, 1981,
section candidate
no_agent (1.47 ms) : 1458, 1481
. : milestone, 1470,
appsec (2.373 ms) : 2327, 2420
. : milestone, 2373,
iast (2.153 ms) : 2094, 2213
. : milestone, 2153,
iast_GLOBAL (2.19 ms) : 2131, 2249
. : milestone, 2190,
profiling (2.013 ms) : 1965, 2062
. : milestone, 2013,
tracing (1.985 ms) : 1939, 2030
. : milestone, 1985,
|
harels
reviewed
Mar 13, 2025
mobuchowski
force-pushed
the
mobuchowski/openlineage-agent-direct
branch
from
d36e910 to
d48bb91
Compare
kr-igor
approved these changes
Mar 13, 2025
| "_[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$", ""); | ||
| } | ||
|
|
||
| private Optional<Class> loadClass(String className) { |
Contributor
There was a problem hiding this comment.
Why don't we just instrument OL listener instead? I.e. add listener setup code which sets all required settings if not present?
| private final int MAX_COLLECTION_SIZE = 5000; | ||
| private final int MAX_ACCUMULATOR_SIZE = 50000; | ||
| private final String RUNTIME_TAGS_PREFIX = "spark.datadog.tags."; | ||
| private static final String AGENT_OL_ENDPOINT = "openlineage/api/v1/lineage"; |
Contributor
There was a problem hiding this comment.
Not directly related to this PR since it is an agent change, but to keep the same default as openlineage, wondering if it could be easier to keep the endpoint name as /api/v1/lineage in the agent
Contributor
Author
There was a problem hiding this comment.
My idea was that /api/v1/lineage endpoint was too confusing, and that as we're setting this - not customer - the additional endpoint setting does not matter that much. We can do a change on the client still, it was not yet released.
| finishApplication(System.currentTimeMillis(), null, 0, null); | ||
| } | ||
| })); | ||
| initApplicationSpanIfNotInitialized(); |
Contributor
There was a problem hiding this comment.
I don't think we should create the application span in the constructor/onApplicationStart.
In the case of Databricks or streaming jobs, we are currently not creating a application span:
- in databricks the parent of a spark job is the databricks task that launched it
- in streaming, we don't attach the streaming span to the application span
In those cases, I think we should skip overriding OpenLineage for now
Contributor
Author
There was a problem hiding this comment.
Removed this here, added check for databricks/streaming
mobuchowski
force-pushed
the
mobuchowski/openlineage-agent-direct
branch
5 times, most recently
from
c3e99e1 to
a582374
Compare
mobuchowski
force-pushed
the
mobuchowski/openlineage-agent-direct
branch
6 times, most recently
from
55d9b0d to
ccd94f3
Compare
mobuchowski
force-pushed
the
mobuchowski/openlineage-agent-direct
branch
from
b4637a8 to
1c43c96
Compare
amarziali
reviewed
Mar 27, 2025
| public static void exit(@Advice.This Object self) throws IllegalAccessException { | ||
| LoggerFactory.getLogger(Config.class).debug("Checking for OpenLineageSparkListener"); | ||
| try { | ||
| Field conf = self.getClass().getDeclaredField("conf"); |
Contributor
There was a problem hiding this comment.
can you use @FieldValue to have the same without using the reflection?
Contributor
Author
There was a problem hiding this comment.
I would like to - but the instrumented class clones the object so it's not the same one as what's passed to the method.
Contributor
There was a problem hiding this comment.
I was not proposing to capture the argument of the constructor but to use @FieldValue to directly map the field (https://javadoc.io/doc/net.bytebuddy/byte-buddy/1.10.2/net/bytebuddy/asm/Advice.FieldValue.html) like this:
public static void exit(@Advice.This Object self, @FieldValue("conf") SparkConf conf) {
...
amarziali
reviewed
Mar 27, 2025
| @Override | ||
| public String[] knownMatchingTypes() { | ||
| return new String[] { | ||
| "io.openlineage.spark.agent.OpenLineageSparkListener", "org.apache.spark.util.Utils" |
Contributor
There was a problem hiding this comment.
why org.apache.spark.util.Utils is also matching the advice activation while the method advice is only filtering by io.openlineage.spark.agent.OpenLineageSparkListener?
amarziali
reviewed
Mar 27, 2025
amarziali
reviewed
Mar 27, 2025
amarziali
reviewed
Mar 27, 2025
amarziali
reviewed
Mar 27, 2025
amarziali
reviewed
Mar 27, 2025
amarziali
requested changes
Mar 27, 2025
amarziali
left a comment
amarziali
left a comment
Contributor
There was a problem hiding this comment.
Thanks for that addition. I left few comments that might require small changes to improve that instrumentation before merging it
mobuchowski
force-pushed
the
mobuchowski/openlineage-agent-direct
branch
from
1c43c96 to
34d6cb4
Compare
Contributor
Author
|
Thanks @amarziali, applied the changes. Added one more change - using |
mobuchowski
force-pushed
the
mobuchowski/openlineage-agent-direct
branch
from
34d6cb4 to
5cd3c14
Compare
amarziali
approved these changes
Mar 31, 2025
amarziali
left a comment
amarziali
left a comment
Contributor
There was a problem hiding this comment.
looks ok. there is a fix for the system test that is failing. I recommend to merge latest master changes to this branch to pass the CI
mobuchowski
force-pushed
the
mobuchowski/openlineage-agent-direct
branch
from
5cd3c14 to
f596261
Compare
7 tasks
mobuchowski
force-pushed
the
mobuchowski/openlineage-agent-direct
branch
3 times, most recently
from
bdb681c to
c34a423
Compare
Signed-off-by: Maciej Obuchowski <[email protected]>
Signed-off-by: Maciej Obuchowski <[email protected]>
mobuchowski
force-pushed
the
mobuchowski/openlineage-agent-direct
branch
from
c34a423 to
83cf093
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Does This Do
If OpenLineage library is present on the classpath, instrument SparkListenerBus to inject the OpenLineageSparkListener with additional tags ourselves.
Motivation
https://docs.google.com/document/d/14nmf3UcqhzoOfooiveoEBUO-HHejKalLR-qAKzbR2PI/edit?tab=t.0#heading=h.dpfdi5vyt85h
https://docs.google.com/document/d/1EwNCsnhnilL6YHYMg4UYwqK_yKgBEUsxys669h8l_-M/edit?tab=t.0#heading=h.o8z5ox632qxx
Additional Notes
Contributor Checklist
type:and (comp:orinst:) labels in addition to any usefull labelsclose,fixor any linking keywords when referencing an issue.Use
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]