Automated nightly release, will be updated every day at 20:00 UTC

What's Changed

• fix(log): fix consumer offsets load missing by @superhx in #1606
• fix(auto_balancer): fix potential memory leak (#1612) by @SCNieh in #1613
• fix(build): fix version check for e2e test by @ShadowySpirits in #1624
• fix(build): update docker-compose.yaml by @ShadowySpirits in #1628

Full Changelog: 1.1.1...1.1.2-rc0

What's Changed

• fix(auto_balancer): fix metrics compatibility for version V0 by @SCNieh in #1431
• fix(config): fix default config for auto balancer listener name by @SCNieh in #1436
• fix(metadata): fix mark destoryed object queue won't be cleanup in follower by @superhx in #1473
• feat(metadata): cleanup object wait last round cleanup record replay … by @superhx in #1475
• fix(auto_balancer): fix broker status by @SCNieh in #1479
• fix(auto_balancer): remove partition from cluster model when it has n… by @SCNieh in #1483
• fix(auto_balancer): fix npe when remove partition from cluster by @SCNieh in #1485
• fix(api): fix channel isn't switch when encounter NOT_CONTROLLER by @superhx in #1490
• fix(s3shell): fix object cleanup by @ShadowySpirits in #1492
• feat: bump version to 1.1.1 by @superhx in #1568

Full Changelog: 1.1.0...1.1.1

What's Changed

• docs(metrics): refine telemetry README and remove unused components by @SCNieh in #1333
• fix(issues1334): fix stream reader endless looping by @superhx in #1335
• fix(perf): increase request timeout of the admin client by @Chillax-0v0 in #1336
• feat(core): forward stream apis by @ShadowySpirits in #1338
• chore: set code owners for the whole repo by @Chillax-0v0 in #1340
• feat(core): add command config by @ShadowySpirits in #1342
• fix(auto_balancer): fix broker status change by @SCNieh in #1339
• feat(metadata): add missing tags for the old stream by @superhx in #1346
• fix(s3stream): refine network rate limiter for better precision by @SCNieh in #1344
• fix(s3stream): fix variable typo by @yx9o in #1351
• docs(telemetry): simplify grafana dashboard templates by @SCNieh in #1354
• feat(shell): remove automq cli by @ShadowySpirits in #1355
• feat(core): log the block operation in controller event loop by @ShadowySpirits in #1352
• fix(s3stream): temporarily disable percentile metrics for performance issue by @SCNieh in #1353
• perf(s3stream/allocator): increase trunk size of the netty allocator by @Chillax-0v0 in #1364
• fix(issues1357): fix missing deletes when load from snapshot by @superhx in #1358
• chore(log): fix log by @superhx in #1365
• fix(s3stream): fix mismatched placeholders by @yx9o in #1369
• feat(shell): add basic command for automq cli by @ShadowySpirits in #1370
• fix(issues1367): stream reader prevent adding invalid blocks by @superhx in #1372
• chore(s3stream): add toString methods to fix logs by @Chillax-0v0 in #1381
• fix(cache): fix readhead fail when user read fast than readahead by @superhx in #1382
• fix(core): add tolerance delay when append with non-zero seq number with empty producer snapshot (#1377) by @SCNieh in #1384
• refactor(s3stream): remove DefaultS3Client's dependency on KafkaConfig by @Chillax-0v0 in #1388
• fix(issues604): fix response size not match by @superhx in #1392
• feat(tools/perf): adjust send rate during warmup and catchup by @Chillax-0v0 in #1395
• feat(s3stream): add new object storage abstract by @superhx in #1396
• fix(action): fix release action (#1398) by @superhx in #1399
• feat(core): optimize the block operation log in controller event loop by @ShadowySpirits in #1403
• fix(auto_balancer): remove unnecessary map copy by @SCNieh in #1404
• feat(core): make controller builder extensible by @SCNieh in #1406
• perf(s3stream): reduce lock granularity in StreamMetadataManager by @lifepuzzlefun in #1411
• fix(TimeIndex): wait for the last append to finish before fetching the stream by @Chillax-0v0 in #1407
• feat(auto_balancer): opt cluster model interface by @SCNieh in #1414
• chore(EventQueue): increase EVENT_PROCESS_TIME_THRESHOLD_MICROSECOND to 5ms by @Chillax-0v0 in #1412
• fix(cache): fix cache reuse for re-created topic by @superhx in #1417
• perf(metadata): use Timeline for objects image by @superhx in #1419
• feat(s3stream): add aws s3 rangeRead implementation for object storage by @warr99 in #1416
• fix(metadata): fix metadata access outdated image by @superhx in #1420
• fix(core): fix unexpected truncation of file cache by @ShadowySpirits in #1421
• feat(core): report s3 object number only on active controller by @SCNieh in #1424
• perf(FileCache): avoid evicting too many blocks at once by @Chillax-0v0 in #1425
• feat: merge from apache kafka 3.8 0971924 by @superhx in #1427
• fix(docs): fix contributing guide by @jitokim in #1428
• chore(examples): remove hardcoded value in TransactionProducer by @jitokim in #1426
• fix(config): fix default config for auto balancer listener name by @SCNieh in #1429
• fix(tools/perf): discard invalid latency values by @Chillax-0v0 in #1430
• test(s3stream): add objectStorage rangeRead test by @warr99 in #1423
• feat(s3stream): add aws s3 write implementation for object storage by @warr99 in #1434
• fix(auto_balancer): fix metrics compatibility for version V0 by @SCNieh in #1433
• fix(auto_balancer): match all endpoints when listener name is not spe… by @SCNieh in #1435
• feat(s3stream): composite object reader&writer by @superhx in #1432
• feat(s3stream): add s3 multipart implementation for object storage by @warr99 in #1438
• feat(s3stream): composite object add object attributes by @superhx in #1437
• fix(s3stream/wal): check the block size before start by @Chillax-0v0 in #1439
• feat(tools): fix StorageTool by @ShadowySpirits in #1442
• feat(s3stream): compact to composite object by @superhx in #1443
• feat(s3stream): add composite object deletion by @superhx in #1444
• feat(s3stream): add s3 delete implementation for object storage by @warr99 in #1441
• perf(s3stream): Avoid non-reuse TimerUtil object creation on write path by @lifepuzzlefun in #1447
• feat(s3stream): object storage writer by @warr99 in #1445
• feat(s3stream): add stream object compaction v1 trigger by @superhx in #1448
• feat(issue1134): prevent trivial scheduling by @SCNieh in #1449
• feat(auto_balancer): move internal topic partition only when necessary by @SCNieh in #1450
• feat(s3stream): adapt to new object storage abstraction by @superhx in #1451
• feat(core): implement broker and cluster level quota by @ShadowySpirits in #1440
• fix: stream e2e by @superhx in #1453
• feat: add composite object test by @superhx in #1454
• feat(s3stream): integrate composite object by @superhx in #1456
• fix(test): fix e2e by @superhx in #1457
• test(s3stream): add test case to verify end-of-object read behavior by @warr99 in #1455
• feat(auto_balancer): refactor cluster load snapshot for better extens… by @SCNieh in #1459
• chore: update README.md by @wensongz in #1460
• refactor(license): organize license files together by @KaimingWan in #1458
• chore: add a readme doc for s3stream by @daniel-y in #1461
• feat(issue1462): Optimize the Readme. #1462 by @vintagewang in #1463
• feat(s3stream): implement a prototype for the S3 Write-Ahead Log by @ShadowySpirits in #1465
• feat(s3stream): remove dependency on stream range offset by @SCNieh in #1466
• feat(metadata): optimize streamsetobject to support huge cluster by @superhx in #1464
• fix(test): fix connect e2e task timeout by @superhx in #1467
• fix(test): skip unsupported test by @superhx in #1468
• fix(s3stream): ensure wal header shutdown type UNGRACEFULLY when startup by @lifepuzzlefun in #1469
• chore(all): adapt to java21 by @superhx in #1471
• feat(s3stream): implement AcknowledgmentService by @ShadowySpirits in #1470
• feat(core): add extension point for TelemetryManager by @SCNieh in #1472
• fix(auto_balancer): fix broker status by @SCNieh in #1480
• fix(auto_balancer): remove partition from cluster model when it has n… ...

What's Changed

• fix(auto_balancer): fix metrics compatibility for version V0 by @SCNieh in #1431
• fix(config): fix default config for auto balancer listener name by @SCNieh in #1436
• fix(metadata): fix mark destoryed object queue won't be cleanup in follower by @superhx in #1473
• feat(metadata): cleanup object wait last round cleanup record replay … by @superhx in #1475
• fix(auto_balancer): fix broker status by @SCNieh in #1479
• fix(auto_balancer): remove partition from cluster model when it has n… by @SCNieh in #1483
• fix(auto_balancer): fix npe when remove partition from cluster by @SCNieh in #1485
• fix(api): fix channel isn't switch when encounter NOT_CONTROLLER by @superhx in #1490
• fix(s3shell): fix object cleanup by @ShadowySpirits in #1492

Full Changelog: 1.1.0...1.1.1-rc0

Highlights

• Upgrade Apache Kafka from version 3.4.0 to 3.7.0

What's Changed

• feat(auto_balancer): support dynamic configs for AutoBalancer by @SCNieh in #1076
• fix(issues1078): fix RequestChannel shutdown blocking by @superhx in #1080
• feat(auto_balancer): optimzed distribution goal action score calculation by @SCNieh in #1083
• docs(config): add config usage suggestion by @superhx in #1084
• fix(auto_balancer): avoid broker load affected by excluded topics by @SCNieh in #1086
• fix(maven): correct the maven build system for s3stream by @daniel-y in #1089
• fix(auto_balancer): add missing variable initialization by @SCNieh in #1090
• feat(auto_balancer): record network metrics by throttle priority by @SCNieh in #1093
• feat(MM2): allow customizing MM2 offset-syncs/checkpoints/heartbeats topic easily by @aaron-ai in #1095
• build(issues1033): switch log appender from DailyRollingFileAppender to RollingFileAppender by @warr99 in #1092
• feat(issues1097): increase api call timeout after timeout by @superhx in #1098
• fix(gradle): specify the classifier for netty-tcnative by @Chillax-0v0 in #1103
• fix(s3stream): fail fast if openssl not available by @Chillax-0v0 in #1105
• feat(issues1087): add data block cache by @superhx in #1107
• fix(docker): use specific localstack version; update aws-cli version by @mooc9988 in #1108
• feat(auto_balancer): split and merge swap actions by @SCNieh in #1109
• chore(util): print commit id when calling scripts wich "--version" as it used to be by @Chillax-0v0 in #1110
• fix(bin): support to run in Windows by @Chillax-0v0 in #1112
• feat(issues1087): add stream reader by @superhx in #1114
• feat(auto_balancer): support broker rack in cluster model by @SCNieh in #1115
• fix(s3): ControllerKVClient putKV should return result by @yangl in #1113
• feat(issues1087): add stream readers by @superhx in #1116
• fix(issues1087): event loop reject task after a task fail by @superhx in #1117
• fix(issues1087): fix long running issues by @superhx in #1118
• fix(issues1087): fix object not exist handle by @superhx in #1119
• fix(core): fix race issue in transaction verification by @ShadowySpirits in #1121
• feat(issues1087): add cold down time to reset readahead by @superhx in #1123
• feat(issue1120): support slow broker detect in controller by @SCNieh in #1122
• docs(config): refine metrics related config documentations by @SCNieh in #1125
• feat(issue1120): support reporting broker metrics in AutoBalancerMetricsReporter by @SCNieh in #1124
• feat(issues1087): allow readahead when cache is free by @superhx in #1126
• fix(metrics): fix method name by @SCNieh in #1129
• feat(issues1087): set s3.object.block.size default from 1MB to 512KB by @superhx in #1130
• chore: update README.md by @KaimingWan in #1131
• feat(metrics): use delta value on reporting percentile metrics by @SCNieh in #1132
• feat(issues1087): increase readahead size when cache miss by @superhx in #1133
• fix(issues1127): fix 1.0 to 1.1-rc0 launch fail by @superhx in #1128
• fix(metrics): remove redundant error log by @SCNieh in #1135
• chore(grafana): fix broken panels by @SCNieh in #1136
• fix(issues1142): retry get index when object not exist by @superhx in #1143
• feat(auto_balancer): support client ACL configuration by @SCNieh in #1146
• fix(core): compose future result when writing stream object by @SCNieh in #1147
• feat(issue1144): start next compaction in advance when needed by @SCNieh in #1148
• fix(issues1097): specify cleanup/minor/major stream object compaction… by @superhx in #1149
• fix(issues1140): fix S3ObjectControlManager NPE by @superhx in #1151
• fix(auto_balancer): suppress error log for OutOfOrderSequenceException by @SCNieh in #1154
• Revert "fix(issues1140): fix S3ObjectControlManager NPE" by @superhx in #1157
• feat(s3stream): change the checksum algorithm to crc32c by @ShadowySpirits in #1155
• chore: update readme and add comparision by @KaimingWan in #1141
• fix(issues1140): fix S3ObjectControlManager NPE by @superhx in #1158
• chore: adjust table and cloud supported badge by @KaimingWan in #1162
• feat(s3stream): reduce stream object compaction write amplification by @superhx in #1159
• feat(throttle): subtract extra token from total available token by @SCNieh in #1163
• feat(throttle): add constructor for compatibility by @SCNieh in #1164
• Revert "feat(s3stream): change the checksum algorithm to crc32c (#1155)" by @ShadowySpirits in #1166
• fix(issues1072): fix MultiPartWriterTest unstable test by @warr99 in #1165
• fix(s3stream): precise record get objects metrics by @superhx in #1170
• chore: support one line quick start by @KaimingWan in #1168
• feat(issues1171): improve objects deletion performance by @superhx in #1172
• feat(s3stream): speed up block cache get objects by @superhx in #1167
• feat(auto_balancer): reduce massive small objects copy during scheduling by @SCNieh in #1176
• fix(auto_balancer): add client ACL to load retriever consumer by @SCNieh in #1182
• feat(issue1179): accelerate partition reassignments on load balancing by @SCNieh in #1181
• chore: update readme by @KaimingWan in #1183
• chore: Update README.md for more accurate descriptions by @daniel-y in #1185
• feat(issues1169): optimize partition reassignment logic to avoid NotLeaderOrFollowerException by @superhx in #1180
• feat(s3stream): active evict expired data by @superhx in #1186
• feat(auto_balancer): change log level to warning when failed on sendi… by @SCNieh in #1188
• style(auto_balancer): remove unused import by @SCNieh in #1189
• feat(issue668): disable swap action temporarily by @SCNieh in #1190
• feat(auto_balancer): add missing action logging by @SCNieh in #1194
• chore: Update README.md for explanations for Kafka compatibility (#1191) by @amos201600 in #1193
• chore(readme): remove controller from the comparison table, and mark proxy of pulsar as optional by @daniel-y in #1197
• feat(issue1195): introduce goal group on score calculation for goals on different scale by @SCNieh in #1196
• fix(issue1198): use timeline structure to avoid incorrect s3 object size by @SCNieh in #1199
• feat(s3stream): add tags to stream by @superhx in #1200
• chore(controller): more logs by @Chillax-0v0 in #1201
• feat(test): add e2e tests for s3stream by @ShadowySpirits in #1204
• feat(auto_balancer): optimize resource type for better scalability by @SCNieh in #1206
• chore(cicd): removing site-docs file in GitHub release workflow by @Gezi-lzq in #1207
• feat(issues1169): prevent re-open partition when the broker is shutting down by @superhx in #1210
• feat: update AWS S3 bucket and destination directory in GitHub releas… by @Gezi-lzq in #1211
• feat(#1212): update readme，add alibaba cloud by @amos201600 in #1214
• fix(s3stream): fix stream reader busy loop when metadata replay delay by @superhx in https://github...

What's Changed

• chore: switch to dev mode by @Chillax-0v0 in #1324
• refactor(config): remove config "s3.object.tagging" by @Chillax-0v0 in #1326
• revert: revert to 1.0.6-rc1 by @Chillax-0v0 in #1376
• fix(KAFKA-14644): Process should crash after failure in Raft IO thread (#13140) by @SCNieh in #1361
• fix(KAFKA-14693): Kafka node should halt instead of exit (#13227) by @SCNieh in #1378
• fix(KAFKA-14704): Follower should truncate before incrementing high watermark by @SCNieh in #1379
• fix(metadata): fix broken logic on catching up check by @SCNieh in #1385
• fix(KAFKA-14857): Fix some MetadataLoader bugs by @SCNieh in #1390
• fix(kafka): merge OSS Kafka 3.4 latest by @SCNieh in #1391
• build(gradle): fix s3stream repo by @SCNieh in #1409
• chore(version): release 1.0.6-rc2 by @SCNieh in #1415

Full Changelog: 1.0.6-rc1...1.0.6-rc2

What's Changed

• feat(auto_balancer): support dynamic configs for AutoBalancer by @SCNieh in #1076
• fix(issues1078): fix RequestChannel shutdown blocking by @superhx in #1080
• feat(auto_balancer): optimzed distribution goal action score calculation by @SCNieh in #1083
• docs(config): add config usage suggestion by @superhx in #1084
• fix(auto_balancer): avoid broker load affected by excluded topics by @SCNieh in #1086
• fix(maven): correct the maven build system for s3stream by @daniel-y in #1089
• fix(auto_balancer): add missing variable initialization by @SCNieh in #1090
• feat(auto_balancer): record network metrics by throttle priority by @SCNieh in #1093
• feat(MM2): allow customizing MM2 offset-syncs/checkpoints/heartbeats topic easily by @aaron-ai in #1095
• build(issues1033): switch log appender from DailyRollingFileAppender to RollingFileAppender by @warr99 in #1092
• feat(issues1097): increase api call timeout after timeout by @superhx in #1098
• fix(gradle): specify the classifier for netty-tcnative by @Chillax-0v0 in #1103
• fix(s3stream): fail fast if openssl not available by @Chillax-0v0 in #1105
• feat(issues1087): add data block cache by @superhx in #1107
• fix(docker): use specific localstack version; update aws-cli version by @mooc9988 in #1108
• feat(auto_balancer): split and merge swap actions by @SCNieh in #1109
• chore(util): print commit id when calling scripts wich "--version" as it used to be by @Chillax-0v0 in #1110
• fix(bin): support to run in Windows by @Chillax-0v0 in #1112
• feat(issues1087): add stream reader by @superhx in #1114
• feat(auto_balancer): support broker rack in cluster model by @SCNieh in #1115
• fix(s3): ControllerKVClient putKV should return result by @yangl in #1113
• feat(issues1087): add stream readers by @superhx in #1116
• fix(issues1087): event loop reject task after a task fail by @superhx in #1117
• fix(issues1087): fix long running issues by @superhx in #1118
• fix(issues1087): fix object not exist handle by @superhx in #1119
• fix(core): fix race issue in transaction verification by @ShadowySpirits in #1121
• feat(issues1087): add cold down time to reset readahead by @superhx in #1123
• feat(issue1120): support slow broker detect in controller by @SCNieh in #1122
• docs(config): refine metrics related config documentations by @SCNieh in #1125
• feat(issue1120): support reporting broker metrics in AutoBalancerMetricsReporter by @SCNieh in #1124
• feat(issues1087): allow readahead when cache is free by @superhx in #1126
• fix(metrics): fix method name by @SCNieh in #1129
• feat(issues1087): set s3.object.block.size default from 1MB to 512KB by @superhx in #1130
• chore: update README.md by @KaimingWan in #1131
• feat(metrics): use delta value on reporting percentile metrics by @SCNieh in #1132
• feat(issues1087): increase readahead size when cache miss by @superhx in #1133
• fix(issues1127): fix 1.0 to 1.1-rc0 launch fail by @superhx in #1128
• fix(metrics): remove redundant error log by @SCNieh in #1135
• chore(grafana): fix broken panels by @SCNieh in #1136
• fix(issues1142): retry get index when object not exist by @superhx in #1143
• feat(auto_balancer): support client ACL configuration by @SCNieh in #1146
• fix(core): compose future result when writing stream object by @SCNieh in #1147
• feat(issue1144): start next compaction in advance when needed by @SCNieh in #1148
• fix(issues1097): specify cleanup/minor/major stream object compaction… by @superhx in #1149
• fix(issues1140): fix S3ObjectControlManager NPE by @superhx in #1151
• fix(auto_balancer): suppress error log for OutOfOrderSequenceException by @SCNieh in #1154
• Revert "fix(issues1140): fix S3ObjectControlManager NPE" by @superhx in #1157
• feat(s3stream): change the checksum algorithm to crc32c by @ShadowySpirits in #1155
• chore: update readme and add comparision by @KaimingWan in #1141
• fix(issues1140): fix S3ObjectControlManager NPE by @superhx in #1158
• chore: adjust table and cloud supported badge by @KaimingWan in #1162
• feat(s3stream): reduce stream object compaction write amplification by @superhx in #1159
• feat(throttle): subtract extra token from total available token by @SCNieh in #1163
• feat(throttle): add constructor for compatibility by @SCNieh in #1164
• Revert "feat(s3stream): change the checksum algorithm to crc32c (#1155)" by @ShadowySpirits in #1166
• fix(issues1072): fix MultiPartWriterTest unstable test by @warr99 in #1165
• fix(s3stream): precise record get objects metrics by @superhx in #1170
• chore: support one line quick start by @KaimingWan in #1168
• feat(issues1171): improve objects deletion performance by @superhx in #1172
• feat(s3stream): speed up block cache get objects by @superhx in #1167
• feat(auto_balancer): reduce massive small objects copy during scheduling by @SCNieh in #1176
• fix(auto_balancer): add client ACL to load retriever consumer by @SCNieh in #1182
• feat(issue1179): accelerate partition reassignments on load balancing by @SCNieh in #1181
• chore: update readme by @KaimingWan in #1183
• chore: Update README.md for more accurate descriptions by @daniel-y in #1185
• feat(issues1169): optimize partition reassignment logic to avoid NotLeaderOrFollowerException by @superhx in #1180
• feat(s3stream): active evict expired data by @superhx in #1186
• feat(auto_balancer): change log level to warning when failed on sendi… by @SCNieh in #1188
• style(auto_balancer): remove unused import by @SCNieh in #1189
• feat(issue668): disable swap action temporarily by @SCNieh in #1190
• feat(auto_balancer): add missing action logging by @SCNieh in #1194
• chore: Update README.md for explanations for Kafka compatibility (#1191) by @amos201600 in #1193
• chore(readme): remove controller from the comparison table, and mark proxy of pulsar as optional by @daniel-y in #1197
• feat(issue1195): introduce goal group on score calculation for goals on different scale by @SCNieh in #1196
• fix(issue1198): use timeline structure to avoid incorrect s3 object size by @SCNieh in #1199
• feat(s3stream): add tags to stream by @superhx in #1200
• chore(controller): more logs by @Chillax-0v0 in #1201
• feat(test): add e2e tests for s3stream by @ShadowySpirits in #1204
• feat(auto_balancer): optimize resource type for better scalability by @SCNieh in #1206
• chore(cicd): removing site-docs file in GitHub release workflow by @Gezi-lzq in #1207
• feat(issues1169): prevent re-open partition when the broker is shutting down by @superhx in #1210
• feat: update AWS S3 bucket and destination directory in GitHub releas… by @Gezi-lzq in #1211
• feat(#1212): update readme，add alibaba cloud by @amos201600 in #1214
• fix(s3stream): fix stream reader busy loop when metadata replay delay by @superhx in #1215
• feat(issues1174): cleanup deleted topic by controller by @superhx in #1213
• chore(lo...

What's Changed

• chore: switch to dev mode by @Chillax-0v0 in #1290
• fix(issues1222): fix stream advance check by @superhx in #1307
• chore(version): release 1.0.6-rc1 by @superhx in #1310

Full Changelog: 1.0.6-rc0...1.0.6-rc1

What's Changed

• fix(issues1222): fix #openStream resp nextOffset by @superhx in #1226
• chore: switch to dev mode by @Chillax-0v0 in #1230
• fix: compatible with s3-Stream by @warr99 in #1251
• fix(issues1248): fix partition reassignment deadlock by @superhx in #1250
• feat(metadata): fix data race when close and reopen coordinator partition by @ShadowySpirits in #1255
• fix(metadata): ensure that the consumer offset is deleted along with the deletion of the topic by @ShadowySpirits in #1263
• fix(metadata): clean up the consumption offset for the deleted partitions by @ShadowySpirits in #1265
• fix(issues1276): group the batch request by nodeId and apiKey (#1277) by @superhx in #1278
• ci: bump version to 1.0.6-rc0 by @Chillax-0v0 in #1289

Full Changelog: 1.0.5...1.0.6-rc0
S3Stream Changelog: 1.0.5-s3stream...1.0.6-s3stream-rc0