Apache Pulsar changelog

Changelog History

Page 2

• v2.5.1 Changes

  May 11, 2020

  2.5.1 — 2020-04-20

  🔋 Features

  • PIP-55: Refresh Authentication Credentials #6074
  • 👍 Namespace level support offloader #6183
  • ⬆️ Upgrade Avro to 1.9.1 #5938
    • Avro 1.9.1 enables the JSR310 datetimes by default, which might introduce some regression problems if users use generated source code by Avro compiler 1.8.x and contains datetimes fields. It's better to use Avro 1.9.x compiler to recompile.
  • 👌 Support unload all partitions of a partitioned topic #6187
  • 👌 Supports evenly distribute topics count when splits bundle #6241
  • 👍 KeyValue schema support for pulsar sql #6325
  • ⬆️ Bump netty version to 4.1.45.Final #6424
  • 👌 Support BouncyCastle FIPS provider #6588
  • 👌 Improve Key_Shared subscription message dispatching performance. #6647
  • ➕ Add JSR310 logical type conversion. #6704
  • 🔧 Introduce maxMessagePublishBufferSizeInMB configuration to avoid broker OOM #6178

  🛠 Fixes

  Broker

  • 🛠 [Broker] Fixed NPE occurs when getting partitioned topic stats #6060
  • 🛠 [Broker] Fixed zero queue consumer message redelivery #6076
  • 🛠 [Broker] Fixed message redelivery for zero queue consumer while using async api to receive messages #6090
  • 🛠 [broker] Fixed bug that backlog message that has not yet expired could be deleted due to TTL #6211
  • 🚚 [Broker] Remove problematic semicolon from conf #6303
  • 🛠 [Broker] Fixed broker to specify a list of bookie groups #6349
  • 🛠 [Broker] Fixed create consumer on partitioned topic while disable topic auto creation #5572
  • [Broker] Fix maven broken link #6068
  • 🛠 [Broker] Fixed java code errors reported by lgtm. #6398
  • 🛠 [Broker] Fixed memory leak when running topic compaction. #6485
  • 🛠 [Broker] Fixed admin getLastMessageId return batchIndex #6511
  • ⚡️ [Broker] Fixed topic with one partition cannot be updated #6560
  • 🛠 [Broker] Fixed negative un-ack messages in consumer stats #5929
  • ⚡️ [broker] Fixed bug that tenants whose allowed clusters include global cannot be created/updated #6275
  • 🛠 [Broker] Fixed log compaction for flow control/empty topic/last deletion #6237
  • ⏪ [Broker] Fixed restore clusterDispatchRate policy for compatibility #6176
  • [Broker] Fix some async method problems at PersistentTopicsBase. #6483
  • [Broker] This "earlier" message should be avoided to emit when reset cursor.#6393

  Managed Ledger

  • 🛠 [Managed Ledger] Fixed consumer received duplicated delayed messages upon restart #6404

  Pulsar Proxy

  • 🛠 [Proxy] Fixed correct name for proxy thread executor name #6460
  • 🛠 [Proxy] Fixed logging for published messages #6474
  • 👷 [Proxy] Fixed proxy routing to functions worker #6486

  Zookeeper

  • 🛠 [Zookeeper] Fixed casting in ZooKeeperCache.getDataIfPresent() #6313

  Pulsar Functions

  • 🚚 [Function] remove future.join() from PulsarSinkEffectivelyOnceProcessor #6361

  Pulsar SQL

  • ✅ [SQL] Fixed integration Pulsar SQL test failed #6279

  🔒 Security

  • 🛠 Fixed publish buffer limit does not take effect #6431
  • 🛠 Fixed the bug of authenticationData is't initialized. #6440

  Pulsar Schema

  • 🛠 [Schema] Fixed get schema version in HttpLookupService. #6193
  • 🛠 [Schema] Fixed avro schema decode error ClassCastException in Pulsar Function #6662
  • 🛠 [Schema] Fixed channel write error handling for send get raw schema request #6650

  Java client

  • 🛠 [Client] Fixed available permits may be greater than 1 even though queue size is 0. #6106
  • 🛠 [Client] Fixed broker client tls settings error #6128
  • 🛠 [Client]Fixed hasMessageAvailable() #6362
  • 🛠 [Client] Fixed duplicate key to send propertys #6390
  • 🛠 [Client] fixed deadlock on send failure #6488
  • 🛠 [Client] Fixed NPE while call getLastMessageId #6562
  • 🔧 [Client] Fixed the max backoff configuration for lookups #6444

  C++ client

  • 🍎 [C++] Fixed static linking on C++ lib on MacOS #5581
  • 🛠 [C++] Fixed memory corruption on ExecutorService destructor #6270
  • 🛠 [C++] Fixed handling of canceled timer events on NegativeAcksTracker #6272
  • 🛠 [C++] Fixed for possible deadlock when closing Pulsar client #6277
  • 🛠 [C++] Fixed Unacked Message Tracker by Using Time Partition on C++ #6391
  • 🛠 [C++] Fixed Redelivery of Messages on UnackedMessageTracker When Ack Messages . #6498

  Python Client

  • 📦 [Python Client]Fixed the enum34 package not found #6401

  Pulsar Websocket

  • 🛠 [Websocket] Fixed Websocket doesn't set the correct cluster data #6102

  🚀 Deployments

  • 🛠 [Helm] Autorecovery - Fixed could not find or load main class #6373
  • [Helm]: Start proxy pods when at least one broker pod is running #6158

  ✨ Enhancements

  Pulsar Broker

  • [Broker] close managed-ledgers before giving up bundle ownership to avoid bad zk-version #5599
  • 🌐 [Broker] Add timeout to search for web service URLs to avoid web threads getting stuck #6124
  • [Broker] Flush the potential duplicated message when add messages to a batch. #6326
  • 📇 [Broker] Avoid getting partition metadata while the topic name is a partition name. #6339
  • 🛠 [Broker] Fixed create partitioned topic with a substring of an existing topic name. #6478
  • [Broker] Do not retry on authorization failure #6577
  • ⚡️ [Broker]Handle BadVersionException thrown by updateSchemaLocator() #6683
  • 🔧 [Broker] Expose bookkeeper expose explicit lac configuration in broker.conf #5822
  • [Broker] Allow to enable/disable delayed delivery for messages on namespace #5915
  • [Broker] Prevent creation of regular topic with the same name as existing partitioned topic #5943
  • [Broker] Reset cursor with a non-exists position #6120
  • 0️⃣ [Broker] Use fully qualified hostname as default to advertise brokers #6235
  • ⏱ [broker] Timeout API calls in BrokerService #6489
  • [Broker] Start namespace service and schema registry service before start broker. #6499
  • [Broker] Disable channel auto read when publish rate or publish buffer exceeded #6550
  • [Broker] Resume some servercnx method to public #6581
  • [Broker] Enable get precise backlog and backlog without delayed messages. #6310
  • [Broker] Avoid using same OpAddEntry between different ledger handles #5942
  • [Broker] Clean up closed producer to avoid publish-time for producer #5988
  • 👍 [Broker] Support delete inactive topic when subscriptions caught up #6077
  • 🌲 [Broker] Add a message on how to make log refresh immediately when starting a component #6078
  • [Pulsar Admin] allow tenant admin to manage subscription permission #6122
  • 🌲 [Broker] Output resource usage rate to log on broker #6152
  • [Broker] Creating a topic does not wait for creating cursor of replicators #6364
  • [Broker] Stop increase unacked messages for the consumer with Exclusive/Failover subscription mode. #6558
  • [Broker] Not allow sub auto create by admin when disable topic auto create #6685

  Zookeeper

  • 👻 [Zookeeper] Close ZK before canceling future with exception #6399
  • ⬆️ [ZooKeeper] Upgrade ZooKeeper to 3.5.7 #6329

  Pulsar IO

  • ✅ [IO] Adds integration test for RabbitMQ #6033

  Pulsar Functions

  • 🚚 [Function] remove future.join() from PulsarSinkEffectivelyOnceProcessor #6361

  Stats & Monitoring

  • 🔊 [Broker] Add backlogSize in topicStats #5914
  • [Broker] Expose lastConsumedTimestamp and lastAckedTimestamp to consumer stats #6051
  • 👌 Improve backlogSize stats in the topic. #6700

  🔒 Security

  • Validate tokens for binary connections #6233
  • Create namespace failed when TLS is enabled in PulsarStandalone #6457
  • 👉 Use more granular permissions for topics #6504

  Pulsar Schema

  • [Schema] Independent schema is set for each consumer generated by topic #6356
  • [Schema] Extract an original avro schema from the "$SCHEMA" field using reflection. If it doesn't work, the process falls back generation of the schema from POJO.#6406
  • [Schema] Add verification for SchemaDefinitionBuilderImpl.java #6405

  Java client

  • [Client] Start reader inside batch result in read first message in batch. #6345
  • [Client] Stop shade snappy-java in pulsar-client-shaded #6375
  • ⏱ [Client] MultiTopics discovery is broken due to discovery task scheduled twice instead of pendingBatchReceiveTask #6407
  • [Client] Make SubscriptionMode a member of ConsumerConfigurationData #6337
  • [Client] Should set either start message id or start message from roll back duration. #6392
  • [Client] BatchReceivePolicy implements Serializable. #6423
  • 🚚 [Client] Remove duplicate cnx method #6490
  • [Client] Pulsar Java client: Use System.nanoTime() instead of System.currentTimeMillis() to measure elapsed time #6454
  • ✅ [Client] Make tests more stable by using JSONAssert equals #6247
  • [Client] make acker in BatchMessageIdImpl transient #6064

  C++ client

  • 🏁 [C++] Windows CMake corrections #6336
  • [C++] Avoid calling redeliverMessages() when message list is empty #6480
  • [C++] Improve cpp-client-lib: provide another libpulsarwithdeps.a in dep/rpm #6458

  Python Client

  • 👍 [Python Client] Support generate pulsar-client for python3.8#6741

  🚀 Deployments

  • [Helm] Explicit statement env-var 'BOOKIE_MEM' and 'BOOKIE_GC' for values-mini.yaml #6340
  • [Helm] Add missing check to dashboard-ingress #6160
  • 👉 Make kubernetes yamls for aws operational #6192
  • Ensure JVM memory and GC options are set for bookie #6201
  • 0️⃣ Default functionAuthProvider when running in k8s #6203

  Adaptors

  • [Adaptor] Skip javadoc task for pulsar-client-kafka-compact modules #5836
  • [Flink-Connector] Get PulsarClient from cache should always return an open instance #6436

• v2.5.1-candidate

  April 15, 2020

• v2.5.0

  January 15, 2020

• v2.5.0-candidate

  December 09, 2019

• v2.4.2

  December 04, 2019

• v2.4.2-candidate Changes

  November 28, 2019

  2019-11-25

• v2.4.1 Changes

  September 04, 2019

  2.4.1 — 2019-08-30

  🛠 Fixes

  • 🛠 Fixed wrong serialize of batchBuilder in ProducerConfigurationData #4620
  • 🛠 Fixed SchemaInfo properties losing when registering schema using admin api #4617
  • 🛠 Fixed wrong topic domain returned by get partitioned topic #4613
  • 🛠 Fixed pulsar-dashboard wrong count number of bundles #4637
  • 🛠 Fixed partitionIndex error in consumer for a single partitioned topic #4591
  • 🛠 Fixed deadlock on get-status rest-api call in broker #4616
  • 🛠 Fixed deadlock in subscribeAsync demo #4649
  • 🛠 Fixed C++ client lookup error over HTTP in standalone #4625
  • 🛠 Fixed NPE at managed-ledger when fetch reader internal-stats #4615
  • 🛠 Fixed C++ client producer sendAsync() hang when no enough batched message #4657
  • 🛠 Fixed issue when submitting NAR via file url #4577
  • 📇 Renamed C++ logger enum names to avoid conflicts with compiler macros #4664
  • 🛠 Fixed leaking of pulsar-io-influxdb in distribution #4678
  • 🛠 Fixed the default port for https and http in admin client #4623
  • 🛠 Fixed wrong schema delete when checking compatibility #4669
  • 🛠 Fixed docker/publish.sh doesn't work for pulsar-all and pulsar-standalone images #4705
  • 🛠 Fixed integration-test failure when execute pip3 install pulsar_client #4754
  • ➕ Added log folder in pulsa-function-go back #4736
  • 🛠 Fixed Pulsar SQL NPE when predicate pushdown for publish_time #4744
  • 🛠 Fixed redelivered message logic of partition topic #4653
  • 🛠 Fixed C++ log level names in Log4cxxLogger #4735
  • 🛠 Fixed go function package not executable #4743
  • ➕ Added anonymous role to proxy configuration #4733
  • ➕ Added kubernetes namespace to function instance url #4701
  • 🛠 Fixed go function not parse conf content first #4746
  • Made PulsarKafkaProducer thread safe #4745
  • 🛠 Fixed messages not get acked if there is not sink topic #4815
  • 🛠 Fixed bug when function package jar/py/go and runtime is not set #4814
  • 🛠 Fixed topic loading in rest-api not time out in zooKeeperOperationTimeoutSeconds #4805
  • 🛠 Fixed default retention policy miss upload to zk #4810
  • ➕ Added checking function implements correct interface #4844
  • 🛠 Fixed retention size policy bug #4825
  • Catch throwable in interceptors of consumer and producer #4860
  • 🛠 Fixed first position in managedLedger is ahead of the last #4853
  • 🛠 Fixed concurrent access of uninitializedCursors in ManagedLedgerImpl.asyncOpenCursor #4837
  • 🛠 Fixed schema not found handling in pulsar-sql #4890
  • 🛠 Fixed requests not respect no_proxy env variable in dashboard #4867
  • 🛠 Fixed broken replication msg to specific cluster #4930
  • 🛠 Fixed dashboard peek parse message error #4918
  • 🛠 Fixed StructSchema reader cache loading logic #4962
  • 🛠 Fixed fd leakage in FunctionActioner.downloadFile #4970
  • Fixed python function str bytes convert in example custom_object_function.py #4946
  • 🛠 Fixed NPE while cleaning up namespace node #4965
  • 🛠 Fixed the getSchema logic in pulsar proxy #4975
  • 🛠 Fixed warning by add default loader for latest pyyaml #4974
  • 🛠 Fixed snappy compressor compile error in pulsar-client-cpp #4972
  • Reinitialize certain components for externally managed runtimes when moving functions #5007
  • ⬆️ Upgraded jackson-databind #5011
  • 🛠 Fixed the problem of missing dependencies less in docker #5034
  • 🛠 Fixed duplicated Prometheus TYPE in broker metrics #4183
  • 🛠 Fixed pulsar sink and source state by init state earlier #5046

  ✨ Enhancements

  • Re-factored Component implementation #4541
  • Provided a clock for generating publish timestamp for producers #4562
  • ⬇️ Reduced unnecessary track message calls #4595
  • 👌 Supported dynamic configure with escape char #4611
  • ➕ Added null check for function/source/sink configs #4627
  • ➕ Added delete dynamic config api #4614
  • 🔧 Made broker replication mtls configuration dynamic #4609
  • ➕ Added authorization to function worker REST endpoints #4628
  • 👌 Improved and add authorization to function download and upload #4644
  • 👍 Allowed consumer retrieve the sequence id that the producer set #4645
  • ➕ Added perPartition parameter to partitioned-stats API #4639
  • 👌 Supported Pulsar schema for pulsar kafka client wrapper #4534
  • 👌 Supported delete and update event for JDBC Sink #4358
  • ✅ Cleaned up tests in the presto module #4683
  • ➕ Added allowAutoTopicCreation to broker.conf and related configuration #4694
  • 🔄 Changed to use classloaders to load Java functions #4685
  • ✂ Removed fixed server type check in kerberos #4758
  • 🔄 Changed type of publish_time to timestamp #4757
  • ➕ Added read-timeout to admin-request #4762
  • ➕ Added checking of deleted schema when adding schema #4731
  • ➕ Added strand to C++ client for exclusive control #4750
  • ➕ Added support to create partitioned topic with 1 partition #4764
  • ➕ Added getters and setters to PulsarService & BrokerService #4709
  • ➕ Added configure ack-timeout tick time #4760
  • ➕ Added options to rewrite namespace delimiter for pulsar sql #4749
  • 👍 Made Pulsar SQL supports pulsar's primitive schema #4728
  • ➕ Added basic authentication capabilities to Pulsar SQL #4779
  • 👌 Improved SchemaInfoProvider to fetch schema info asynchronously #4836
  • 👌 Supported KeyValue schema use AUTO_CONSUME as key/value schema #4839
  • Provided action type for insert in pulsar-io-jdbc #4862
  • Made partition as internal column in pulsar-sql #4888
  • ➕ Added option to disable authentication for proxy /metrics #4921

• v2.4.1-candidate

  August 28, 2019

• v2.4.0 Changes

  July 05, 2019

  Apache

  2.4.0 — 2019-06-30

  PIPs

  • 👍 PIP-26: Added support for delayed message delivery
    #4062, note that messages are only delayed on shared subscriptions.
  • PIP-28: Pulsar Proxy Gateway Improvement #3915
  • 📦 PIP-29: One package for both pulsar client and pulsar admin #3662
  • 👍 PIP-30: Change authentication provider API to support mutual authentication for Kerberos Authentication #3677
    #3821 #3997
  • PIP-32: Go Function API, Instance and LocalRun #3854 #4008 #4174
  • 🔀 PIP-33: Replicated subscriptions to keep subscription state in-sync across multiple geographical regions #4299 #4340 #4354 #4396
  • PIP-34: Key_Shared subscription, Architecture for Key_Shared#4079 #4120 #4335 #4372 #4407 #4406 4462
  • 👍 PIP-36: Support set message size in broker.conf #4247

  🛠 Fixes

  • 🛠 Fix possible message loss using peer-cluster feature #3426
  • 🛠 Fix pulsar standalone does not read zk port from conf/standalone.conf #3790
  • 🛠 Fix some issues of ZkIsolatedBookieEnsemblePlacementPolicy #3917 #3918
  • 🛠 Fix NPE when unload non-existent topic #3946
  • 🛠 Fix race condition while deleting global topic #4173
  • 🛠 Fix deadlock on skip messages #4411
  • 🛠 Fix NPE when closing batch during a reconnection #4427
  • 🛠 Fix race condition of read-timeout task in managed ledger #4437
  • 0️⃣ Disable sticky read by default #4526
  • 🛠 Fix race condition between timeout-task and add-call complete #4455

  ✨ Enhancements

  • ⚡️ Optimize message replay for large backlog consumer #3732
  • ➕ Added support for websocket produce/consume command #3835
  • ➕ Added support for TTL config in broker.conf #3898
  • ⬇️ Reduce memory used in ClientCnx for pending lookups #4104
  • ⬇️ Reduce number of hashmap sections for ledger handles cache #4102
  • ➕ Added backlog and offloaded size in Prometheus stats #4150
  • ➕ Added support for configure the managed ledger cache eviction frequency #4066
  • ➕ Added support to avoid payload copy when inserting into managed ledger cache #4197
  • ➕ Added support to cache unack-messageId into OpenRangeSet #3819
  • ➕ Added support configure static PulsarByteBufAllocator to handle OOM errors #4196
  • Auto refresh new tls certs for jetty webserver #3645
  • Create non-persistent topic by pulsar-admin/rest api #3625
  • Consumer priority-level in Failover subscription #2954

  🔒 Security

  • ➕ Added support for other algorithms in token auth #4528

  Namespace Policies

  • ➕ Added support for tenant based bookie isolation #3933
  • ➕ Added support for secondary bookie isolation group at namespace #4458
  • ➕ Added support for secondary bookie-isolation-group #4261
  • ➕ Added support for replicator rate limit between clusters #4273
  • 0️⃣ Disable backlog quota check by default #4320

  Tiered Storage

  • ➕ Added support for Pulsar SQL to read data from tiered storage #4045

  Pulsar Schema

  • ➕ Added schema versioning to support multi version messages produce and consume #3876 #3670 #4211 #4325 #4548
  • ➕ Added TRANSITIVE schema check strategies to support compatibility check over all existing schemas #4214
  • ➕ Added schema data validator #4360
  • ➕ Added support for delete schema when deleting a topic #3941
  • ➕ Added generic record builder #3690

  Pulsar IO

  • ➕ Added IO connector for flume source and sink #3597
  • ➕ Added IO connector for redis sink #3700
  • ➕ Added IO connector for solr sink #3885
  • Hide kafka-connecter details for easy use debezium connector #3825
  • ➕ Added IO connector for debezium PostgreSQL source #3924
  • ✨ Enhancements for RabbitMQ source configuration #3937
  • ➕ Added IO connector for RabbitMQ sink #3967
  • ➕ Added IO connector for InfluxDB sink #4017

  Pulsar functions

  • ➕ Added support for authentication #3735 #3874 #4198
  • 🛠 Fix NPE when stats manager not initialized #3891
  • ➕ Added async state manipulation methods #3798
  • 🛠 Fix fail to update functions in effectively-once mode #3993
  • ➕ Added labels to function statefulsets and services #4038
  • ➕ Added support for set key for message when using function publish #4005
  • 👉 Use negative acknowledge to instead ackTimeout #4103
  • 🛠 Fix backward compatibility with 2.2 auth not working #4241

  Java client

  • ➕ Added negative acks #3703
  • ➕ Added support for backoff strategy configuration #3848
  • ➕ Added support for configure TypedMessageBuilder through a Map conf object #4015
  • ➕ Added interceptor for negative ack send #3962
  • ➕ Added support for seek operate on reader #4031
  • Store key part of KeyValue schema into pulsar message keys #4117
  • ➕ Added interceptor for ack timeout #4300
  • ➕ Added support for snappy compression #4259
  • ➕ Added support for key based batcher #4435

  Python client

  • ➕ Added negative acks #3816
  • ➕ Added support for snappy compression #4319

  Go client

  • ➕ Added negative acks #3817
  • ➕ Added support for go schema #3904
  • ➕ Added support for snappy compression #4319
  • ➕ Added support for Key_Shared subscription #4465

  C++ client

  • ➕ Added negative acks #3750
  • 🛠 Fix ack timeout when subscribing to regex topic #3897
  • ➕ Added support for Key_Shared subscription #4366

  Adaptors

  • 🔧 In Kafka client wrapper, added some configurations #3753 #3797 #3843 #3887 #3991
  • In Apache Flink connector, allow to specify a custom Pulsar producer #3894 and client authentication #3949
  • 👍 In Apache Flink connector, added support for accept ClientConfigurationData, ProducerConfigurationData, ConsumerConfigurationData #4232
  • In Apache Storm connector, fix NPE while emitting next tuple #3991 and some add some enhancements #4280 #4239#4238 #4236 #4495 #4494

  👀 For a complete list of issues fixed, see

  https://github.com/apache/pulsar/milestone/20?closed=1

  🚀 https://github.com/apache/pulsar/releases/tag/v2.4.0

• v2.4.0-candidate Changes

  June 26, 2019

  2019-06-21

• « First
• ‹ Prev
• 1
• 2
• 3
• Next ›
• Last »