Timeouts are not working with speculative execution

[sirainen]

We have a test for Dovecot's Cassandra behavior which does:

• Connects to Cassandra and does a lookup
• Drops all network traffic to Cassandra: iptables -I OUTPUT -p tcp --destination-port $CASSANDRAPORT -d $CASSANDRAHOST -j DROP
• Does another lookup

We have timeouts configured, e.g.:

cass_cluster_set_connect_timeout(db->cluster, 1000);
cass_cluster_set_request_timeout(db->cluster, 1000);

This works as expected. But when adding:

cass_cluster_set_constant_speculative_execution_policy(db->cluster, 500, 1);

The driver seems to hang when doing a lookup. The lookup won't finish until the whole cluster is destroyed (at least not for the first 130 seconds). This happens with the latest Scylladb cpp-driver master (and Datastax cpp-driver too). Attached trace logs for the test.

trace.log

[sirainen]

Added also to Datastax Jira: https://datastax-oss.atlassian.net/browse/CPP-1010

[Lorak-mmk]

@sirainen Could you please test this with https://github.com/scylladb/cpp-rust-driver ?
It is our project that aims to be a drop-in replacement for this driver, using our Rust Driver underneath. Ultimately we want to deprecate this driver in favor of cpp-rust-driver.
Apart from verifying that this bug is not present there it would be awesome to check which (if any) functionality is missing for some real project.

There were some important improvements since 0.3 version of cpp-rust-driver. @muzarski could you provide the build of current master here for @sirainen to test?

[muzarski]

There were some important improvements since 0.3 version of cpp-rust-driver. @muzarski could you provide the build of current master here for @sirainen to test?

Certainly. @sirainen please tell me what you need to test your example: just the library? compressed build folder? all packaging payload for specific distribution (.deb or rpm)? You could always try to build the cpp-rust-driver from source if it's more convenient for you. Feel free to ask questions.

[sirainen]

I just tested it by compiling myself (in Ubuntu 24.04 arm64 / Macbook). First it was complaining about some symbols, like:

Apr 08 17:21:56.060514 dict(3359680): Error: Couldn't load plugin /usr/local/lib/dovecot/dict/libdriver_cassandra.so: dlopen() failed: /usr/local/lib/dovecot/dict/libdriver_cassandra.so: undefined symbol: cass_timestamp_gen_free

So I commented out all it complained about

• cass_session_get_metrics
• cass_timestamp_gen_monotonic_new
• cass_cluster_set_timestamp_gen
• cass_cluster_set_num_threads_io
• cass_retry_policy_logging_new
• cass_timestamp_gen_free

After that it loaded, but then it gave an error:

Apr 08 17:22:27.120618 dict(3360561): Error: sqltest: LOOKUP priv/test1str (user testuser1): Lookup failed: Couldn't get value as bytes: Invalid value type

Apparently cass_value_get_bytes() couldn't be used? test1str is "text" type.

[muzarski]

Apparently cass_value_get_bytes() couldn't be used? test1str is "text" type.

This is going to be supported after lazy deserialization PR is merged...

cass_value_get_bytes on master currently works only for Blob and Varint.

[Lorak-mmk]

@sirainen Thanks for this list, it will surely help us when prioritizing implementations in cpp-rust-driver. We'll let you know when they are implemented.

[sirainen]

I tested with lazy deserialization PR and it seems like my original test now works. Just some differences in expected error messages. OK, so thanks, need to keep watching cpp-rust-driver to see when we can actually use it.

[muzarski]

I tested with lazy deserialization PR and it seems like my original test now works. Just some differences in expected error messages. OK, so thanks, need to keep watching cpp-rust-driver to see when we can actually use it.

Feel free to open issues about missing functionalities/discrepancies with original cpp-driver! We will be more than happy to take it into account during planning/task prioritization.