diff --git a/doc/admin-guide/configuration/hrw4u.en.rst b/doc/admin-guide/configuration/hrw4u.en.rst index 4a0fb3f6961..049454f33c0 100644 --- a/doc/admin-guide/configuration/hrw4u.en.rst +++ b/doc/admin-guide/configuration/hrw4u.en.rst @@ -710,9 +710,9 @@ Denied Sections --------------- The ``sections`` list accepts any of the HRW4U section names listed in the -`Sections`_ table, plus ``VARS`` to deny the variable declaration block. -A denied section causes the entire block to be rejected; the body is not -validated. +`Sections`_ table, plus ``VARS`` and ``SESSION_VARS`` to deny the variable +declaration blocks. A denied section causes the entire block to be rejected; +the body is not validated. Functions --------- @@ -775,7 +775,7 @@ The ``language`` list accepts a fixed set of constructs: Construct What it controls ================ =================================================== ``break`` The ``break;`` statement (early section exit) -``variables`` The entire ``VARS`` section and all variable usage +``variables`` The entire ``VARS`` and ``SESSION_VARS`` section and all variable usage ``else`` The ``else { ... }`` branch of conditionals ``elif`` The ``elif ... { ... }`` branch of conditionals ``in`` The ``in [...]`` and ``!in [...]`` set membership operators diff --git a/doc/admin-guide/files/records.yaml.en.rst b/doc/admin-guide/files/records.yaml.en.rst index 129114ed344..975e31eb58a 100644 --- a/doc/admin-guide/files/records.yaml.en.rst +++ b/doc/admin-guide/files/records.yaml.en.rst @@ -2033,13 +2033,15 @@ Origin Server Connect Attempts this setting resolve to the same group -- that is, the same key under :ts:cv:`proxy.config.http.per_server.connection.match` -- the transaction that creates the group determines its metrics, and later transactions do not change them. A group is discarded once its - connection count reaches zero, so *raising* the level of publication is picked up the next time - that upstream is reopened: enabling metrics, or enabling the aggregates, takes effect as upstreams - reconnect. Lowering it does not. Metrics are never retired once published, so disabling this - setting, or switching - :ts:cv:`proxy.config.http.per_server.connection.metric_aggregate` to ``2``, leaves the names that - are already published in place, frozen at their last sampled value, until |TS| is restarted. This - affects only which metrics exist; enforcement of + connection count reaches zero, so a change is picked up the next time that upstream is reopened. + A group that never goes idle keeps whatever was in effect when it was created. + + Disabling this setting does not retire metrics that are already published: they stay in place, + frozen at their last sampled value, until |TS| is restarted. Changing + :ts:cv:`proxy.config.http.per_server.connection.metric_aggregate` does retract what it no longer + asks for, as each group is rebuilt. + + This affects only which metrics exist; enforcement of :ts:cv:`proxy.config.http.per_server.connection.max` uses the group's own connection count and is unaffected. @@ -2051,24 +2053,30 @@ Origin Server Connect Attempts :ts:cv:`proxy.config.http.per_server.connection.metric_enabled`. Has no effect when that setting is ``0``. - A per hostname aggregate sums a counter across every group belonging to that hostname that has - aggregation enabled, and exists only for + There are two kinds of per hostname aggregate. The *sums* add ``current_connection``, + ``total_connection`` and ``blocked_connection`` across every group belonging to that hostname + that has aggregation enabled. The *max* is ``current_connection.max``, the largest + ``current_connection`` among those groups, which is the one that answers how close the busiest + group is to :ts:cv:`proxy.config.http.per_server.connection.max`. Both exist only for :ts:cv:`match type ` ``both``, since that is the only match type whose group key carries the hostname. See :ref:`per-server-connection-metrics`. - ===== ====================================================================================== - Value Effect - ===== ====================================================================================== - ``0`` No aggregates. The per group metrics are published under their own names. - ``1`` Publish the per hostname aggregates and the per group metrics. - ``2`` Publish only the per hostname aggregates. The per group metrics from which they are - computed are collected but not published, which keeps the number of published metrics - proportional to hostnames rather than to groups. - ===== ====================================================================================== + ===== =========== ====== ===== + Value Per group Sums Max + ===== =========== ====== ===== + ``0`` published no no + ``1`` published yes yes + ``2`` hidden no yes + ``3`` hidden yes yes + ===== =========== ====== ===== + + ``2`` is the smallest useful configuration: one metric per hostname. ``3`` adds that hostname's + totals. Both keep the number of published metrics proportional to hostnames rather than to + groups. - With value ``2``, a group that has no aggregate to belong to -- any match type other than - ``both`` -- has its per group metrics published anyway, since otherwise nothing at all would be - reported for it. + With values ``2`` and ``3``, a group that has no aggregate to belong to -- any match type other + than ``both`` -- has its per group metrics published anyway, since otherwise nothing at all would + be reported for it. Values ``0`` and ``1`` can produce a very large number of metrics when the match type includes the address or port, since there is then one set per address and port rather than one per hostname. @@ -2079,12 +2087,17 @@ Origin Server Connect Attempts upstream had aggregation enabled, so mappings that disagree for one hostname produce an aggregate that covers only part of it. - The reload is one-directional for the same reason given under - :ts:cv:`proxy.config.http.per_server.connection.metric_enabled`. Raising the value takes effect - as upstreams reconnect, but moving to ``2`` does not hide per group metrics that are already - published, and moving from ``1`` to ``0`` does not stop the hostname aggregates from publishing. - Reducing the number of published metrics therefore requires a restart, which matters most for - ``2``, the value chosen specifically to bound that number. + A change in either direction takes effect as upstreams reconnect: a group publishes what the + new value asks for and withdraws what it does not, when that group is next rebuilt. Metrics + withdrawn this way stop appearing in :program:`traffic_ctl` output and in the other metric + consumers; they are not destroyed, and moving back republishes them with their accumulated + values intact. + + The per group metrics belong to a single group, so raising the value withdraws them as that group + is rebuilt. The sums and the max are named per hostname and shared by its groups, so a group + rebuilt for a value that does not publish them only stops contributing; they are withdrawn once + no group of that hostname publishes them. Mappings that disagree for one hostname therefore + cannot hide each other's aggregate. .. ts:cv:: CONFIG proxy.config.http.per_server.connection.metric_prefix STRING NULL :reloadable: @@ -2761,11 +2774,48 @@ Cache Control using the newest key generation. This might be temporarily necessary if a large cache was created by the previous version of ATS but the new version changed the way cache keys are generated. If this is turned on, - a metric called `proxy.process.http.cache.compat_key_reads` will be + a metric called ``proxy.process.http.cache.compat_key_reads`` will be incremented any time the compat cache lookup successfully finds the object. You can monitor this metric and know when its safe to turn this feature off as the cache wraps around. + Three costs come with enabling this. Every cache miss performs a second + lookup, so a tier with a low hit ratio roughly doubles its cache lookup + load for the duration. Every request that invalidates a cached copy + (``DELETE``, ``PURGE``, ``PUT``, and ``POST`` unless + :ts:cv:`proxy.config.http.cache.post_method` is enabled) issues a second + remove under the previous key, because a migrated object exists under both + keys until the old copy ages out. That remove takes the whole object under + the previous key, so anything that invalidates one variant of a resource + that varies on a request header, including a revalidation whose response + cannot be cached, also removes every other variant still held under the + previous key. Those variants are fetched from the origin again when next + requested. And an object found under the previous + key cannot be revalidated with a ``304``, because the write that would carry + the update is a create under the new key rather than an update of the old + one. When a ``GET`` needs such an object revalidated, |TS| sends the request + without conditional headers, neither its own nor the client's, even when + :ts:cv:`proxy.config.http.cache.when_to_revalidate` is ``4``. The origin + returns the full response, which is stored under the new key, and the copy + under the previous key is left to age out. The client still receives a + ``304`` if its conditions match the full response. A ``HEAD`` or a range + request revalidates as usual instead, since neither response is stored: + when the origin answers ``304`` the object stays under the previous key to + be migrated by the next full ``GET``, and when it answers with a changed + object the copy under the previous key is removed. + Each object pays this once, but on a large cache the aggregate is a + bandwidth event worth sizing before enabling the setting in production. + + For the same reason a plugin cannot modify such an object in place. + ``TSHttpTxnUpdateCachedObject`` still returns ``TS_SUCCESS`` and the client + receives the modified headers, but they are not stored, as when an update + cannot get the cache write lock. + + Objects whose path contains a ``;`` are unaffected. The previous algorithm + hashed the path and the deprecated ``;params`` segment as separate + components, which produces the same string the current algorithm produces + for such a path, so no compatibility lookup is issued for them. + .. ts:cv:: CONFIG proxy.config.http.cache.range.lookup INT 1 :overridable: diff --git a/doc/admin-guide/monitoring/statistics/core/http-connection.en.rst b/doc/admin-guide/monitoring/statistics/core/http-connection.en.rst index 47075ce6fa9..d5b438fcb01 100644 --- a/doc/admin-guide/monitoring/statistics/core/http-connection.en.rst +++ b/doc/admin-guide/monitoring/statistics/core/http-connection.en.rst @@ -234,21 +234,23 @@ blocked_connection Counter. The total number of connection attempts to the group blocked by :ts:cv:`proxy.config.http.per_server.connection.max`. Never decreases. -For a hostname aggregate, ```` is one of those three, each summed across the groups of that -hostname which have aggregation enabled, plus: +For a hostname aggregate there are two kinds. The *sums* are those same three counters, each added +across the groups of that hostname which have aggregation enabled, published at +:ts:cv:`metric_aggregate ` ``1`` and +``3``. The *max*, published at ``1``, ``2`` and ``3``, is: -current_connection_max +current_connection.max Gauge. The largest ``current_connection`` value among the groups of that hostname at the moment of sampling, so the maximum rather than the sum of the groups' current counts. This is useful because :ts:cv:`proxy.config.http.per_server.connection.max` is enforced per group rather than per hostname, so the busiest group is what determines whether connections are about to be blocked. Like ``current_connection`` it rises and falls with traffic and is not a high-water - mark. There is no per group ``current_connection_max``; it exists only as a hostname aggregate. + mark. There is no per group ``current_connection.max``; it exists only as a hostname aggregate. Because :ts:cv:`proxy.config.http.per_server.connection.metric_aggregate` is overridable, a group joins its hostname's aggregate only if the mapping that first opened that upstream had aggregation enabled. Mappings that disagree for one hostname therefore produce an aggregate over part of it: the -sums cover a subset of the groups and ``current_connection_max`` takes its maximum over that same +sums cover a subset of the groups and ``current_connection.max`` takes its maximum over that same subset, with nothing in the metric to indicate it. Keeping the setting uniform across the mappings for a hostname avoids this. @@ -261,13 +263,13 @@ Every published per server metric is recomputed periodically, currently every 5 on every connection event, so a reader sees a value up to that interval old. This is true of the hostname aggregates and of the published per group metrics alike: those are mirrored from the internal ones by the same periodic mechanism, not written as connections open and -close. It applies to ``current_connection_max`` too, which reports the maximum across groups as of +close. It applies to ``current_connection.max`` too, which reports the maximum across groups as of the last sample rather than a running peak. To obtain the peak over a longer window, compute a maximum over time from this gauge in the monitoring system. -At :ts:cv:`metric_aggregate ` value ``2`` -the per group metrics still exist internally, since the aggregates are computed from them, but are not -published. They can be listed with ``traffic_ctl metric match per_server --include-hidden``, which +At :ts:cv:`metric_aggregate ` values +``2`` and ``3`` the per group metrics still exist internally, since the aggregates are computed from +them, but are not published. They can be listed with ``traffic_ctl metric match per_server --include-hidden``, which reads them directly and so is not subject to the sampling delay above. That visibility is intended for debugging and is not a stable interface: the existence, granularity and naming of the per group metrics may change independently of the published aggregates. diff --git a/doc/developer-guide/internal-libraries/Metrics.en.rst b/doc/developer-guide/internal-libraries/Metrics.en.rst index 4822170546e..4f9c86b0dae 100644 --- a/doc/developer-guide/internal-libraries/Metrics.en.rst +++ b/doc/developer-guide/internal-libraries/Metrics.en.rst @@ -98,6 +98,24 @@ never as a side effect of a broad query. renamed or removed between releases without notice. Do not build monitoring on them; use the published aggregate instead. +Enumerating metrics +=================== + +``for_each`` visits every listed metric of a store, in creation order: + +.. code-block:: cpp + + ts::Metrics::instance().for_each([](std::string_view name, ts::Metrics::MetricType type, int64_t value) { + // ... + }); + +This is the only way to enumerate a store. There is no public iterator, and deliberately so: +enumeration is always the whole store, so nothing can hold a cursor across changes to the store or +name a position the walk would skip. Reach a single metric by name with ``lookup`` instead. + +The callback must not create a metric, which would be an attempt to grow the store from inside a +pass over it. + Derived metrics =============== @@ -153,6 +171,18 @@ Adding a source that is already registered for that derived metric is a no-op, s re-register the same source, such as one recreating an object for the same key, need not track that itself. The ``type`` and ``op`` arguments are ignored if the derived metric already exists. +``ts::Metrics::Derived::remove_source()`` is the counterpart, for a contributor that goes away or +stops wanting the aggregate published: + +.. code-block:: cpp + + ts::Metrics::Derived::remove_source("proxy.process.example.total", per_thing_metric); + +A derived metric is shared by its sources, so this does not unlist it while any remain. Removing the +last source leaves nothing to report, so the name is unlisted; adding a source again relists it. +Removing a source that is not registered, or naming a derived metric that does not exist, is a +no-op. + A hidden source can feed a published aggregate: .. code-block:: cpp @@ -188,6 +218,42 @@ sampling point*, not the true peak. There are two ways to arrange this, with dif Which is appropriate depends on whether the consumer needs to aggregate over time downstream. +Unlisting a metric +================== + +A metric can be taken out of the store's listing after the fact. An unlisted metric is skipped by +iteration, so it disappears from ``traffic_ctl metric match``, the JSONRPC record lookup and +``stats_over_http``, without either of those consumers needing to know about it: + +.. code-block:: cpp + + auto &m = ts::Metrics::instance(); + + m.unlist(id); // by id + m.unlist("proxy.process.example"); // or by name + + m.relist(id); // put it back + +The slot, the name and the atomic all survive: an unlisted number that still rings. An unlisted +metric still resolves through ``lookup``, so an exact name query, a logging field reference and +``TSStatFindName`` all continue to work, and its value may still be read and written. Creating the +same name again relists it and returns the same id with its accumulated value intact, so a metric +that comes and goes with a configuration setting costs nothing to bring back. + +This exists because the decision to publish a name is otherwise made once, when the metric is first +created, and can never be revisited. Any metric whose name or publication policy depends on a +runtime changeable setting needs a way to retract a name it has already published. + +.. important:: + + Unlisting hides; it does not free. The slot and the name remain allocated against the storage + limit below. Unlisting does not make an unbounded naming scheme safe. + +.. note:: + + The set walked is fixed when ``for_each`` begins, so a metric created while it runs is not + visited. + Storage limits ============== diff --git a/doc/developer-guide/release-process/index.en.rst b/doc/developer-guide/release-process/index.en.rst index 47437dd3163..b17b20a8591 100644 --- a/doc/developer-guide/release-process/index.en.rst +++ b/doc/developer-guide/release-process/index.en.rst @@ -65,10 +65,22 @@ Build #. Check the version in ``CMakeLists.txt``. There is a ``project`` line near the top with the version number. Make sure that is correct for the release. -#. Generate or update the CHANGELOG for the next release. :: - - ./tools/git/changelog.pl -o apache -r trafficserver -m X.Y.Z > - CHANGELOG-X.Y.Z +#. Generate or update the CHANGELOG for the next release. This needs + `uv `__, and ``--use-gh`` additionally needs the + `GitHub CLI `__ already authenticated with + ``gh auth login``. :: + + uv run --project tools/changelog python tools/changelog/changelog.py \ + -o apache -r trafficserver -m X.Y.Z --use-gh > CHANGELOG-X.Y.Z + + Without the GitHub CLI, omit ``--use-gh`` and supply a token through the + ``GH_TOKEN`` environment variable instead. A token is not optional here: an + unauthenticated run exceeds the GitHub API rate limit partway through a + release-sized milestone and exits without writing a changelog. :: + + GH_TOKEN= uv run --project tools/changelog \ + python tools/changelog/changelog.py \ + -o apache -r trafficserver -m X.Y.Z > CHANGELOG-X.Y.Z #. Commit this file to the repository and push it to the release branch. diff --git a/doc/release-notes/upgrading.en.rst b/doc/release-notes/upgrading.en.rst index 3ad23d29443..4c1e725725b 100644 --- a/doc/release-notes/upgrading.en.rst +++ b/doc/release-notes/upgrading.en.rst @@ -115,6 +115,25 @@ Plugins configuration load time rather than at runtime. A configuration that was previously accepted and failed later will now fail to load. +``ts::Metrics``, in the installed ``tsutil/Metrics.h``, no longer has an +iterator. ``Metrics::iterator``, ``begin()``, ``end()`` and ``find()`` are +removed, and enumeration is now ``Metrics::for_each(func)``, which invokes +``func(name, type, value)`` for each metric: + +.. code-block:: cpp + + ts::Metrics::instance().for_each([](std::string_view name, ts::Metrics::MetricType type, int64_t value) { + // ... + }); + +Handing out a position let a caller name a slot the store was free to change +underneath them, which is what the iterator could not be made safe against. +Reaching a single metric by name is ``lookup()``. + +``Metrics::Storage::createSpan()`` and ``Metrics::rename()`` are also removed. +Spans handed out unnamed slots that only ``rename()`` could name, and +``rename()`` mutated a name that the lock free readers hand out views of. + Build ----- diff --git a/include/iocore/net/ConnectionTracker.h b/include/iocore/net/ConnectionTracker.h index e18f6aa66c6..cc574d9846d 100644 --- a/include/iocore/net/ConnectionTracker.h +++ b/include/iocore/net/ConnectionTracker.h @@ -79,29 +79,54 @@ class ConnectionTracker * * This is independent of @c TxnConfig::metric_enabled, which decides only whether per server * metrics exist for a group at all. The per group metrics are always created in the hidden metric - * store; what varies here is what gets published from them: - * - @c AGGREGATE_NONE: no aggregate. The per group metrics are published under their own names. - * This is the default and matches the behavior of releases that had no aggregate support. - * - @c AGGREGATE_GROUP: the per hostname aggregates are published, and so are the per group - * metrics they are computed from. - * - @c AGGREGATE_ONLY: the per hostname aggregates are published and the per group metrics stay - * hidden, which keeps the published metric count proportional to hostnames rather than to - * groups. Where a group has no aggregate to belong to -- see @c Group::host_metric_name, which - * only yields a name for match type @c MATCH_BOTH -- the per group metrics are published - * anyway, since otherwise nothing at all would be reported for that group. + * store; what varies here is what gets published from them. + * + * Two kinds of per hostname aggregate exist. The *sums* are @c current_connection, + * @c total_connection and @c blocked_connection added across the groups of a hostname. The *max* + * is @c current_connection.max, the largest @c current_connection among those groups. Which of + * them are published, and whether the per group metrics are published alongside, is what this + * selects: + * + * | value | per group | sums | max | + * |----------------------|-----------|------|-----| + * | @c AGGREGATE_NONE | yes | no | no | + * | @c AGGREGATE_GROUP | yes | yes | yes | + * | @c AGGREGATE_MAX | no | no | yes | + * | @c AGGREGATE_SUM | no | yes | yes | + * + * @c AGGREGATE_NONE is the default and matches the behavior of releases that had no aggregate + * support. @c AGGREGATE_MAX is the smallest useful configuration: one metric per hostname, + * answering how close the busiest group is to @c per_server.connection.max. @c AGGREGATE_SUM adds + * the totals for that hostname. Both keep the published metric count proportional to hostnames + * rather than to groups. + * + * Where a group has no aggregate to belong to -- see @c Group::host_metric_name, which only + * yields a name for match type @c MATCH_BOTH -- the per group metrics are published whatever this + * says, since otherwise nothing at all would be reported for that group. * * Keeping the per group metrics in the hidden store in every case means changing this at runtime * is only a change of what is registered for publication, with no metric to migrate between the * two stores. * - * The records layer validates and clamps this to 0..2. A plugin setting the overridable config + * A change is applied per group, when that group is next constructed, which happens on the first + * connection after its count last fell to zero. A group that never goes idle keeps whatever was + * in effect when it was created. + * + * The per group names belong to one group, so a group that stops publishing them retracts them, + * see @c ts::Metrics::unlist. The sums and the max are named per hostname and shared by its + * groups, so a group that stops publishing those only stops contributing to them, see + * @c ts::Metrics::Derived::remove_source; they are retracted once no group of that hostname + * publishes them. + * + * The records layer validates and clamps this to 0..3. A plugin setting the overridable config * directly is not clamped, see @c METRIC_AGGREGATE_CONV; any other value behaves as - * @c AGGREGATE_GROUP, publishing both the aggregate and the per group metrics. + * @c AGGREGATE_GROUP, publishing everything. */ enum MetricAggregate : int { AGGREGATE_NONE = 0, ///< No hostname aggregate; the per group metrics are published. - AGGREGATE_GROUP = 1, ///< Hostname aggregates published, along with the per group metrics. - AGGREGATE_ONLY = 2, ///< Hostname aggregates published, per group metrics kept hidden. + AGGREGATE_GROUP = 1, ///< Hostname sums and max, published along with the per group metrics. + AGGREGATE_MAX = 2, ///< Hostname max only; the per group metrics stay hidden. + AGGREGATE_SUM = 3, ///< Hostname sums and max; the per group metrics stay hidden. }; /// Per transaction configuration values. diff --git a/include/proxy/ProxySession.h b/include/proxy/ProxySession.h index 5eaa6ce0605..4ebc2389b71 100644 --- a/include/proxy/ProxySession.h +++ b/include/proxy/ProxySession.h @@ -185,7 +185,7 @@ class ProxySession : public VConnection, public PluginUserArgs IpAllow::ACL acl; ///< IpAllow based method ACL. - HttpSessionAccept::Options const *accept_options{nullptr}; ///< connection info + HttpSessionAcceptBase const *acceptor{nullptr}; ///< Acceptor with the connection properties. protected: // Hook dispatching state diff --git a/include/proxy/hdrs/URL.h b/include/proxy/hdrs/URL.h index e8aeee0978d..e4d2de03978 100644 --- a/include/proxy/hdrs/URL.h +++ b/include/proxy/hdrs/URL.h @@ -28,6 +28,7 @@ #include "proxy/hdrs/HdrHeap.h" #include "tscore/CryptoHash.h" #include "proxy/hdrs/MIME.h" +#include #include #include "tscore/ink_apidefs.h" @@ -101,6 +102,13 @@ class URLImpl : public HdrHeapObjImpl void set_type_code(unsigned int typecode); std::string_view get_params() const noexcept; void set_params(HdrHeap *heap, std::string_view value, bool copy_string); + /** Whether the path carries a deprecated ";params" segment. + + ATS 9.2 and earlier split "/path;params" into separate path and params + components. That parsing was removed, so the segment now stays inside the + path. Reproducing a 9.2 cache key has to know which form it is looking at. + */ + bool has_path_params() const noexcept; std::string_view get_query() const noexcept; void set_query(HdrHeap *heap, std::string_view value, bool copy_string); std::string_view get_fragment() const noexcept; @@ -261,6 +269,7 @@ class URL : public HdrHeapSDKHandle char *string_get_buf(char *dstbuf, int dsbuf_size, int *length = nullptr) const; void hash_get(CryptoHash *hash, bool ignore_query = false, cache_generation_t generation = -1) const; void hash_get92(CryptoHash *hash, bool ignore_query = false, cache_generation_t generation = -1) const; + bool has_path_params() const noexcept; void host_hash_get(CryptoHash *hash) const; std::string_view scheme_get() const noexcept; @@ -488,6 +497,22 @@ URL::hash_get92(CryptoHash *hash, bool ignore_query, cache_generation_t generati url_CryptoHash_get_92(m_url_impl, hash, ignore_query, generation); } +/*------------------------------------------------------------------------- + -------------------------------------------------------------------------*/ + +inline bool +URLImpl::has_path_params() const noexcept +{ + return m_ptr_path != nullptr && std::memchr(m_ptr_path, ';', m_len_path) != nullptr; +} + +inline bool +URL::has_path_params() const noexcept +{ + ink_assert(valid()); + return m_url_impl->has_path_params(); +} + /*------------------------------------------------------------------------- -------------------------------------------------------------------------*/ diff --git a/include/proxy/http/HttpSM.h b/include/proxy/http/HttpSM.h index c2128eeca20..9945273c768 100644 --- a/include/proxy/http/HttpSM.h +++ b/include/proxy/http/HttpSM.h @@ -35,8 +35,6 @@ #include #include -#include "tscore/ink_platform.h" -#include "iocore/eventsystem/EventSystem.h" #include "proxy/http/HttpCacheSM.h" #include "proxy/http/HttpTransact.h" #include "proxy/http/HttpUserAgent.h" @@ -45,7 +43,6 @@ #include "proxy/http/HttpTunnel.h" #include "api/InkAPIInternal.h" #include "proxy/ProxyTransaction.h" -#include "proxy/hdrs/HdrUtils.h" // inknet #include "proxy/http/PreWarmManager.h" @@ -182,6 +179,28 @@ enum class CompatibilityCacheLookup { COMPAT_CACHE_LAST, }; +/// Policy for an object found under the previous (9.2) cache key. Shared by the +/// HTTP state machine, its cache sub-machine and HttpTransact, which is why it +/// is not a member of any of them. +namespace CompatCacheKey +{ +/// Whether this lookup addresses the cache with the previous (9.2) key. +inline bool +is_legacy(CompatibilityCacheLookup lookup) +{ + return lookup == CompatibilityCacheLookup::COMPAT_CACHE_LOOKUP_92; +} + +/// The object info to hand to a cache write, which a compatibility read must not +/// carry: it belongs to the legacy key and would turn the write into an update +/// of a vector the canonical key does not have. +inline CacheHTTPInfo * +write_info(CompatibilityCacheLookup lookup, CacheHTTPInfo *object_read_info) +{ + return is_legacy(lookup) ? nullptr : object_read_info; +} +} // namespace CompatCacheKey + class HttpSM : public Continuation, public PluginUserArgs { friend class HttpTransact; @@ -406,6 +425,7 @@ class HttpSM : public Continuation, public PluginUserArgs void do_hostdb_lookup(); void do_hostdb_reverse_lookup(); + URL *cache_lookup_url(); void do_cache_lookup_and_read(); void do_http_server_open(bool raw = false, bool only_direct = false); bool apply_ip_allow_filter(); @@ -421,6 +441,7 @@ class HttpSM : public Continuation, public PluginUserArgs void do_cache_prepare_update(); void do_cache_prepare_action(HttpCacheSM *c_sm, CacheHTTPInfo *object_read_info, bool retry, bool allow_multiple = false); void do_cache_delete_all_alts(); + void do_cache_delete_compat_alts(); void do_auth_callout(); int do_api_callout(); int do_api_callout_internal(); diff --git a/include/proxy/http/HttpSessionAccept.h b/include/proxy/http/HttpSessionAccept.h index 8a9441d6a25..ad4041ce415 100644 --- a/include/proxy/http/HttpSessionAccept.h +++ b/include/proxy/http/HttpSessionAccept.h @@ -31,6 +31,9 @@ #include "iocore/net/Net.h" #include "iocore/net/SessionAccept.h" +#include +#include + namespace detail { /** Options for @c HttpSessionAccept. @@ -154,18 +157,43 @@ HttpSessionAcceptOptions::setSessionProtocolPreference(SessionProtocolSet const } } // namespace detail -/** - The continuation mutex is NULL to allow parellel accepts in NT. No - state is recorded by the handler and values are required to be set - during construction via the @c Options struct and never changed. So - a NULL mutex is safe. +/** Common state for protocol-specific HTTP acceptors. + * + * A proxy port can dispatch connections to several HTTP protocol acceptors. + * This base keeps one shared, immutable set of connection properties for all + * of those acceptors and provides access to the complete proxy port. Sessions + * retain a pointer to their acceptor so the properties remain available + * without protocol-specific copies. + * + * The continuation mutex is @c NULL because the acceptors do not mutate their + * shared state after construction. + */ +class HttpSessionAcceptBase : public SessionAccept +{ +public: + using Options = detail::HttpSessionAcceptOptions; + using OptionsHandle = std::shared_ptr; + + HttpSessionAcceptBase(OptionsHandle options, HttpProxyPort *proxy_port = nullptr) + : SessionAccept(nullptr), _options(std::move(options)) + { + ink_release_assert(_options != nullptr); + proxyPort = proxy_port; + } + + ~HttpSessionAcceptBase() override = default; - Most of the state is simply passed on to the @c ClientSession after - an accept. It is done here because this is the least bad pathway - from the top level configuration to the HTTP session. -*/ + Options const & + options() const + { + return *_options; + } -class HttpSessionAccept : public SessionAccept, private detail::HttpSessionAcceptOptions +private: + OptionsHandle _options; +}; + +class HttpSessionAccept : public HttpSessionAcceptBase { private: using self = HttpSessionAccept; ///< Self reference type. @@ -173,17 +201,20 @@ class HttpSessionAccept : public SessionAccept, private detail::HttpSessionAccep /** Construction options. Provide an easier to remember typedef for clients. */ - using Options = detail::HttpSessionAcceptOptions; + using Options = HttpSessionAcceptBase::Options; + using OptionsHandle = HttpSessionAcceptBase::OptionsHandle; /** Default constructor. @internal We don't use a static default options object because of initialization order issues. It is important to pick up data that is read from the config file and a static is initialized long before that point. */ - HttpSessionAccept(Options const &opt = Options()) : SessionAccept(nullptr), detail::HttpSessionAcceptOptions(opt) // copy these. + HttpSessionAccept(Options const &opt = Options()) : HttpSessionAccept(std::make_shared(opt)) {} + + HttpSessionAccept(OptionsHandle options, HttpProxyPort *proxy_port = nullptr) + : HttpSessionAcceptBase(std::move(options), proxy_port) { SET_HANDLER(&HttpSessionAccept::mainEvent); - return; } ~HttpSessionAccept() override { return; } diff --git a/include/proxy/http2/Http2SessionAccept.h b/include/proxy/http2/Http2SessionAccept.h index 79c8bd6b5ef..77fefaab437 100644 --- a/include/proxy/http2/Http2SessionAccept.h +++ b/include/proxy/http2/Http2SessionAccept.h @@ -26,8 +26,6 @@ #include "tscore/ink_platform.h" #include "iocore/net/Net.h" -// XXX HttpSessionAccept::Options needs to be refactored and separated from HttpSessionAccept so that -// it can generically apply to all protocol implementations. #include "proxy/http/HttpSessionAccept.h" // HTTP/2 Session Accept. @@ -38,8 +36,8 @@ // // CONFIG proxy.config.http.server_ports STRING 80:proto=http2 443:ssl:proto=h2-12 -struct Http2SessionAccept : public SessionAccept { - explicit Http2SessionAccept(const HttpSessionAccept::Options &); +struct Http2SessionAccept : public HttpSessionAcceptBase { + explicit Http2SessionAccept(OptionsHandle options, HttpProxyPort *proxy_port = nullptr); ~Http2SessionAccept() override; bool accept(NetVConnection *, MIOBuffer *, IOBufferReader *) override; @@ -48,7 +46,4 @@ struct Http2SessionAccept : public SessionAccept { // noncopyable Http2SessionAccept(const Http2SessionAccept &) = delete; Http2SessionAccept &operator=(const Http2SessionAccept &) = delete; - -private: - HttpSessionAccept::Options options; }; diff --git a/include/proxy/http3/Http09App.h b/include/proxy/http3/Http09App.h index 7397afe1189..411669ceb55 100644 --- a/include/proxy/http3/Http09App.h +++ b/include/proxy/http3/Http09App.h @@ -41,7 +41,7 @@ class Http09Session; class Http09App : public QUICApplication { public: - Http09App(NetVConnection *client_vc, QUICConnection *qc, IpAllow::ACL &&session_acl, const HttpSessionAccept::Options &options); + Http09App(NetVConnection *client_vc, QUICConnection *qc, IpAllow::ACL &&session_acl, HttpSessionAcceptBase const *acceptor); ~Http09App(); void on_stream_open(QUICStream &stream) override; diff --git a/include/proxy/http3/Http3App.h b/include/proxy/http3/Http3App.h index 70e7377beff..d198ab673eb 100644 --- a/include/proxy/http3/Http3App.h +++ b/include/proxy/http3/Http3App.h @@ -47,7 +47,7 @@ class Http3Session; class Http3App : public QUICApplication { public: - Http3App(NetVConnection *client_vc, QUICConnection *qc, IpAllow::ACL &&session_acl, const HttpSessionAccept::Options &options); + Http3App(NetVConnection *client_vc, QUICConnection *qc, IpAllow::ACL &&session_acl, HttpSessionAcceptBase const *acceptor); virtual ~Http3App(); void on_stream_open(QUICStream &stream) override; diff --git a/include/proxy/http3/Http3SessionAccept.h b/include/proxy/http3/Http3SessionAccept.h index 798d4ef5fb5..92c95a76ca0 100644 --- a/include/proxy/http3/Http3SessionAccept.h +++ b/include/proxy/http3/Http3SessionAccept.h @@ -26,8 +26,6 @@ #include "tscore/ink_platform.h" #include "iocore/net/Net.h" -// XXX HttpSessionAccept::Options needs to be refactored and separated from HttpSessionAccept so that -// it can generically apply to all protocol implementations. #include "proxy/http/HttpSessionAccept.h" // HTTP/QUIC Session Accept. @@ -37,10 +35,10 @@ // // CONFIG proxy.config.http.server_ports STRING 443:quic -class Http3SessionAccept : public SessionAccept +class Http3SessionAccept : public HttpSessionAcceptBase { public: - explicit Http3SessionAccept(const HttpSessionAccept::Options &); + explicit Http3SessionAccept(OptionsHandle options, HttpProxyPort *proxy_port = nullptr); ~Http3SessionAccept(); bool accept(NetVConnection *, MIOBuffer *, IOBufferReader *) override; @@ -49,6 +47,4 @@ class Http3SessionAccept : public SessionAccept private: Http3SessionAccept(const Http3SessionAccept &); Http3SessionAccept &operator=(const Http3SessionAccept &); - - HttpSessionAccept::Options options; }; diff --git a/include/proxy/logging/Log.h b/include/proxy/logging/Log.h index fffc8adf2f2..68cb0ecc5b9 100644 --- a/include/proxy/logging/Log.h +++ b/include/proxy/logging/Log.h @@ -158,6 +158,10 @@ class Log static void init_fields(); static void load_config(); + // Register a field in both global_field_list and field_symbol_hash, keyed by + // its own symbol so the two can never drift apart. + static void register_field(LogField *field); + static bool transaction_logging_enabled() { diff --git a/include/proxy/logging/LogField.h b/include/proxy/logging/LogField.h index 1fbb3dc2168..abdf935e1d7 100644 --- a/include/proxy/logging/LogField.h +++ b/include/proxy/logging/LogField.h @@ -273,7 +273,7 @@ class LogFieldList ~LogFieldList(); void clear(); - void add(LogField *field, bool copy = true); + void add(LogField *field); void remove(LogField *field); LogField *find_by_name(const char *name) const; LogField *find_by_symbol(const char *symbol) const; diff --git a/include/tsutil/Metrics.h b/include/tsutil/Metrics.h index 741b1e069a5..b389ba3017f 100644 --- a/include/tsutil/Metrics.h +++ b/include/tsutil/Metrics.h @@ -33,6 +33,7 @@ #include #include #include +#include #include #include @@ -93,14 +94,25 @@ class Metrics static constexpr int METRIC_TYPE_BITS = 29; static constexpr int METRIC_TYPE_MASK = 0x1FFF; + /// The reserved slot 0 of every store, returned when an id cannot be produced. Present under this + /// name in both stores, so a consumer querying both has to expect it twice. + static constexpr std::string_view BAD_ID_NAME{"proxy.process.api.metrics.bad_id"}; + private: - using NameAndId = std::tuple; - using LookupTable = std::unordered_map; - using NameStorage = std::array; - using AtomicStorage = std::array; - using NamesAndAtomics = std::tuple; + using NameAndId = std::tuple; + using LookupTable = std::unordered_map; + using NameStorage = std::array; + using AtomicStorage = std::array; + /// Per slot flag bits, see @c UNLISTED. A parallel array rather than a member of @c NameAndId + /// because an atomic member would make that tuple neither copyable nor movable, and the slot is + /// written there with a tuple assignment. + using FlagStorage = std::array, MAX_SIZE>; + using NamesAndAtomics = std::tuple; using BlobStorage = std::array, MAX_BLOBS>; + /// The slot exists and is still resolvable by name or id, but is skipped by iteration. + static constexpr uint8_t UNLISTED = 0x01; + public: Metrics(const self_type &) = delete; self_type &operator=(const self_type &) = delete; @@ -145,6 +157,57 @@ class Metrics { return _storage->lookup(id, out_name, type); } + + /** Take @a id out of the store's listing. + * + * An unlisted metric keeps its slot, its name and its atomic. It is skipped by iteration, so it + * vanishes from everything that enumerates the store, but it still resolves through @c lookup and + * its value may still be read and written -- an unlisted number that still rings. Creating the + * same name again relists it and returns the same id. + * + * @return @c false if @a id does not name an allocated slot. + */ + bool + unlist(IdType id) + { + return _storage->set_listed(id, false); + } + + /// Put @a id back in the listing. @see unlist + bool + relist(IdType id) + { + return _storage->set_listed(id, true); + } + + /** Whether @a id is enumerated. + * + * @return @c false for an unlisted metric, and also for an id that names no allocated slot -- + * neither appears in iteration. + */ + bool + listed(IdType id) const + { + return _storage->listed(id); + } + + /// Convenience for callers that publish by name and do not retain the id. @see unlist + bool + unlist(std::string_view name) + { + auto id = lookup(name); + + return id != NOT_FOUND && unlist(id); + } + + /// Convenience for callers that publish by name and do not retain the id. @see relist + bool + relist(std::string_view name) + { + auto id = lookup(name); + + return id != NOT_FOUND && relist(id); + } AtomicType & operator[](IdType id) { @@ -191,87 +254,21 @@ class Metrics return _storage->valid(id); } - // Static methods to encapsulate access to the atomic's - class iterator - { - public: - using iterator_category = std::input_iterator_tag; - using value_type = std::tuple; - using difference_type = ptrdiff_t; - using pointer = value_type *; - using reference = value_type &; - - iterator(const Metrics &m, IdType pos) : _metrics(m), _it(pos) {} - - iterator & - operator++() - { - next(); - - return *this; - } - - iterator - operator++(int) - { - iterator result = *this; - - next(); - - return result; - } - - value_type - operator*() const - { - std::string_view name; - MetricType type; - auto metric = _metrics.lookup(_it, &name, &type); - - return std::make_tuple(name, type, metric->_value.load()); - } - - bool - operator==(const iterator &o) const - { - return _it == o._it && std::addressof(_metrics) == std::addressof(o._metrics); - } - - bool - operator!=(const iterator &o) const - { - return _it != o._it || std::addressof(_metrics) != std::addressof(o._metrics); - } - - private: - void next(); - - const Metrics &_metrics; - Metrics::IdType _it; - }; - - iterator - begin() const - { - return iterator(*this, 0); - } - - iterator - end() const + /** Visit every listed metric. + * + * @a func is called as func(std::string_view name, MetricType type, int64_t value) for + * each listed metric, in creation order. Unlisted metrics are skipped, @see unlist. + * + * The set walked is fixed when the call begins: a metric created while it runs is not visited. + * Enumeration is deliberately the whole store and nothing less. There is no cursor to hold, so + * nothing can outlive the walk or name a slot the walk would not visit, and @a func may not + * create a metric, which would be an attempt to grow the store from inside a pass over it. + */ + template + void + for_each(F &&func) const { - return iterator(*this, _storage->next_free_id()); - } - - iterator - find(const std::string_view name) const - { - auto id = lookup(name); - - if (id == NOT_FOUND) { - return end(); - } else { - return iterator(*this, id); - } + _storage->for_each(std::forward(func)); } private: @@ -337,7 +334,7 @@ class Metrics _blobs[0] = std::make_unique(); release_assert(_blobs[0]); // Reserve slot 0 for errors, this should always be 0 - release_assert(0 == create("proxy.process.api.metrics.bad_id", MetricType::COUNTER)); + release_assert(0 == create(BAD_ID_NAME, MetricType::COUNTER)); } ~Storage() {} @@ -349,6 +346,46 @@ class Metrics AtomicType *lookup(Metrics::IdType id, std::string_view *out_name = nullptr, MetricType *out_type = nullptr) const; std::string_view name(IdType id) const; MetricType type(IdType id) const; + bool set_listed(IdType id, bool listed); + bool listed(IdType id) const; + + /** Visit every listed slot, in creation order. + * + * @see Metrics::for_each, which is how callers reach this. + * + * The bound is read once, up front. Acquiring it acquires every slot below it, which is what + * lets the walk read names and values without the mutex: a slot's name is written before the + * release store that publishes it, and never changes. + */ + template + void + for_each(F &&func) const + { + auto const [last_blob, last_off] = _splitID(next_free_id()); + + for (uint16_t blob = 0; blob <= last_blob; ++blob) { + NamesAndAtomics const *entries = _blobs[blob].get(); + + // The bound covers every blob below it, so this is belt and braces. + if (entries == nullptr) { + break; + } + + uint16_t const limit = blob == last_blob ? last_off : MAX_SIZE; + + for (uint16_t off = 0; off < limit; ++off) { + if ((std::get<2>(*entries)[off].load(MEMORY_ORDER) & UNLISTED) != 0) { + continue; + } + + auto const &slot = std::get<0>(*entries)[off]; + + // The type comes from the slot's own id, not from the position, so it is the type the + // metric was created with. + func(std::string_view{std::get<0>(slot)}, _extractType(std::get<1>(slot)), std::get<1>(*entries)[off].load()); + } + } + } /// The id the next slot will get, which is also iteration's exclusive bound. IdType @@ -682,6 +719,21 @@ class Metrics */ static void add_source(std::string_view derived_name, Metrics::MetricType type, Metrics::AtomicType *source, Op op = Op::SUM); + /** Stop @a source contributing to a derived metric. + * + * The counterpart to @c add_source, for a contributor that goes away or stops wanting the + * aggregate published. A derived metric is shared by its sources, so this does not unlist it + * while any remain; when the last one is removed there is nothing left to report and the name + * is unlisted. Re-adding a source relists it. + * + * A source that is not registered for @a derived_name, or a name with no derived metric, is a + * no-op. + * + * Safe against a concurrent @c add_source for the same name: the listing change is made with + * the membership change, so a metric cannot be left unlisted with a source still contributing. + */ + static void remove_source(std::string_view derived_name, Metrics::AtomicType *source); + /** * Update derived metrics. * diff --git a/lib/swoc/unit_tests/test_TextView.cc b/lib/swoc/unit_tests/test_TextView.cc index 2b5ac27fd2e..e7a245c5924 100644 --- a/lib/swoc/unit_tests/test_TextView.cc +++ b/lib/swoc/unit_tests/test_TextView.cc @@ -12,6 +12,7 @@ #include #include #include +#include #include "swoc/TextView.h" #include diff --git a/plugins/cache_promote/configs.cc b/plugins/cache_promote/configs.cc index a9d8b960927..68bd8e41bba 100644 --- a/plugins/cache_promote/configs.cc +++ b/plugins/cache_promote/configs.cc @@ -20,6 +20,7 @@ #include "configs.h" #include "lru_policy.h" #include "chance_policy.h" +#include "tsutil/StringCompare.h" ////////////////////////////////////////////////////////////////////////////////////////////// // ToDo: It's ugly that this is a "global" options list, clearly each policy should be able @@ -58,9 +59,9 @@ PromotionConfig::factory(int argc, char *argv[]) if (opt == -1) { break; } else if (opt == 'p') { - if (0 == strncasecmp(optarg, "chance", 6)) { + if (ts::iequals(optarg, "chance")) { _policy = new ChancePolicy(); - } else if (0 == strncasecmp(optarg, "lru", 3)) { + } else if (ts::iequals(optarg, "lru")) { _policy = new LRUPolicy(); } else { TSError("[%s] Unknown policy --policy=%s", PLUGIN_NAME, optarg); diff --git a/plugins/cachekey/configs.cc b/plugins/cachekey/configs.cc index 1eeebdb64a2..f5a0203d896 100644 --- a/plugins/cachekey/configs.cc +++ b/plugins/cachekey/configs.cc @@ -28,6 +28,7 @@ #include /* strlen() */ #include "configs.h" +#include "tsutil/StringCompare.h" template static void @@ -44,7 +45,7 @@ commaSeparateString(ContainerType &c, const String &input) static bool isTrue(const char *arg) { - return (nullptr == arg || 0 == strncasecmp("true", arg, 4) || 0 == strncasecmp("1", arg, 1) || 0 == strncasecmp("yes", arg, 3)); + return (nullptr == arg || ts::iequals("true", arg) || ts::iequals("1", arg) || ts::iequals("yes", arg)); } void diff --git a/plugins/compress/compress.cc b/plugins/compress/compress.cc index 3994be9b72e..1cae68a992b 100644 --- a/plugins/compress/compress.cc +++ b/plugins/compress/compress.cc @@ -29,6 +29,7 @@ #include #include +#include #include "ts/ts.h" #include "tscore/ink_defs.h" @@ -250,7 +251,7 @@ vary_header(TSMBuffer bufp, TSMLoc hdr_loc) count = TSMimeHdrFieldValuesCount(bufp, hdr_loc, ce_loc); for (idx = 0; idx < count; idx++) { const char *value = TSMimeHdrFieldValueStringGet(bufp, hdr_loc, ce_loc, idx, &len); - if (len && strncasecmp("Accept-Encoding", value, len) == 0) { + if (len > 0 && ts::iequals("Accept-Encoding", std::string_view{value, static_cast(len)})) { // Bail, Vary: Accept-Encoding already sent from origin TSHandleMLocRelease(bufp, hdr_loc, ce_loc); return TS_SUCCESS; diff --git a/plugins/escalate/escalate.cc b/plugins/escalate/escalate.cc index 67b6d775185..56c6ad8f805 100644 --- a/plugins/escalate/escalate.cc +++ b/plugins/escalate/escalate.cc @@ -35,6 +35,7 @@ #include "swoc/IPEndpoint.h" #include "swoc/TextView.h" +#include "tsutil/StringCompare.h" // Constants and some declarations @@ -258,11 +259,11 @@ TSRemapNewInstance(int argc, char *argv[], void **instance, char *errbuf, int er char *sep, *token, *save; // Ugly, but we set the precedence before with non-command line parsing of args - if (0 == strncasecmp(argv[i], "--pristine", 10)) { + if (ts::iequals(argv[i], "--pristine")) { es->use_pristine = true; - } else if (0 == strncasecmp(argv[i], "--no-redirect-header", 20)) { + } else if (ts::iequals(argv[i], "--no-redirect-header")) { es->add_redirect_header = false; - } else if (0 == strncasecmp(argv[i], "--escalate-non-get-methods", 26)) { + } else if (ts::iequals(argv[i], "--escalate-non-get-methods")) { es->escalate_non_get_methods = true; } else { // Each token should be a status code then a URL, separated by ':'. diff --git a/plugins/experimental/access_control/config.cc b/plugins/experimental/access_control/config.cc index 0a2f435ab21..0bbf1d8db20 100644 --- a/plugins/experimental/access_control/config.cc +++ b/plugins/experimental/access_control/config.cc @@ -28,11 +28,12 @@ #include "common.h" #include "config.h" +#include "tsutil/StringCompare.h" static bool isTrue(const char *arg) { - return (nullptr == arg || 0 == strncasecmp("true", arg, 4) || 0 == strncasecmp("1", arg, 1) || 0 == strncasecmp("yes", arg, 3)); + return (nullptr == arg || ts::iequals("true", arg) || ts::iequals("1", arg) || ts::iequals("yes", arg)); } /** diff --git a/plugins/experimental/cache_fill/configs.h b/plugins/experimental/cache_fill/configs.h index 81e31b05abe..489dc90c883 100644 --- a/plugins/experimental/cache_fill/configs.h +++ b/plugins/experimental/cache_fill/configs.h @@ -32,6 +32,7 @@ #include #include "rules.h" +#include "tsutil/StringCompare.h" /////////////////////////////////////////////////////////////////////////// // This holds one complete background fetch rule @@ -73,7 +74,7 @@ class BgFetchConfig static bool isTrue(const char *arg) { - return (nullptr == arg || 0 == strncasecmp("true", arg, 4) || 0 == strncasecmp("1", arg, 1) || 0 == strncasecmp("yes", arg, 3)); + return (nullptr == arg || ts::iequals("true", arg) || ts::iequals("1", arg) || ts::iequals("yes", arg)); } // This parses and populates the BgFetchRule linked list (_rules). diff --git a/plugins/experimental/cookie_remap/cookiejar.cc b/plugins/experimental/cookie_remap/cookiejar.cc index cca2982e6dd..4c5a0c4bf48 100644 --- a/plugins/experimental/cookie_remap/cookiejar.cc +++ b/plugins/experimental/cookie_remap/cookiejar.cc @@ -18,6 +18,7 @@ #include "cookiejar.h" #include "strip.h" +#include #include /* allowed cookie-name definition from RFC diff --git a/plugins/experimental/custom_redirect/custom_redirect.cc b/plugins/experimental/custom_redirect/custom_redirect.cc index 84ee69e123a..91949eab537 100644 --- a/plugins/experimental/custom_redirect/custom_redirect.cc +++ b/plugins/experimental/custom_redirect/custom_redirect.cc @@ -31,6 +31,7 @@ #include #include #include +#include #include #include @@ -63,8 +64,8 @@ handle_response(TSHttpTxn txnp, TSCont /* contp ATS_UNUSED */) } else { int method_len; const char *method = TSHttpHdrMethodGet(req_bufp, req_loc, &method_len); - if ((return_code == TS_HTTP_STATUS_NONE || return_code == status) && - ((strncasecmp(method, TS_HTTP_METHOD_GET, TS_HTTP_LEN_GET) == 0))) { + if ((return_code == TS_HTTP_STATUS_NONE || return_code == status) && method != nullptr && + ts::iequals(std::string_view{method, static_cast(method_len)}, TS_HTTP_METHOD_GET)) { redirect_url_loc = TSMimeHdrFieldFind(resp_bufp, resp_loc, redirect_url_header, redirect_url_header_len); if (redirect_url_loc) { diff --git a/plugins/experimental/inliner/cache-handler.h b/plugins/experimental/inliner/cache-handler.h index e520a55bd99..22589c00656 100644 --- a/plugins/experimental/inliner/cache-handler.h +++ b/plugins/experimental/inliner/cache-handler.h @@ -25,6 +25,7 @@ #include #include +#include #include #include #include diff --git a/plugins/experimental/inliner/png.h b/plugins/experimental/inliner/png.h index a1464d3cdaa..22b75a68503 100644 --- a/plugins/experimental/inliner/png.h +++ b/plugins/experimental/inliner/png.h @@ -25,6 +25,7 @@ #include #include +#include #include #include diff --git a/plugins/header_rewrite/header_rewrite_test.cc b/plugins/header_rewrite/header_rewrite_test.cc index cb166fd7040..750b6870090 100644 --- a/plugins/header_rewrite/header_rewrite_test.cc +++ b/plugins/header_rewrite/header_rewrite_test.cc @@ -557,6 +557,24 @@ test_tokenizer() END_TEST(); } + { + SimpleTokenizerTest p("{a}%{PATH}"); + CHECK_EQ(p.get_tokens().size(), 2UL); + CHECK_EQ(p.get_tokens()[0], "{a}"); + CHECK_EQ(p.get_tokens()[1], "%{PATH}"); + + END_TEST(); + } + + { + SimpleTokenizerTest p("%{PATH}"); + CHECK_EQ(p.get_tokens().size(), 2UL); + CHECK_EQ(p.get_tokens()[0], ""); + CHECK_EQ(p.get_tokens()[1], "%{PATH}"); + + END_TEST(); + } + return errors; } diff --git a/plugins/header_rewrite/parser.cc b/plugins/header_rewrite/parser.cc index c1f467d39a8..ed3a7e27da1 100644 --- a/plugins/header_rewrite/parser.cc +++ b/plugins/header_rewrite/parser.cc @@ -344,7 +344,7 @@ HRWSimpleTokenizer::HRWSimpleTokenizer(const std::string &line) switch (state) { case PARSER_DEFAULT: if ((line[i] == '{') || (line[i] == '<')) { - if (line[i - 1] == '%') { + if (i > 0 && line[i - 1] == '%') { // pickup what we currently have cur_token_length = i - cur_token_start - 1; if (cur_token_length > 0) { diff --git a/plugins/lua/ts_lua.cc b/plugins/lua/ts_lua.cc index f4af729070b..943a3e12d84 100644 --- a/plugins/lua/ts_lua.cc +++ b/plugins/lua/ts_lua.cc @@ -24,6 +24,7 @@ #include #include "ts_lua_util.h" +#include "tsutil/StringCompare.h" extern "C" { #include "luajit.h" @@ -242,7 +243,7 @@ lifecycleHandler(TSCont contp, TSEvent /* event ATS_UNUSED */, void *edata) { // ensure the message is for ts_lua TSPluginMsg *const msgp = (TSPluginMsg *)edata; - if (0 != strncasecmp(msgp->tag, TS_LUA_DEBUG_TAG, strlen(msgp->tag))) { + if (!ts::iequals(msgp->tag, TS_LUA_DEBUG_TAG)) { return TS_EVENT_NONE; } diff --git a/plugins/multiplexer/ats-multiplexer.cc b/plugins/multiplexer/ats-multiplexer.cc index a6e9773351e..d4ab7fb5c3c 100644 --- a/plugins/multiplexer/ats-multiplexer.cc +++ b/plugins/multiplexer/ats-multiplexer.cc @@ -21,6 +21,7 @@ limitations under the License. */ #include +#include #include #include diff --git a/plugins/prefetch/configs.cc b/plugins/prefetch/configs.cc index 9e86a9a2bcc..9263f787a34 100644 --- a/plugins/prefetch/configs.cc +++ b/plugins/prefetch/configs.cc @@ -21,14 +21,14 @@ * @brief Plugin configuration. */ -#include /* std::from_chars() */ -#include /* strlen() */ -#include /* std::ifstream */ -#include /* getopt_long() */ -#include /* std::istringstream */ -#include /* strncasecmp() */ +#include /* std::from_chars() */ +#include /* strlen() */ +#include /* std::ifstream */ +#include /* getopt_long() */ +#include /* std::istringstream */ #include "configs.h" +#include "tsutil/StringCompare.h" DbgCtl Bg_dbg_ctl{PLUGIN_NAME}; @@ -47,7 +47,7 @@ commaSeparateString(ContainerType &c, const String &input) static bool isTrue(const char *arg) { - return (0 == strncasecmp("true", arg, 4) || 0 == strncasecmp("1", arg, 1) || 0 == strncasecmp("yes", arg, 3)); + return (ts::iequals("true", arg) || ts::iequals("1", arg) || ts::iequals("yes", arg)); } static String @@ -66,13 +66,6 @@ fetchOverflowString(const EvalPolicy policy) } } -static bool -iequals(const StringView lhs, const StringView rhs) -{ - return std::equal(lhs.begin(), lhs.end(), rhs.begin(), rhs.end(), - [](const char a, const char b) { return tolower(a) == tolower(b); }); -} - bool PrefetchConfig::setFetchOverflow(const char *optarg) { @@ -83,7 +76,7 @@ PrefetchConfig::setFetchOverflow(const char *optarg) _fetchOverflow = EvalPolicy::Overflow32; } else if (StringView("64") == optarg) { _fetchOverflow = EvalPolicy::Overflow64; - } else if (iequals("bignum", optarg)) { + } else if (ts::iequals("bignum", optarg)) { _fetchOverflow = EvalPolicy::Bignum; } else { return false; diff --git a/plugins/regex_revalidate/regex_revalidate.cc b/plugins/regex_revalidate/regex_revalidate.cc index 5e8d8cf2f7b..7825da23d41 100644 --- a/plugins/regex_revalidate/regex_revalidate.cc +++ b/plugins/regex_revalidate/regex_revalidate.cc @@ -34,6 +34,7 @@ #include #include "tsutil/Regex.h" +#include "tsutil/StringCompare.h" #define CONFIG_TMOUT 60000 #define FREE_TMOUT 300000 @@ -318,9 +319,9 @@ load_state(plugin_state_t *pstate, invalidate_t **ilist) } auto const type = matches[4]; - if (0 == strncasecmp(type.data(), RESULT_STALE, type.length())) { + if (ts::iequals(type, RESULT_STALE)) { Dbg(dbg_ctl, "state: regex line set to result type %s: '%s'", RESULT_STALE, inv->regex_text); - } else if (0 == strncasecmp(type.data(), RESULT_MISS, type.length())) { + } else if (ts::iequals(type, RESULT_MISS)) { Dbg(dbg_ctl, "state: regex line set to result type %s: '%s'", RESULT_MISS, inv->regex_text); inv->new_result = TS_CACHE_LOOKUP_MISS; } else { @@ -434,10 +435,10 @@ load_config(plugin_state_t *pstate, invalidate_t **ilist) if (5 == rc) { auto const type = matches[4]; - if (0 == strncasecmp(type.data(), RESULT_MISS, type.length())) { + if (ts::iequals(type, RESULT_MISS)) { Dbg(dbg_ctl, "Regex line set to result type %s: '%s'", RESULT_MISS, i->regex_text); i->new_result = TS_CACHE_LOOKUP_MISS; - } else if (0 != strncasecmp(type.data(), RESULT_STALE, type.length())) { + } else if (!ts::iequals(type, RESULT_STALE)) { Dbg(dbg_ctl, "Unknown regex line result type '%.*s', using default '%s' '%s'", (int)type.length(), type.data(), RESULT_STALE, i->regex_text); } diff --git a/plugins/stats_over_http/stats_over_http.cc b/plugins/stats_over_http/stats_over_http.cc index c5b208261e8..958569fcc14 100644 --- a/plugins/stats_over_http/stats_over_http.cc +++ b/plugins/stats_over_http/stats_over_http.cc @@ -49,6 +49,7 @@ #include "swoc/TextView.h" #include "tscore/ink_config.h" #include +#include #if HAVE_BROTLI_ENCODE_H #include @@ -1150,21 +1151,27 @@ stats_origin(TSCont contp, TSEvent /* event ATS_UNUSED */, void *edata) my_state->output_format = output_format_t::JSON_OUTPUT; // default to json output // accept header exists, use it to determine response type if (accept_field != TS_NULL_MLOC) { - int len = -1; - const char *str = TSMimeHdrFieldValueStringGet(reqp, hdr_loc, accept_field, -1, &len); + int len = -1; + const char *str = TSMimeHdrFieldValueStringGet(reqp, hdr_loc, accept_field, -1, &len); + std::string_view accept{}; + + if (str != nullptr && len > 0) { + accept = std::string_view{str, static_cast(len)}; + } // Parse the Accept header, default to JSON output unless its another supported format - if (!strncasecmp(str, "text/csv", len)) { + if (ts::iequals(accept, "text/csv")) { Dbg(dbg_ctl, "Saw text/csv in accept header, sending CSV output."); my_state->output_format = output_format_t::CSV_OUTPUT; - } else if (!strncasecmp(str, "text/plain; version=0.0.4", len)) { + } else if (ts::iequals(accept, "text/plain; version=0.0.4")) { Dbg(dbg_ctl, "Saw text/plain; version=0.0.4 in accept header, sending Prometheus output."); my_state->output_format = output_format_t::PROMETHEUS_OUTPUT; - } else if (!strncasecmp(str, "text/plain; version=2.0.0", len)) { + } else if (ts::iequals(accept, "text/plain; version=2.0.0")) { Dbg(dbg_ctl, "Saw text/plain; version=2.0.0 in accept header, sending Prometheus v2 output."); my_state->output_format = output_format_t::PROMETHEUS_V2_OUTPUT; } else { - Dbg(dbg_ctl, "Saw %.*s in accept header, defaulting to JSON output.", len, str); + Dbg(dbg_ctl, "Saw %.*s in accept header, defaulting to JSON output.", static_cast(accept.size()), + accept.empty() ? "" : accept.data()); my_state->output_format = output_format_t::JSON_OUTPUT; } } diff --git a/src/api/InkAPI.cc b/src/api/InkAPI.cc index 5914d96313a..c99fbdfbf99 100644 --- a/src/api/InkAPI.cc +++ b/src/api/InkAPI.cc @@ -9262,8 +9262,7 @@ TSLogFieldRegister(std::string_view name, std::string_view symbol, TSLogType typ name.data(), symbol.data(), static_cast(type), [marshal_cb = std::move(marshal_cb)](void *sm, char *buf) -> int { return marshal_cb(reinterpret_cast(sm), buf); }, unmarshal_cb); - Log::global_field_list.add(field, false); - Log::field_symbol_hash.emplace(symbol.data(), field); + Log::register_field(field); return TS_SUCCESS; } diff --git a/src/iocore/cache/HttpTransactCache.cc b/src/iocore/cache/HttpTransactCache.cc index 47117ad90ee..55a12afbc93 100644 --- a/src/iocore/cache/HttpTransactCache.cc +++ b/src/iocore/cache/HttpTransactCache.cc @@ -30,6 +30,7 @@ #include "proxy/hdrs/HTTP.h" #include "proxy/hdrs/HttpCompat.h" #include "tscore/ink_time.h" +#include "tsutil/StringCompare.h" #include @@ -941,9 +942,7 @@ HttpTransactCache::calculate_quality_of_accept_encoding_match(MIMEField *accept_ } else { // does this document have the identity encoding? // for (c_value = c_values_list.head; c_value; c_value = c_value->next) { - auto c_encoding = c_value->str; - auto c_encoding_len = c_value->len; - if ((c_encoding_len >= 8) && (strncasecmp(c_encoding, "identity", 8) == 0)) { + if (ts::iequals(std::string_view{c_value->str, c_value->len}, "identity")) { is_identity_encoding = true; break; } diff --git a/src/iocore/net/CMakeLists.txt b/src/iocore/net/CMakeLists.txt index 97688a9b438..7f8f3317e50 100644 --- a/src/iocore/net/CMakeLists.txt +++ b/src/iocore/net/CMakeLists.txt @@ -144,6 +144,7 @@ if(BUILD_TESTING) test_net libinknet_stub.cc NetVCTest.cc + unit_tests/test_ConnectionTracker.cc unit_tests/test_ProxyProtocol.cc unit_tests/test_SSLCertLookup.cc unit_tests/test_SSLNetVConnectionAsyncEp.cc diff --git a/src/iocore/net/ConnectionTracker.cc b/src/iocore/net/ConnectionTracker.cc index 85519616328..51ed5a734ec 100644 --- a/src/iocore/net/ConnectionTracker.cc +++ b/src/iocore/net/ConnectionTracker.cc @@ -27,6 +27,7 @@ #include "swoc/IPAddr.h" #include +#include using namespace std::literals; @@ -77,8 +78,8 @@ const MgmtConverter ConnectionTracker::SERVER_MATCH_CONV{ // records paths do the range checking instead -- records.yaml validates the value and the reload // callbacks below clamp -- so an out of range value is only reachable by a plugin that sets one // deliberately. Both settings degrade safely if that happens: any non-zero metric_enabled enables -// metrics, and any metric_aggregate outside 0..2 publishes both the aggregate and the per group -// metrics, the same as AGGREGATE_GROUP. +// metrics, and any metric_aggregate outside 0..3 publishes everything, the same as +// AGGREGATE_GROUP. const MgmtConverter ConnectionTracker::METRIC_ENABLED_CONV{ [](const void *data) -> MgmtInt { return static_cast(*static_cast(data)); }, [](void *data, MgmtInt i) -> void { @@ -189,7 +190,7 @@ Config_Update_Conntrack_Metric_Aggregate(const char * /* name ATS_UNUSED */, Rec if (RECD_INT == dtype) { auto level = std::clamp(static_cast(data.rec_int), static_cast(ConnectionTracker::AGGREGATE_NONE), - static_cast(ConnectionTracker::AGGREGATE_ONLY)); + static_cast(ConnectionTracker::AGGREGATE_SUM)); config->metric_aggregate = static_cast(level); return true; } @@ -487,33 +488,70 @@ ConnectionTracker::Group::Group(DirectionType direction, Key const &key, std::st std::string _host_metric_name = host_metric_name(key, fqdn, _global_config->metric_prefix); bool const has_aggregate = !_host_metric_name.empty(); - if (has_aggregate && metric_aggregate != AGGREGATE_NONE) { - Metrics::Derived::add_source("proxy.process.http.per_server.current_connection." + _host_metric_name, - Metrics::MetricType::GAUGE, _count_metric, Metrics::Derived::Op::SUM); - Metrics::Derived::add_source("proxy.process.http.per_server.total_connection." + _host_metric_name, - Metrics::MetricType::COUNTER, _count_total_metric, Metrics::Derived::Op::SUM); - Metrics::Derived::add_source("proxy.process.http.per_server.blocked_connection." + _host_metric_name, - Metrics::MetricType::COUNTER, _blocked_metric, Metrics::Derived::Op::SUM); + // A plugin can set an out of range value through the overridable config, see + // METRIC_AGGREGATE_CONV. Anything unrecognized publishes everything. + if (metric_aggregate < AGGREGATE_NONE || metric_aggregate > AGGREGATE_SUM) { + metric_aggregate = AGGREGATE_GROUP; + } + + // See MetricAggregate for the table these three implement. A group with no hostname to + // aggregate under keeps its own metrics whatever the setting says, since suppressing them would + // report nothing at all for that upstream. + bool const publish_sums = has_aggregate && (metric_aggregate == AGGREGATE_GROUP || metric_aggregate == AGGREGATE_SUM); + bool const publish_max = has_aggregate && metric_aggregate != AGGREGATE_NONE; + bool const publish_group = !has_aggregate || metric_aggregate == AGGREGATE_NONE || metric_aggregate == AGGREGATE_GROUP; + + std::array const sum_names{ + "proxy.process.http.per_server.current_connection." + _host_metric_name, + "proxy.process.http.per_server.total_connection." + _host_metric_name, + "proxy.process.http.per_server.blocked_connection." + _host_metric_name, + }; + std::array const group_names{ + "proxy.process.http.per_server.current_connection." + _metric_name, + "proxy.process.http.per_server.total_connection." + _metric_name, + "proxy.process.http.per_server.blocked_connection." + _metric_name, + }; + std::string const max_name = "proxy.process.http.per_server.current_connection.max." + _host_metric_name; + + // metric_aggregate is dynamic and overridable, so this group may well have published a name + // under an earlier value. A published name is never removed from the store, so without + // withdrawing it here it would report for the life of the process no matter what the setting + // says. Re-registering a source republishes it if the setting changes back. + if (publish_sums) { + Metrics::Derived::add_source(sum_names[0], Metrics::MetricType::GAUGE, _count_metric, Metrics::Derived::Op::SUM); + Metrics::Derived::add_source(sum_names[1], Metrics::MetricType::COUNTER, _count_total_metric, Metrics::Derived::Op::SUM); + Metrics::Derived::add_source(sum_names[2], Metrics::MetricType::COUNTER, _blocked_metric, Metrics::Derived::Op::SUM); + } else if (has_aggregate) { + // Stop contributing rather than unlist: every group of this hostname shares these names, so + // one that does not want them must not remove a name another is still publishing. + Metrics::Derived::remove_source(sum_names[0], _count_metric); + Metrics::Derived::remove_source(sum_names[1], _count_total_metric); + Metrics::Derived::remove_source(sum_names[2], _blocked_metric); + } + + if (publish_max) { // The largest current count among this hostname's groups, sampled. Deliberately taken over // the instantaneous gauge rather than each group's all time peak, so the value falls again // and a maximum over time can be computed by whatever scrapes it. - Metrics::Derived::add_source("proxy.process.http.per_server.current_connection_max." + _host_metric_name, - Metrics::MetricType::GAUGE, _count_metric, Metrics::Derived::Op::MAX); + Metrics::Derived::add_source(max_name, Metrics::MetricType::GAUGE, _count_metric, Metrics::Derived::Op::MAX); + } else if (has_aggregate) { + Metrics::Derived::remove_source(max_name, _count_metric); } - // AGGREGATE_ONLY suppresses the per group metrics to keep the published count proportional to - // hostnames. Without an aggregate to stand in for them there would be nothing at all reported - // for this group, so in that case publish them regardless. - if (metric_aggregate != AGGREGATE_ONLY || !has_aggregate) { + if (publish_group) { // Mirror the per group metrics into the published store under their own name. A single // source SUM combines nothing, but the published value is still a sample: it is whatever // the last derived tick read, and it reads 0 from creation until that first tick. - Metrics::Derived::add_source("proxy.process.http.per_server.current_connection." + _metric_name, Metrics::MetricType::GAUGE, - _count_metric, Metrics::Derived::Op::SUM); - Metrics::Derived::add_source("proxy.process.http.per_server.total_connection." + _metric_name, Metrics::MetricType::COUNTER, - _count_total_metric, Metrics::Derived::Op::SUM); - Metrics::Derived::add_source("proxy.process.http.per_server.blocked_connection." + _metric_name, Metrics::MetricType::COUNTER, - _blocked_metric, Metrics::Derived::Op::SUM); + Metrics::Derived::add_source(group_names[0], Metrics::MetricType::GAUGE, _count_metric, Metrics::Derived::Op::SUM); + Metrics::Derived::add_source(group_names[1], Metrics::MetricType::COUNTER, _count_total_metric, Metrics::Derived::Op::SUM); + Metrics::Derived::add_source(group_names[2], Metrics::MetricType::COUNTER, _blocked_metric, Metrics::Derived::Op::SUM); + } else { + // Same mechanism as the aggregates above, though these names have only this group as a + // source. It leaves nothing behind for the derived pass to keep recomputing into a name that + // is no longer published. + Metrics::Derived::remove_source(group_names[0], _count_metric); + Metrics::Derived::remove_source(group_names[1], _count_total_metric); + Metrics::Derived::remove_source(group_names[2], _blocked_metric); } if (dbg_ctl.on()) { diff --git a/src/iocore/net/unit_tests/test_ConnectionTracker.cc b/src/iocore/net/unit_tests/test_ConnectionTracker.cc new file mode 100644 index 00000000000..e427b5bd0d9 --- /dev/null +++ b/src/iocore/net/unit_tests/test_ConnectionTracker.cc @@ -0,0 +1,262 @@ +/** @file + + Catch based unit tests for connection tracking configuration. + + @section license License + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#include "iocore/net/ConnectionTracker.h" +#include "iocore/net/Net.h" +#include "tscore/ink_inet.h" +#include "tsutil/Metrics.h" + +#include + +#include + +namespace +{ + +constexpr std::string_view FQDN{"unit.test.origin"}; + +// Whether the published store enumerates this name. Deliberately for_each rather than lookup(), +// because enumeration is what traffic_ctl, the JSONRPC record lookup and stats_over_http walk, and +// so is what "published" means to an operator. +bool +is_published(std::string_view metric_name) +{ + bool found = false; + + ts::Metrics::instance().for_each( + [&](std::string_view name, ts::Metrics::MetricType, int64_t) { found |= (name == metric_name); }); + + return found; +} + +std::string +group_metric(std::string_view stem, std::string_view addr) +{ + return std::string("proxy.process.http.per_server.").append(stem).append(".").append(FQDN).append(".").append(addr); +} + +std::string +host_metric(std::string_view stem) +{ + return std::string("proxy.process.http.per_server.").append(stem).append(".").append(FQDN); +} + +// One upstream connection, opened and closed, following the same path as production: HttpSM +// reserves and then drops the group into the PoolableSession, and the session releases it when the +// connection closes. Group::release() is what erases the group at a zero count, and only that makes +// the next transaction to the same upstream construct a fresh Group and re-evaluate +// metric_aggregate. TxnState::release() alone decrements without erasing. +void +open_and_close_connection(ConnectionTracker::TxnConfig const &txn, IpEndpoint const &addr) +{ + auto state = ConnectionTracker::obtain_outbound(txn, FQDN, addr); + + REQUIRE(state.is_active()); + state.reserve(); + + auto group = state.drop(); + group->release(); +} + +ConnectionTracker::TxnConfig & +test_config() +{ + // config_init keeps pointers to these for the records callbacks, so they must outlive the test. + static ConnectionTracker::GlobalConfig global; + static ConnectionTracker::TxnConfig txn; + static bool initialized = false; + + if (!initialized) { + ink_net_init(NET_SYSTEM_MODULE_PUBLIC_VERSION); + ConnectionTracker::config_init(&global, &txn, [](const char *, RecDataT, RecData, void *) -> int { return REC_ERR_OKAY; }); + initialized = true; + } + + return txn; +} + +} // namespace + +TEST_CASE("ConnectionTracker aggregate metric publication", "[libinknet][ConnectionTracker]") +{ + auto &txn = test_config(); + + txn.metric_enabled = 1; + txn.server_match = ConnectionTracker::MATCH_BOTH; + + IpEndpoint addr; + REQUIRE(ats_ip_pton("10.9.8.7:443", &addr) == 0); + + const std::string current_group = group_metric("current_connection", "10.9.8.7:443"); + const std::string total_group = group_metric("total_connection", "10.9.8.7:443"); + const std::string blocked_group = group_metric("blocked_connection", "10.9.8.7:443"); + + SECTION("AGGREGATE_NONE publishes the per group metrics and no aggregate") + { + txn.metric_aggregate = ConnectionTracker::AGGREGATE_NONE; + open_and_close_connection(txn, addr); + + CHECK(is_published(current_group)); + CHECK(is_published(total_group)); + CHECK(is_published(blocked_group)); + CHECK_FALSE(is_published(host_metric("current_connection.max"))); + } + + SECTION("AGGREGATE_GROUP publishes the per group metrics, the sums and the max") + { + txn.metric_aggregate = ConnectionTracker::AGGREGATE_GROUP; + open_and_close_connection(txn, addr); + + CHECK(is_published(current_group)); + CHECK(is_published(host_metric("current_connection"))); + CHECK(is_published(host_metric("total_connection"))); + CHECK(is_published(host_metric("blocked_connection"))); + CHECK(is_published(host_metric("current_connection.max"))); + } + + SECTION("AGGREGATE_MAX publishes the max and nothing else") + { + txn.metric_aggregate = ConnectionTracker::AGGREGATE_MAX; + open_and_close_connection(txn, addr); + + CHECK(is_published(host_metric("current_connection.max"))); + + CHECK_FALSE(is_published(host_metric("current_connection"))); + CHECK_FALSE(is_published(host_metric("total_connection"))); + CHECK_FALSE(is_published(host_metric("blocked_connection"))); + CHECK_FALSE(is_published(current_group)); + CHECK_FALSE(is_published(total_group)); + CHECK_FALSE(is_published(blocked_group)); + } + + SECTION("AGGREGATE_SUM publishes the sums and the max, but not the per group metrics") + { + txn.metric_aggregate = ConnectionTracker::AGGREGATE_SUM; + open_and_close_connection(txn, addr); + + CHECK(is_published(host_metric("current_connection"))); + CHECK(is_published(host_metric("total_connection"))); + CHECK(is_published(host_metric("blocked_connection"))); + CHECK(is_published(host_metric("current_connection.max"))); + + CHECK_FALSE(is_published(current_group)); + CHECK_FALSE(is_published(total_group)); + CHECK_FALSE(is_published(blocked_group)); + } + + SECTION("switching to AGGREGATE_MAX retracts already published per group metrics") + { + // The production sequence: run for a while with the per group metrics published, then change + // the setting. Without a retraction the first set of names is published forever. + txn.metric_aggregate = ConnectionTracker::AGGREGATE_NONE; + open_and_close_connection(txn, addr); + REQUIRE(is_published(current_group)); + + txn.metric_aggregate = ConnectionTracker::AGGREGATE_MAX; + open_and_close_connection(txn, addr); + + CHECK_FALSE(is_published(current_group)); + CHECK_FALSE(is_published(total_group)); + CHECK_FALSE(is_published(blocked_group)); + CHECK(is_published(host_metric("current_connection.max"))); + } + + SECTION("switching from AGGREGATE_SUM to AGGREGATE_MAX retracts the sums") + { + // The sums are aggregates rather than per group names, but they are published the same way and + // so need withdrawing the same way when the setting stops asking for them. + txn.metric_aggregate = ConnectionTracker::AGGREGATE_SUM; + open_and_close_connection(txn, addr); + REQUIRE(is_published(host_metric("current_connection"))); + + txn.metric_aggregate = ConnectionTracker::AGGREGATE_MAX; + open_and_close_connection(txn, addr); + + CHECK_FALSE(is_published(host_metric("current_connection"))); + CHECK_FALSE(is_published(host_metric("total_connection"))); + CHECK_FALSE(is_published(host_metric("blocked_connection"))); + CHECK(is_published(host_metric("current_connection.max"))); + } + + SECTION("switching from AGGREGATE_MAX to AGGREGATE_SUM republishes the sums") + { + txn.metric_aggregate = ConnectionTracker::AGGREGATE_MAX; + open_and_close_connection(txn, addr); + REQUIRE_FALSE(is_published(host_metric("total_connection"))); + + txn.metric_aggregate = ConnectionTracker::AGGREGATE_SUM; + open_and_close_connection(txn, addr); + + CHECK(is_published(host_metric("current_connection"))); + CHECK(is_published(host_metric("total_connection"))); + CHECK(is_published(host_metric("blocked_connection"))); + } + + SECTION("switching back to AGGREGATE_GROUP republishes the per group metrics") + { + txn.metric_aggregate = ConnectionTracker::AGGREGATE_MAX; + open_and_close_connection(txn, addr); + REQUIRE_FALSE(is_published(current_group)); + + txn.metric_aggregate = ConnectionTracker::AGGREGATE_GROUP; + open_and_close_connection(txn, addr); + + CHECK(is_published(current_group)); + CHECK(is_published(host_metric("current_connection"))); + } + + SECTION("one hostname's groups do not unlist each other's aggregate") + { + // metric_aggregate is overridable, so two mappings to one hostname can disagree. Both groups + // share the hostname's aggregate names, so a group that does not want them must stop + // contributing rather than unlist a name the other one is still publishing. + IpEndpoint other; + REQUIRE(ats_ip_pton("10.9.8.5:443", &other) == 0); + + txn.metric_aggregate = ConnectionTracker::AGGREGATE_SUM; + open_and_close_connection(txn, addr); + REQUIRE(is_published(host_metric("current_connection"))); + + txn.metric_aggregate = ConnectionTracker::AGGREGATE_MAX; + open_and_close_connection(txn, other); + + CHECK(is_published(host_metric("current_connection"))); + CHECK(is_published(host_metric("current_connection.max"))); + } + + SECTION("a group with no aggregate keeps its own metrics whatever the setting") + { + // Only MATCH_BOTH yields a hostname to gather under, so a MATCH_PORT group has no aggregate. + // Suppressing it would report nothing at all for that upstream. + txn.server_match = ConnectionTracker::MATCH_PORT; + + IpEndpoint port_addr; + REQUIRE(ats_ip_pton("10.9.8.6:80", &port_addr) == 0); + + for (auto level : {ConnectionTracker::AGGREGATE_MAX, ConnectionTracker::AGGREGATE_SUM}) { + txn.metric_aggregate = level; + open_and_close_connection(txn, port_addr); + CHECK(is_published("proxy.process.http.per_server.current_connection.10.9.8.6:80")); + } + } +} diff --git a/src/mgmt/config/ReloadCoordinator.cc b/src/mgmt/config/ReloadCoordinator.cc index 6add40d8f00..7c0e08a0e05 100644 --- a/src/mgmt/config/ReloadCoordinator.cc +++ b/src/mgmt/config/ReloadCoordinator.cc @@ -24,6 +24,7 @@ #include "mgmt/config/ReloadCoordinator.h" #include +#include #include #include diff --git a/src/proxy/ControlBase.cc b/src/proxy/ControlBase.cc index 564889183dc..0cfbb4ac3f7 100644 --- a/src/proxy/ControlBase.cc +++ b/src/proxy/ControlBase.cc @@ -34,6 +34,8 @@ #include "tscore/ink_defs.h" #include "tscore/ink_time.h" +#include "tsutil/StringCompare.h" + #include "proxy/hdrs/URL.h" #include "tscore/Tokenizer.h" #include "proxy/ControlBase.h" @@ -661,9 +663,9 @@ InternalMod::make(char *value, const char **error) { InternalMod tmp; - if (0 == strncasecmp("false", value, 5)) { + if (ts::iequals("false", value)) { tmp.flag = false; - } else if (0 == strncasecmp("true", value, 4)) { + } else if (ts::iequals("true", value)) { tmp.flag = true; } else { *error = "Value must be true or false"; diff --git a/src/proxy/ProxyTransaction.cc b/src/proxy/ProxyTransaction.cc index 05fb0c8877e..4658acca3b7 100644 --- a/src/proxy/ProxyTransaction.cc +++ b/src/proxy/ProxyTransaction.cc @@ -38,7 +38,12 @@ DbgCtl dbg_ctl_http_txn{"http_txn"}; extern ClassAllocator httpSMAllocator; -ProxyTransaction::ProxyTransaction(ProxySession *session) : VConnection(nullptr), _proxy_ssn(session) {} +ProxyTransaction::ProxyTransaction(ProxySession *session) : VConnection(nullptr), _proxy_ssn(session) +{ + if (_proxy_ssn != nullptr && _proxy_ssn->acceptor != nullptr) { + upstream_outbound_options = _proxy_ssn->acceptor->options(); + } +} ProxyTransaction::~ProxyTransaction() { @@ -55,6 +60,9 @@ ProxyTransaction::new_transaction(bool from_early_data) // connection re-use ink_release_assert(_proxy_ssn != nullptr); + if (_proxy_ssn->acceptor != nullptr) { + upstream_outbound_options = _proxy_ssn->acceptor->options(); + } _sm = THREAD_ALLOC(httpSMAllocator, this_thread()); _sm->init(from_early_data); HttpTxnDebug("[%" PRId64 "] Starting transaction %d using sm [%" PRId64 "]", _proxy_ssn->connection_id(), diff --git a/src/proxy/hdrs/URL.cc b/src/proxy/hdrs/URL.cc index d579cf706c1..877dabe30c1 100644 --- a/src/proxy/hdrs/URL.cc +++ b/src/proxy/hdrs/URL.cc @@ -76,8 +76,10 @@ int URL_WKSIDX_MMST; namespace { -// Whether we should implement url_CryptoHash_get() using url_CryptoHash_get_fast(). Note that -// url_CryptoHash_get_fast() does NOT produce the same result as url_CryptoHash_get_general(). +// Whether url_CryptoHash_get_92() should use url_CryptoHash_get_fast_92(). +// The fast implementation emits the 9.2 ";" path/params separator and is +// therefore valid only for 9.2 keys; the canonical url_CryptoHash_get() +// always uses url_CryptoHash_get_general(). constexpr int url_hash_method = 0; // Buffer size for url_CryptoHash_get() and url_CryptoHash_get_92(). @@ -1718,11 +1720,18 @@ memcpy_tolower(char *d, const char *s, int n) } } -// fast path for CryptoHash, HTTP, no user/password/params/query, -// no buffer overflow, no unescaping needed +// Fast path for the 9.2 CryptoHash: HTTP, no user/password/params/query, +// no buffer overflow, no unescaping needed. +// +// Emits the ";" path/params separator that 9.2 hashed between the path and +// params components, so it is valid only for url_CryptoHash_get_92(): the +// canonical url_CryptoHash_get_general() does not emit the separator, and +// using this fast path there would make canonical keys collide with 9.2 keys. +// The 9.2 caller must also ensure the path carries no literal ';' (see +// has_path_params()), otherwise the separator would be duplicated. static inline void -url_CryptoHash_get_fast(const URLImpl *url, CryptoContext &ctx, CryptoHash *hash, cache_generation_t generation) +url_CryptoHash_get_fast_92(const URLImpl *url, CryptoContext &ctx, CryptoHash *hash, cache_generation_t generation) { char buffer[BUFSIZE]; char *p; @@ -1862,19 +1871,8 @@ void url_CryptoHash_get(const URLImpl *url, CryptoHash *hash, bool ignore_query, cache_generation_t generation) { URLHashContext ctx; - if ((url_hash_method != 0) && (url->m_url_type == URLType::HTTP) && - ((url->m_len_user + url->m_len_password + (ignore_query ? 0 : url->m_len_query)) == 0) && - (10u + url->m_len_scheme + url->m_len_host + url->m_len_path < BUFSIZE) && - (memchr(url->m_ptr_host, '%', url->m_len_host) == nullptr) && (memchr(url->m_ptr_path, '%', url->m_len_path) == nullptr)) { - url_CryptoHash_get_fast(url, ctx, hash, generation); -#ifdef DEBUG - CryptoHash hash_general; - url_CryptoHash_get_general(url, ctx, hash_general, ignore_query, generation); - ink_assert(*hash == hash_general); -#endif - } else { - url_CryptoHash_get_general(url, ctx, *hash, ignore_query, generation); - } + // No fast path here: the only one left emits the 9.2 ";" separator. + url_CryptoHash_get_general(url, ctx, *hash, ignore_query, generation); } static inline void @@ -1908,8 +1906,16 @@ url_CryptoHash_get_general_92(const URLImpl *url, CryptoContext &ctx, CryptoHash ends[7] = strs[7] + 1; ends[8] = strs[8] + url->m_len_path; - strs[9] = ";"; - strs[10] = url->m_ptr_params; + // ATS 9.2 split "/path;params" into separate path and params components and + // hashed them as path + ";" + params. That parsing was removed, so ";params" + // now stays inside the path and already spells the same byte sequence. Adding + // the separator again would append a ";" that 9.2 never emitted, so only add + // it when the path does not carry one. The params component itself is always + // empty now; it is left out rather than read back as an empty string. + bool const path_has_params = url->has_path_params(); + + strs[9] = path_has_params ? nullptr : ";"; + strs[10] = nullptr; strs[11] = "?"; // Special case for the query paramters, allowing us to ignore them if requested @@ -1921,8 +1927,8 @@ url_CryptoHash_get_general_92(const URLImpl *url, CryptoContext &ctx, CryptoHash ends[12] = nullptr; } - ends[9] = strs[9] + 1; - ends[10] = strs[10] + url->m_len_params; + ends[9] = path_has_params ? nullptr : strs[9] + 1; + ends[10] = nullptr; ends[11] = strs[11] + 1; p = buffer; @@ -1979,11 +1985,11 @@ void url_CryptoHash_get_92(const URLImpl *url, CryptoHash *hash, bool ignore_query, cache_generation_t generation) { URLHashContext ctx; - if ((url_hash_method != 0) && (url->m_url_type == URLType::HTTP) && + if ((url_hash_method != 0) && (url->m_url_type == URLType::HTTP) && !url->has_path_params() && ((url->m_len_user + url->m_len_password + url->m_len_params + (ignore_query ? 0 : url->m_len_query)) == 0) && (10u + url->m_len_scheme + url->m_len_host + url->m_len_path < BUFSIZE) && (memchr(url->m_ptr_host, '%', url->m_len_host) == nullptr) && (memchr(url->m_ptr_path, '%', url->m_len_path) == nullptr)) { - url_CryptoHash_get_fast(url, ctx, hash, generation); + url_CryptoHash_get_fast_92(url, ctx, hash, generation); #ifdef DEBUG CryptoHash hash_general; url_CryptoHash_get_general_92(url, ctx, hash_general, ignore_query, generation); diff --git a/src/proxy/hdrs/unit_tests/test_URL.cc b/src/proxy/hdrs/unit_tests/test_URL.cc index f1963c9039f..8f83d79165d 100644 --- a/src/proxy/hdrs/unit_tests/test_URL.cc +++ b/src/proxy/hdrs/unit_tests/test_URL.cc @@ -19,6 +19,8 @@ */ #include +#include +#include #include #include @@ -852,6 +854,119 @@ TEST_CASE("UrlPathGet", "[url][path_get]") } } +// ATS 9.2 hashed "path" ";" "params" as separate cache-key components, always +// emitting the separator. The params component was removed from the parser, so +// ";params" now lives inside the path and the separator is already there. That +// makes the 9.2 key expressible through the current algorithm: it is the +// current key of the same URL with exactly one ";" between path and query. +namespace +{ +/// A failing REQUIRE unwinds out of these helpers, so the heap is released by +/// scope exit rather than by a call that the unwind would skip. +struct HdrHeapDeleter { + void + operator()(HdrHeap *heap) const noexcept + { + heap->destroy(); + } +}; + +using HdrHeapPtr = std::unique_ptr; + +CryptoHash +hash92(char const *text) +{ + HdrHeapPtr heap{new_HdrHeap()}; + URL url; + + url.create(heap.get()); + REQUIRE(url.parse(text, strlen(text)) == ParseResult::DONE); + + CryptoHash hash; + + url.hash_get92(&hash); + + return hash; +} + +CryptoHash +hash_current(char const *text) +{ + HdrHeapPtr heap{new_HdrHeap()}; + URL url; + + url.create(heap.get()); + REQUIRE(url.parse(text, strlen(text)) == ParseResult::DONE); + + CryptoHash hash; + + url.hash_get(&hash); + + return hash; +} + +bool +has_params(char const *text) +{ + HdrHeapPtr heap{new_HdrHeap()}; + URL url; + + url.create(heap.get()); + REQUIRE(url.parse(text, strlen(text)) == ParseResult::DONE); + + return url.has_path_params(); +} +} // namespace + +TEST_CASE("UrlHashGet92 reproduces the 9.2 cache key", "[url][hash_get92]") +{ + SECTION("a path without params gains the 9.2 separator") + { + CHECK(hash92("http://foo.test/path") == hash_current("http://foo.test/path;")); + CHECK(hash92("http://foo.test/path?q=1") == hash_current("http://foo.test/path;?q=1")); + // A ';' inside the query is not a params segment: 9.2 stopped the params + // component at '?', so the path still gained a separator of its own. + CHECK(hash92("http://foo.test/a?b;c") == hash_current("http://foo.test/a;?b;c")); + } + + SECTION("a path carrying params already spells the 9.2 key") + { + CHECK(hash92("http://foo.test/a;b=1") == hash_current("http://foo.test/a;b=1")); + CHECK(hash92("http://foo.test/a;b;c") == hash_current("http://foo.test/a;b;c")); + CHECK(hash92("http://foo.test/a;b=1?q=1") == hash_current("http://foo.test/a;b=1?q=1")); + } + + SECTION("the keys diverge only when the path has no params segment") + { + // This divergence is the whole reason the compatibility lookup exists. + CHECK(hash92("http://foo.test/path") != hash_current("http://foo.test/path")); + CHECK(hash92("http://foo.test/path?q=1") != hash_current("http://foo.test/path?q=1")); + // ... and when it converges there is nothing for a second lookup to find. + CHECK(hash92("http://foo.test/a;b=1") == hash_current("http://foo.test/a;b=1")); + } + + SECTION("an escaped %3B is not a params segment") + { + // 9.2 split on the literal ';' only and hashed the path without unescaping, + // so "%3B" stayed in the path and the separator was still appended. The + // params check has to look at the raw path bytes, not the decoded ones. + CHECK(hash92("http://foo.test/a%3Bb=1") == hash_current("http://foo.test/a%3Bb=1;")); + CHECK(hash92("http://foo.test/a%3Bb=1?q=1") == hash_current("http://foo.test/a%3Bb=1;?q=1")); + CHECK(hash92("http://foo.test/a%3bb=1") == hash_current("http://foo.test/a%3bb=1;")); + // The escaped and literal forms were cached separately and must stay apart. + CHECK(hash92("http://foo.test/a%3Bb=1") != hash92("http://foo.test/a;b=1")); + } + + SECTION("has_path_params identifies the converging case") + { + CHECK(!has_params("http://foo.test/path")); + CHECK(!has_params("http://foo.test/a?b;c")); + CHECK(!has_params("http://foo.test/a%3Bb=1")); + CHECK(has_params("http://foo.test/a;b=1")); + CHECK(has_params("http://foo.test/a;b=1?q=1")); + } +} + // URL getters must not construct std::string_view from a nullptr pointer // (which is UB). Parts that are not present in the URL should return an // empty string_view with data() == nullptr. diff --git a/src/proxy/http/Http1ClientSession.cc b/src/proxy/http/Http1ClientSession.cc index e2c11b71d2c..32121e97b10 100644 --- a/src/proxy/http/Http1ClientSession.cc +++ b/src/proxy/http/Http1ClientSession.cc @@ -207,7 +207,6 @@ Http1ClientSession::new_connection(NetVConnection *new_vc, MIOBuffer *iobuf, IOB _reader = reader ? reader : read_buffer->alloc_reader(); trans.set_reader(_reader); - trans.upstream_outbound_options = *accept_options; _handle_if_ssl(new_vc); diff --git a/src/proxy/http/HttpCacheSM.cc b/src/proxy/http/HttpCacheSM.cc index 4ddcc27ddcc..eb28c820efb 100644 --- a/src/proxy/http/HttpCacheSM.cc +++ b/src/proxy/http/HttpCacheSM.cc @@ -294,7 +294,8 @@ HttpCacheSM::state_cache_open_write(int event, void *data) // than or equal to the max number of open write retries ink_assert(!write_retry_done()); - open_write(&cache_key, lookup_url, read_request_hdr, master_sm->t_state.cache_info.object_read, + open_write(&cache_key, lookup_url, read_request_hdr, + CompatCacheKey::write_info(master_sm->compatibility_cache_lookup, master_sm->t_state.cache_info.object_read), static_cast( (master_sm->t_state.cache_control.pin_in_cache_for < 0) ? 0 : master_sm->t_state.cache_control.pin_in_cache_for), retry_write, false); diff --git a/src/proxy/http/HttpProxyServerMain.cc b/src/proxy/http/HttpProxyServerMain.cc index 4701d9771cf..03a4537d876 100644 --- a/src/proxy/http/HttpProxyServerMain.cc +++ b/src/proxy/http/HttpProxyServerMain.cc @@ -173,19 +173,19 @@ make_net_accept_options(const HttpProxyPort *port, unsigned nthreads) static void MakeHttpProxyAcceptor(HttpProxyAcceptor &acceptor, HttpProxyPort &port, unsigned nthreads) { - NetProcessor::AcceptOptions &net_opt = acceptor._net_opt; - HttpSessionAccept::Options accept_opt; + NetProcessor::AcceptOptions &net_opt = acceptor._net_opt; + auto accept_opt = std::make_shared(); net_opt = make_net_accept_options(&port, nthreads); - accept_opt.f_outbound_transparent = port.m_outbound_transparent_p; - accept_opt.transport_type = port.m_type; - accept_opt.setHostResPreference(port.m_host_res_preference); - accept_opt.setTransparentPassthrough(port.m_transparent_passthrough); - accept_opt.setSessionProtocolPreference(port.m_session_protocol_preference); + accept_opt->f_outbound_transparent = port.m_outbound_transparent_p; + accept_opt->transport_type = port.m_type; + accept_opt->setHostResPreference(port.m_host_res_preference); + accept_opt->setTransparentPassthrough(port.m_transparent_passthrough); + accept_opt->setSessionProtocolPreference(port.m_session_protocol_preference); - accept_opt.outbound += HttpConfig::m_master.outbound; - accept_opt.outbound += port.m_outbound; // top priority, override master and base options. + accept_opt->outbound += HttpConfig::m_master.outbound; + accept_opt->outbound += port.m_outbound; // top priority, override master and base options. // OK the way this works is that the fallback for each port is a protocol // probe acceptor. For SSL ports, we can stack a NPN+ALPN acceptor in front @@ -209,12 +209,12 @@ MakeHttpProxyAcceptor(HttpProxyAcceptor &acceptor, HttpProxyPort &port, unsigned probe->proxy_protocol_ipmap = &HttpConfig::m_master.config_proxy_protocol_ip_addrs; if (port.m_session_protocol_preference.intersects(HTTP_PROTOCOL_SET)) { - http = new HttpSessionAccept(accept_opt); + http = new HttpSessionAccept(accept_opt, &port); probe->registerEndpoint(ProtocolProbeSessionAccept::ProtoGroupKey::HTTP, http); } if (port.m_session_protocol_preference.intersects(HTTP2_PROTOCOL_SET)) { - probe->registerEndpoint(ProtocolProbeSessionAccept::ProtoGroupKey::HTTP2, new Http2SessionAccept(accept_opt)); + probe->registerEndpoint(ProtocolProbeSessionAccept::ProtoGroupKey::HTTP2, new Http2SessionAccept(accept_opt, &port)); } } @@ -235,9 +235,9 @@ MakeHttpProxyAcceptor(HttpProxyAcceptor &acceptor, HttpProxyPort &port, unsigned ssl->enableProtocols(port.m_session_protocol_preference); ssl->registerEndpoint(TS_ALPN_PROTOCOL_HTTP_1_0, http); ssl->registerEndpoint(TS_ALPN_PROTOCOL_HTTP_1_1, http); - ssl->registerEndpoint(TS_ALPN_PROTOCOL_HTTP_2_0, new Http2SessionAccept(accept_opt)); + ssl->registerEndpoint(TS_ALPN_PROTOCOL_HTTP_2_0, new Http2SessionAccept(accept_opt, &port)); #if TS_USE_QMUX - ssl->registerEndpoint(TS_ALPN_PROTOCOL_H3QX, new Http3SessionAccept(accept_opt)); + ssl->registerEndpoint(TS_ALPN_PROTOCOL_H3QX, new Http3SessionAccept(accept_opt, &port)); #endif SCOPED_MUTEX_LOCK(lock, ssl_plugin_mutex, this_ethread()); @@ -251,16 +251,16 @@ MakeHttpProxyAcceptor(HttpProxyAcceptor &acceptor, HttpProxyPort &port, unsigned quic->enableProtocols(port.m_session_protocol_preference); // HTTP/0.9 over QUIC draft-29 (for interop only, will be removed) - quic->registerEndpoint(TS_ALPN_PROTOCOL_HTTP_QUIC_D29, new Http3SessionAccept(accept_opt)); + quic->registerEndpoint(TS_ALPN_PROTOCOL_HTTP_QUIC_D29, new Http3SessionAccept(accept_opt, &port)); // HTTP/3 draft-29 - quic->registerEndpoint(TS_ALPN_PROTOCOL_HTTP_3_D29, new Http3SessionAccept(accept_opt)); + quic->registerEndpoint(TS_ALPN_PROTOCOL_HTTP_3_D29, new Http3SessionAccept(accept_opt, &port)); // HTTP/0.9 over QUIC (for interop only, will be removed) - quic->registerEndpoint(TS_ALPN_PROTOCOL_HTTP_QUIC, new Http3SessionAccept(accept_opt)); + quic->registerEndpoint(TS_ALPN_PROTOCOL_HTTP_QUIC, new Http3SessionAccept(accept_opt, &port)); // HTTP/3 - quic->registerEndpoint(TS_ALPN_PROTOCOL_HTTP_3, new Http3SessionAccept(accept_opt)); + quic->registerEndpoint(TS_ALPN_PROTOCOL_HTTP_3, new Http3SessionAccept(accept_opt, &port)); quic->proxyPort = &port; acceptor._accept = quic; diff --git a/src/proxy/http/HttpSM.cc b/src/proxy/http/HttpSM.cc index 9645a19a925..6b9c96c24e5 100644 --- a/src/proxy/http/HttpSM.cc +++ b/src/proxy/http/HttpSM.cc @@ -23,6 +23,7 @@ */ #include "proxy/http/HttpConfig.h" +#include "proxy/hdrs/HdrUtils.h" #include "tscore/ink_hrtime.h" #include "tscore/ink_time.h" #include "tsutil/Metrics.h" @@ -441,10 +442,10 @@ HttpSM::attach_client_session(ProxyTransaction *txn) t_state.api_skip_all_remapping = netvc->get_is_unmanaged_request(); ink_assert(_ua.get_txn()->get_proxy_ssn()); - ink_assert(_ua.get_txn()->get_proxy_ssn()->accept_options); + ink_assert(_ua.get_txn()->get_proxy_ssn()->acceptor); // default the upstream IP style host resolution order from inbound - t_state.my_txn_conf().host_res_data.order = _ua.get_txn()->get_proxy_ssn()->accept_options->host_res_preference; + t_state.my_txn_conf().host_res_data.order = _ua.get_txn()->get_proxy_ssn()->acceptor->options().host_res_preference; start_sub_sm(); @@ -2105,6 +2106,19 @@ HttpSM::state_read_server_response_header(int event, void *data) // If there is a post body in transit, give up on it if (tunnel.is_tunnel_alive()) { tunnel.abort_tunnel(); + // abort_tunnel() cancels I/O but does not close VCs or clean up + // vc_table entries. When a request transform is active the + // post_transform_info entry still references the TransformVConnection + // with in_tunnel=true, which causes cleanup_entry() to skip + // do_io_close() — leaking the VC. Close it explicitly here. + if (post_transform_info.entry != nullptr) { + post_transform_info.entry->vc->do_io_close(); + vc_table.cleanup_entry(post_transform_info.entry); + post_transform_info.entry = nullptr; + // .vc stays non-null so transform_cleanup() skips the already-closed chain. + // tunnel_handler_post_or_put() is unreachable after + // abort_tunnel() since the tunnel is dead and reset. + } // Make sure client connection is closed when we are done in case there is cruft left over t_state.client_info.keep_alive = HTTPKeepAlive::NO_KEEPALIVE; // Similarly the server connection should also be closed @@ -2718,18 +2732,25 @@ HttpSM::state_cache_open_read(int event, void *data) -cache_sm.get_last_error()); SMDbg(dbg_ctl_http, "open read failed."); - // Inform HttpTransact somebody else is updating the document - // HttpCacheSM already waited so transact should go ahead. + // A path that carries its own ";params" segment already hashes to the 9.2 + // key, so a compatibility lookup would just repeat the lookup that missed. + if (cache_sm.get_last_error() != -ECACHE_DOC_BUSY && t_state.http_config_param->cache_try_compat_key_read && + compatibility_cache_lookup == CompatibilityCacheLookup::COMPAT_CACHE_LOOKUP_NORMAL && + !cache_lookup_url()->has_path_params()) { + // do the retry + compatibility_cache_lookup = CompatibilityCacheLookup::COMPAT_CACHE_LOOKUP_92; + do_cache_lookup_and_read(); + return 0; + } + // Nothing was found under either key, or the document is busy, so no legacy + // object is in play past this point. The reset follows the retry decision + // so that it cannot re-arm the retry. + compatibility_cache_lookup = CompatibilityCacheLookup::COMPAT_CACHE_LOOKUP_NORMAL; if (cache_sm.get_last_error() == -ECACHE_DOC_BUSY) { + // Inform HttpTransact somebody else is updating the document + // HttpCacheSM already waited so transact should go ahead. t_state.cache_lookup_result = HttpTransact::CacheLookupResult_t::DOC_BUSY; } else { - if (t_state.http_config_param->cache_try_compat_key_read && - compatibility_cache_lookup == CompatibilityCacheLookup::COMPAT_CACHE_LOOKUP_NORMAL) { - // do the retry - compatibility_cache_lookup = CompatibilityCacheLookup::COMPAT_CACHE_LOOKUP_92; - do_cache_lookup_and_read(); - return 0; - } t_state.cache_lookup_result = HttpTransact::CacheLookupResult_t::MISS; } @@ -5258,6 +5279,28 @@ HttpSM::do_range_setup_if_necessary() } } +// The URL this transaction looks up in the cache, and that every delete which +// has to match that lookup uses too. +// +// hdr_info.client_request's URL always names the *current* request: +// redirect_request() rewrites it in place, so after a redirect follow it is the +// Location target. cache_info.lookup_url is set once, before the first lookup +// (HttpTransact::DecideCacheLookup), and names the *original* request. Whether +// it also moves with a redirect is an accident of how it was set: with +// pristine_host_hdr off it aliases client_request's URL and follows it, with +// pristine_host_hdr on it is a private copy and stays put. It is therefore not +// a reliable way to reach the redirected URL, which is why the redirect case +// below reads client_request instead. +URL * +HttpSM::cache_lookup_url() +{ + // Follow the redirect unless configured to keep the original cache key. + if (t_state.redirect_info.redirect_in_process && !t_state.txn_conf->redirect_use_orig_cache_key) { + return t_state.hdr_info.client_request.url_get(); + } + return t_state.cache_info.lookup_url; +} + void HttpSM::do_cache_lookup_and_read() { @@ -5268,26 +5311,22 @@ HttpSM::do_cache_lookup_and_read() t_state.request_sent_time = UNDEFINED_TIME; t_state.response_received_time = UNDEFINED_TIME; - Metrics::Counter::increment(http_rsb.cache_lookups); - - ATS_PROBE1(milestone_cache_open_read_begin, sm_id); - milestones[TS_MILESTONE_CACHE_OPEN_READ_BEGIN] = ink_get_hrtime(); - t_state.cache_lookup_result = HttpTransact::CacheLookupResult_t::NONE; - t_state.cache_info.lookup_count++; - // YTS Team, yamsat Plugin - // Changed the lookup_url to c_url which enables even - // the new redirect url to perform a CACHE_LOOKUP - URL *c_url; - if (t_state.redirect_info.redirect_in_process && !t_state.txn_conf->redirect_use_orig_cache_key) { - c_url = t_state.hdr_info.client_request.url_get(); - } else { - c_url = t_state.cache_info.lookup_url; + // A compatibility retry continues the lookup that just missed rather than + // starting a new one, so the counter, the probe, and the milestone all stay + // with the canonical-key lookup that began it. + if (!CompatCacheKey::is_legacy(compatibility_cache_lookup)) { + Metrics::Counter::increment(http_rsb.cache_lookups); + ATS_PROBE1(milestone_cache_open_read_begin, sm_id); + milestones[TS_MILESTONE_CACHE_OPEN_READ_BEGIN] = ink_get_hrtime(); + t_state.cache_info.lookup_count++; } + t_state.cache_lookup_result = HttpTransact::CacheLookupResult_t::NONE; + URL *c_url = cache_lookup_url(); SMDbg(dbg_ctl_http_seq, "Issuing cache lookup for URL %s", c_url->string_get(&t_state.arena)); HttpCacheKey key; - if (compatibility_cache_lookup == CompatibilityCacheLookup::COMPAT_CACHE_LOOKUP_92) { + if (CompatCacheKey::is_legacy(compatibility_cache_lookup)) { Cache::generate_key92(&key, c_url, t_state.txn_conf->cache_ignore_query, t_state.txn_conf->cache_generation_number); } else { Cache::generate_key(&key, c_url, t_state.txn_conf->cache_ignore_query, t_state.txn_conf->cache_generation_number); @@ -5315,11 +5354,47 @@ HttpSM::do_cache_delete_all_alts() // Do not delete a non-existent object. ink_assert(t_state.cache_info.object_read); - SMDbg(dbg_ctl_http_seq, "Issuing cache delete for %s", t_state.cache_info.lookup_url->string_get_ref()); + // Address the object that was looked up. A redirect follow can look up a + // different URL than cache_info.lookup_url, which is set once and does not + // track the redirect when the pristine host header is maintained. + URL *url = cache_lookup_url(); + + SMDbg(dbg_ctl_http_seq, "Issuing cache delete for %s", url->string_get_ref()); + + HttpCacheKey key; + Cache::generate_key(&key, url, t_state.txn_conf->cache_ignore_query, t_state.txn_conf->cache_generation_number); + cacheProcessor.remove(nullptr, &key); + + // A migration leaves the legacy copy in place, so the object can live under + // both keys. Removing only one of them would let the other be served after + // the purge. + do_cache_delete_compat_alts(); +} + +// Remove the object stored under the legacy key. +// +// Only for the delete paths, which have to reach both keys. A successful +// migration deliberately leaves the legacy copy alone: VC_EVENT_WRITE_COMPLETE +// means the tunnel handed the last byte to the cache VC, not that the object +// reached disk, so deleting on that signal loses the object outright whenever +// the write later fails. The copy ages out on its own, and it stops being read +// as soon as the canonical key resolves, so compat_key_reads still decays to +// zero. +void +HttpSM::do_cache_delete_compat_alts() +{ + // Same URL the lookup used; see do_cache_delete_all_alts(). + URL *url = cache_lookup_url(); + + // A path that carries its own ";params" segment hashes to the same key + // under both schemes, so the canonical delete already reached it. + if (!t_state.http_config_param->cache_try_compat_key_read || url->has_path_params()) { + return; + } + SMDbg(dbg_ctl_http_seq, "Issuing compatibility cache delete for %s", url->string_get_ref()); HttpCacheKey key; - Cache::generate_key(&key, t_state.cache_info.lookup_url, t_state.txn_conf->cache_ignore_query, - t_state.txn_conf->cache_generation_number); + Cache::generate_key92(&key, url, t_state.txn_conf->cache_ignore_query, t_state.txn_conf->cache_generation_number); cacheProcessor.remove(nullptr, &key); } @@ -5348,6 +5423,15 @@ HttpSM::do_cache_prepare_update() t_state.cache_info.object_store.valid() && t_state.cache_info.object_store.response_get() != nullptr && t_state.cache_info.object_store.response_get()->valid() && t_state.hdr_info.client_request.method_get_wksidx() == HTTP_WKSIDX_GET) { + // An object found under the 9.2 key cannot be updated in place: the write + // would be a create on the current key, and the cache turns a header-only + // close of a create into an abort. Take the failed-update path instead, + // which serves the updated headers without storing them. + if (CompatCacheKey::is_legacy(compatibility_cache_lookup)) { + ink_assert(t_state.cache_info.write_lock_state != HttpTransact::CacheWriteLock_t::SUCCESS); + call_transact_and_set_next_state(HttpTransact::HandleUpdateCachedObject); + return; + } t_state.cache_info.object_store.request_set(t_state.cache_info.object_read->request_get()); // t_state.cache_info.object_read = NULL; // cache_sm.close_read(); @@ -5398,8 +5482,15 @@ HttpSM::do_cache_prepare_action(HttpCacheSM *c_sm, CacheHTTPInfo *object_read_in HttpCacheKey key; Cache::generate_key(&key, s_url, t_state.txn_conf->cache_ignore_query, t_state.txn_conf->cache_generation_number); + // A compatibility read returns an object stored under the legacy key. Passing + // that object to a write using the canonical key turns the write into an + // update, but the canonical-key vector does not contain the legacy alternate. + // Cache::open_write then fails with ECACHE_NO_DOC instead of creating the + // migrated object. Create a new canonical-key object for compatibility reads. + CacheHTTPInfo *write_object_read_info = CompatCacheKey::write_info(compatibility_cache_lookup, object_read_info); + pending_action = - c_sm->open_write(&key, s_url, &t_state.hdr_info.cache_request, object_read_info, + c_sm->open_write(&key, s_url, &t_state.hdr_info.cache_request, write_object_read_info, static_cast((t_state.cache_control.pin_in_cache_for < 0) ? 0 : t_state.cache_control.pin_in_cache_for), retry, allow_multiple); } @@ -6797,6 +6888,9 @@ HttpSM::perform_cache_write_action() case HttpTransact::CacheAction_t::DELETE: { // Write close deletes the old alternate cache_sm.close_write(); + // That reached only one of the two keys the object can live under while + // the compatibility lookup is enabled. + do_cache_delete_compat_alts(); cache_sm.close_read(); t_state.cache_info.write_lock_state = HttpTransact::CacheWriteLock_t::INIT; break; @@ -8373,6 +8467,10 @@ HttpSM::set_next_state() case HttpTransact::StateMachineAction_t::CACHE_LOOKUP: { HTTP_SM_SET_DEFAULT_HANDLER(&HttpSM::state_cache_open_read); + // Every lookup starts from the canonical key. A redirect follow or a read + // retry looks up a different object than the one a previous compatibility + // lookup found, so the flag must not carry over. + compatibility_cache_lookup = CompatibilityCacheLookup::COMPAT_CACHE_LOOKUP_NORMAL; do_cache_lookup_and_read(); break; } diff --git a/src/proxy/http/HttpSessionAccept.cc b/src/proxy/http/HttpSessionAccept.cc index 4f3752c3b63..5424e72b1b4 100644 --- a/src/proxy/http/HttpSessionAccept.cc +++ b/src/proxy/http/HttpSessionAccept.cc @@ -70,7 +70,7 @@ HttpSessionAccept::accept(NetVConnection *netvc, MIOBuffer *iobuf, IOBufferReade // Set the transport type if not already set if (HttpProxyPort::TRANSPORT_NONE == netvc->attributes) { - netvc->attributes = transport_type; + netvc->attributes = options().transport_type; } if (dbg_ctl_http_seq.on()) { @@ -80,8 +80,8 @@ HttpSessionAccept::accept(NetVConnection *netvc, MIOBuffer *iobuf, IOBufferReade Http1ClientSession *new_session = THREAD_ALLOC_INIT(http1ClientSessionAllocator, this_ethread()); - new_session->accept_options = static_cast(this); - new_session->acl = std::move(acl); + new_session->acceptor = this; + new_session->acl = std::move(acl); // Pin session to current ET_NET thread new_session->setThreadAffinity(this_ethread()); diff --git a/src/proxy/http/HttpTransact.cc b/src/proxy/http/HttpTransact.cc index 2908a1446d0..42be71ae454 100644 --- a/src/proxy/http/HttpTransact.cc +++ b/src/proxy/http/HttpTransact.cc @@ -2581,6 +2581,23 @@ HttpTransact::issue_revalidate(State *s) return; } + // An object found under the 9.2 key cannot take a 304: the write that would + // apply it is a create on the current key, not an update of the legacy + // vector, and the cache aborts a header-only create. Ask for the full + // response instead, without our validators or the client's, so the 200 + // replaces it under the current key and the legacy copy ages out. The + // client's conditionals are still matched against that 200. HEAD and Range + // requests are left alone: neither response is stored, and an unconditional + // range request would draw a 206 that deletes the object rather than + // migrating it. The next full GET migrates it instead. + if (s->cache_info.action == CacheAction_t::PREPARE_TO_UPDATE && s->method == HTTP_WKSIDX_GET && + !s->hdr_info.client_request.presence(MIME_PRESENCE_RANGE) && s->state_machine != nullptr && + CompatCacheKey::is_legacy(s->state_machine->compatibility_cache_lookup)) { + TxnDbg(dbg_ctl_http_trans, "object under the compatibility key, revalidating unconditionally"); + HttpTransactHeaders::remove_conditional_headers(&s->hdr_info.server_request); + return; + } + // if the document is cached, just send a conditional request to the server // So the request does not have preconditions. It can, however diff --git a/src/proxy/http/unit_tests/test_HttpUserAgent.cc b/src/proxy/http/unit_tests/test_HttpUserAgent.cc index 74de69d8ffe..6f815d699e5 100644 --- a/src/proxy/http/unit_tests/test_HttpUserAgent.cc +++ b/src/proxy/http/unit_tests/test_HttpUserAgent.cc @@ -72,7 +72,8 @@ TEST_CASE("tcp_reused should be set correctly when a session is attached.") SSLNetVConnection netvc; ssn.set_vc(&netvc); HttpSessionAccept::Options options; - ssn.accept_options = &options; + HttpSessionAccept acceptor{options}; + ssn.acceptor = &acceptor; Http1ClientTransaction txn{&ssn}; SECTION("When a transaction is the first one, " @@ -91,3 +92,26 @@ TEST_CASE("tcp_reused should be set correctly when a session is attached.") CHECK(user_agent.get_client_tcp_reused() == true); } } + +TEST_CASE("Transactions inherit shared acceptor properties") +{ + HttpSessionAccept::Options options; + HttpProxyPort proxy_port; + + options.setOutboundPort(8080); + options.setOutboundTransparent(true); + options.setTransparentPassthrough(true); + + auto options_handle = std::make_shared(options); + HttpSessionAccept acceptor{options_handle, &proxy_port}; + Http1ClientTestSession ssn; + + ssn.acceptor = &acceptor; + Http1ClientTransaction txn{&ssn}; + + CHECK(&acceptor.options() == options_handle.get()); + CHECK(ssn.acceptor->proxyPort == &proxy_port); + CHECK(txn.get_outbound_port() == 8080); + CHECK(txn.is_outbound_transparent()); + CHECK(txn.is_transparent_passthrough_allowed()); +} diff --git a/src/proxy/http2/Http2SessionAccept.cc b/src/proxy/http2/Http2SessionAccept.cc index 0b52df11d2a..16e073228f5 100644 --- a/src/proxy/http2/Http2SessionAccept.cc +++ b/src/proxy/http2/Http2SessionAccept.cc @@ -32,7 +32,8 @@ DbgCtl dbg_ctl_http2_seq{"http2_seq"}; } // end anonymous namespace -Http2SessionAccept::Http2SessionAccept(const HttpSessionAccept::Options &_o) : SessionAccept(nullptr), options(_o) +Http2SessionAccept::Http2SessionAccept(OptionsHandle options, HttpProxyPort *proxy_port) + : HttpSessionAcceptBase(std::move(options), proxy_port) { SET_HANDLER(&Http2SessionAccept::mainEvent); } @@ -67,7 +68,7 @@ Http2SessionAccept::accept(NetVConnection *netvc, MIOBuffer *iobuf, IOBufferRead return false; } - netvc->attributes = this->options.transport_type; + netvc->attributes = this->options().transport_type; if (dbg_ctl_http2_seq.on()) { ip_port_text_buffer ipb; @@ -78,7 +79,7 @@ Http2SessionAccept::accept(NetVConnection *netvc, MIOBuffer *iobuf, IOBufferRead Http2ClientSession *new_session = THREAD_ALLOC_INIT(http2ClientSessionAllocator, this_ethread()); new_session->acl = std::move(session_acl); - new_session->accept_options = &options; + new_session->acceptor = this; // Pin session to current ET_NET thread new_session->setThreadAffinity(this_ethread()); diff --git a/src/proxy/http2/Http2Stream.cc b/src/proxy/http2/Http2Stream.cc index 611e8b6a4eb..b458b0e4caf 100644 --- a/src/proxy/http2/Http2Stream.cc +++ b/src/proxy/http2/Http2Stream.cc @@ -75,7 +75,6 @@ Http2Stream::Http2Stream(ProxySession *session, Http2StreamId sid, ssize_t initi _receive_header.create(HTTPType::RESPONSE); _send_header.create(HTTPType::REQUEST, HTTP_2_0); } else { - this->upstream_outbound_options = *(session->accept_options); _receive_header.create(HTTPType::REQUEST); _send_header.create(HTTPType::RESPONSE, HTTP_2_0); } diff --git a/src/proxy/http3/Http09App.cc b/src/proxy/http3/Http09App.cc index ac9f1ab2ec1..866433011ef 100644 --- a/src/proxy/http3/Http09App.cc +++ b/src/proxy/http3/Http09App.cc @@ -45,12 +45,12 @@ DbgCtl dbg_ctl_v{debug_tag_v}; } // end anonymous namespace Http09App::Http09App(NetVConnection *client_vc, QUICConnection *qc, IpAllow::ACL &&session_acl, - const HttpSessionAccept::Options &options) + HttpSessionAcceptBase const *acceptor) : QUICApplication(qc) { - this->_ssn = new Http09Session(client_vc); - this->_ssn->acl = std::move(session_acl); - this->_ssn->accept_options = &options; + this->_ssn = new Http09Session(client_vc); + this->_ssn->acl = std::move(session_acl); + this->_ssn->acceptor = acceptor; this->_ssn->new_connection(client_vc, nullptr, nullptr); this->_qc->stream_manager()->set_default_application(this); diff --git a/src/proxy/http3/Http3App.cc b/src/proxy/http3/Http3App.cc index cca7960303f..941598c5446 100644 --- a/src/proxy/http3/Http3App.cc +++ b/src/proxy/http3/Http3App.cc @@ -51,13 +51,12 @@ DbgCtl dbg_ctl_v{debug_tag_v}; } // end anonymous namespace -Http3App::Http3App(NetVConnection *client_vc, QUICConnection *qc, IpAllow::ACL &&session_acl, - const HttpSessionAccept::Options &options) +Http3App::Http3App(NetVConnection *client_vc, QUICConnection *qc, IpAllow::ACL &&session_acl, HttpSessionAcceptBase const *acceptor) : QUICApplication(qc) { - this->_ssn = new Http3Session(client_vc); - this->_ssn->acl = std::move(session_acl); - this->_ssn->accept_options = &options; + this->_ssn = new Http3Session(client_vc); + this->_ssn->acl = std::move(session_acl); + this->_ssn->acceptor = acceptor; this->_ssn->new_connection(client_vc, nullptr, nullptr); this->_qc->stream_manager()->set_default_application(this); diff --git a/src/proxy/http3/Http3SessionAccept.cc b/src/proxy/http3/Http3SessionAccept.cc index 8a7e9da0360..dd10b869c97 100644 --- a/src/proxy/http3/Http3SessionAccept.cc +++ b/src/proxy/http3/Http3SessionAccept.cc @@ -42,7 +42,8 @@ DbgCtl dbg_ctl_http3{"http3"}; } // end anonymous namespace -Http3SessionAccept::Http3SessionAccept(const HttpSessionAccept::Options &_o) : SessionAccept(nullptr), options(_o) +Http3SessionAccept::Http3SessionAccept(OptionsHandle options, HttpProxyPort *proxy_port) + : HttpSessionAcceptBase(std::move(options), proxy_port) { SET_HANDLER(&Http3SessionAccept::mainEvent); } @@ -66,7 +67,7 @@ Http3SessionAccept::accept(NetVConnection *netvc, MIOBuffer * /* iobuf ATS_UNUSE return false; } - netvc->attributes = this->options.transport_type; + netvc->attributes = this->options().transport_type; QUICConnection *qc = netvc->get_service()->get_quic_connection(); @@ -80,12 +81,12 @@ Http3SessionAccept::accept(NetVConnection *netvc, MIOBuffer * /* iobuf ATS_UNUSE if (IP_PROTO_TAG_HTTP_QUIC.compare(alpn) == 0 || IP_PROTO_TAG_HTTP_QUIC_D29.compare(alpn) == 0) { Dbg(dbg_ctl_http3, "[%s] start HTTP/0.9 app (ALPN=%.*s)", qc->cids().data(), static_cast(alpn.length()), alpn.data()); - new Http09App(netvc, qc, std::move(session_acl), this->options); + new Http09App(netvc, qc, std::move(session_acl), this); } else if (IP_PROTO_TAG_HTTP_3.compare(alpn) == 0 || IP_PROTO_TAG_HTTP_3_D29.compare(alpn) == 0 || IP_PROTO_TAG_H3QX.compare(alpn) == 0) { Dbg(dbg_ctl_http3, "[%s] start HTTP/3 app (ALPN=%.*s)", qc->cids().data(), static_cast(alpn.length()), alpn.data()); - Http3App *app = new Http3App(netvc, qc, std::move(session_acl), this->options); + Http3App *app = new Http3App(netvc, qc, std::move(session_acl), this); #if TS_USE_QMUX if (IP_PROTO_TAG_H3QX.compare(alpn) == 0) { diff --git a/src/proxy/logging/Log.cc b/src/proxy/logging/Log.cc index 6e72ac408b2..6c8e33bbb10 100644 --- a/src/proxy/logging/Log.cc +++ b/src/proxy/logging/Log.cc @@ -304,6 +304,21 @@ struct LoggingFlushContinuation : public Continuation { } }; +void +Log::register_field(LogField *field) +{ + ink_assert(field != nullptr); + + auto [it, inserted] = field_symbol_hash.emplace(field->symbol(), field); + if (!inserted) { + Error("Log field symbol '%s' is already registered", field->symbol()); + delete field; + return; + } + + global_field_list.add(field); +} + /*------------------------------------------------------------------------- Log::init_fields @@ -312,8 +327,8 @@ struct LoggingFlushContinuation : public Continuation { standalone programs that do not require more services (e.g., that do not need to read records.yaml) can just call init_fields. - Note that the LogFields are added to the list with the copy flag false so - that the LogFieldList destructor will reclaim this memory. + The LogFields are added to the global list without copying so that the + LogFieldList destructor will reclaim this memory. -------------------------------------------------------------------------*/ void Log::init_fields() @@ -322,393 +337,247 @@ Log::init_fields() return; } - LogField *field; - // // Initializes material to find a milestone name from their // name in a rapid manner. LogField::init_milestone_container(); // client -> proxy fields - field = - new LogField("client_host_ip", "chi", LogField::Type::IP, &LogAccess::marshal_client_host_ip, &LogAccess::unmarshal_ip_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("chi", field); - - field = new LogField("client_host_port", "chp", LogField::Type::sINT, &LogAccess::marshal_client_host_port, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("chp", field); - - field = new LogField("client_host_ip_hex", "chih", LogField::Type::IP, &LogAccess::marshal_client_host_ip, - &LogAccess::unmarshal_ip_to_hex); - global_field_list.add(field, false); - field_symbol_hash.emplace("chih", field); - - field = new LogField("client_host_ip_verified", "chiv", LogField::Type::IP, &LogAccess::marshal_client_host_ip_verified, - &LogAccess::unmarshal_ip_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("chiv", field); + register_field( + new LogField("client_host_ip", "chi", LogField::Type::IP, &LogAccess::marshal_client_host_ip, &LogAccess::unmarshal_ip_to_str)); + + register_field(new LogField("client_host_port", "chp", LogField::Type::sINT, &LogAccess::marshal_client_host_port, + &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("client_host_ip_hex", "chih", LogField::Type::IP, &LogAccess::marshal_client_host_ip, + &LogAccess::unmarshal_ip_to_hex)); + + register_field(new LogField("client_host_ip_verified", "chiv", LogField::Type::IP, &LogAccess::marshal_client_host_ip_verified, + &LogAccess::unmarshal_ip_to_str)); // remote client (Not necessarily the requesting client IP - See proxy protocol) - field = - new LogField("remote_host_ip", "rchi", LogField::Type::IP, &LogAccess::marshal_remote_host_ip, &LogAccess::unmarshal_ip_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("rchi", field); + register_field(new LogField("remote_host_ip", "rchi", LogField::Type::IP, &LogAccess::marshal_remote_host_ip, + &LogAccess::unmarshal_ip_to_str)); - field = new LogField("remote_host_port", "rchp", LogField::Type::sINT, &LogAccess::marshal_remote_host_port, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("rchp", field); + register_field(new LogField("remote_host_port", "rchp", LogField::Type::sINT, &LogAccess::marshal_remote_host_port, + &LogAccess::unmarshal_int_to_str)); - field = new LogField("remote_host_ip_hex", "rchh", LogField::Type::IP, &LogAccess::marshal_remote_host_ip, - &LogAccess::unmarshal_ip_to_hex); - global_field_list.add(field, false); - field_symbol_hash.emplace("rchh", field); + register_field(new LogField("remote_host_ip_hex", "rchh", LogField::Type::IP, &LogAccess::marshal_remote_host_ip, + &LogAccess::unmarshal_ip_to_hex)); // interface ip - field = new LogField("host_interface_ip", "hii", LogField::Type::IP, &LogAccess::marshal_host_interface_ip, - &LogAccess::unmarshal_ip_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("hii", field); + register_field(new LogField("host_interface_ip", "hii", LogField::Type::IP, &LogAccess::marshal_host_interface_ip, + &LogAccess::unmarshal_ip_to_str)); - field = new LogField("host_interface_ip_hex", "hiih", LogField::Type::IP, &LogAccess::marshal_host_interface_ip, - &LogAccess::unmarshal_ip_to_hex); - global_field_list.add(field, false); - field_symbol_hash.emplace("hiih", field); + register_field(new LogField("host_interface_ip_hex", "hiih", LogField::Type::IP, &LogAccess::marshal_host_interface_ip, + &LogAccess::unmarshal_ip_to_hex)); // interface ip end - field = new LogField("client_auth_user_name", "caun", LogField::Type::STRING, &LogAccess::marshal_client_auth_user_name, - &LogAccess::unmarshal_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("caun", field); - - field = new LogField("plugin_identity_id", "piid", LogField::Type::sINT, &LogAccess::marshal_plugin_identity_id, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("piid", field); - - field = new LogField("plugin_identity_tag", "pitag", LogField::Type::STRING, &LogAccess::marshal_plugin_identity_tag, - &LogAccess::unmarshal_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("pitag", field); - - field = new LogField("client_req_timestamp_sec", "cqts", LogField::Type::sINT, &LogAccess::marshal_client_req_timestamp_sec, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("cqts", field); - - field = new LogField("client_req_timestamp_hex_sec", "cqth", LogField::Type::sINT, &LogAccess::marshal_client_req_timestamp_sec, - &LogAccess::unmarshal_int_to_str_hex); - global_field_list.add(field, false); - field_symbol_hash.emplace("cqth", field); - - field = new LogField("client_req_timestamp_squid", "cqtq", LogField::Type::sINT, &LogAccess::marshal_client_req_timestamp_ms, - &LogAccess::unmarshal_ttmsf); - global_field_list.add(field, false); - field_symbol_hash.emplace("cqtq", field); - - field = new LogField("client_req_timestamp_netscape", "cqtn", LogField::Type::sINT, &LogAccess::marshal_client_req_timestamp_sec, - &LogAccess::unmarshal_int_to_netscape_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("cqtn", field); - - field = new LogField("client_req_timestamp_date", "cqtd", LogField::Type::sINT, &LogAccess::marshal_client_req_timestamp_sec, - &LogAccess::unmarshal_int_to_date_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("cqtd", field); - - field = new LogField("client_req_timestamp_time", "cqtt", LogField::Type::sINT, &LogAccess::marshal_client_req_timestamp_sec, - &LogAccess::unmarshal_int_to_time_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("cqtt", field); - - field = new LogField("client_req_http_method", "cqhm", LogField::Type::STRING, &LogAccess::marshal_client_req_http_method, - &LogAccess::unmarshal_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("cqhm", field); - - field = new LogField("client_req_url", "cqu", LogField::Type::STRING, &LogAccess::marshal_client_req_url, - &LogAccess::unmarshal_str, &LogAccess::set_client_req_url); - global_field_list.add(field, false); - field_symbol_hash.emplace("cqu", field); - - field = new LogField("client_req_url", "pqu", LogField::Type::STRING, &LogAccess::marshal_client_req_url, - &LogAccess::unmarshal_str, &LogAccess::set_client_req_url); - global_field_list.add(field, false); - field_symbol_hash.emplace("pqu", field); - - field = new LogField("client_req_url_canonical", "cquc", LogField::Type::STRING, &LogAccess::marshal_client_req_url_canon, - &LogAccess::unmarshal_str, &LogAccess::set_client_req_url_canon); - global_field_list.add(field, false); - field_symbol_hash.emplace("cquc", field); - - field = new LogField("client_req_url_canonical", "pquc", LogField::Type::STRING, &LogAccess::marshal_client_req_url_canon, - &LogAccess::unmarshal_str, &LogAccess::set_client_req_url_canon); - global_field_list.add(field, false); - field_symbol_hash.emplace("pquc", field); - - field = new LogField("client_req_unmapped_url_canonical", "cquuc", LogField::Type::STRING, - &LogAccess::marshal_client_req_unmapped_url_canon, &LogAccess::unmarshal_str, - &LogAccess::set_client_req_unmapped_url_canon); - global_field_list.add(field, false); - field_symbol_hash.emplace("cquuc", field); - - field = - new LogField("client_req_unmapped_url_path", "cquup", LogField::Type::STRING, &LogAccess::marshal_client_req_unmapped_url_path, - &LogAccess::unmarshal_str, &LogAccess::set_client_req_unmapped_url_path); - global_field_list.add(field, false); - field_symbol_hash.emplace("cquup", field); - - field = - new LogField("client_req_unmapped_url_host", "cquuh", LogField::Type::STRING, &LogAccess::marshal_client_req_unmapped_url_host, - &LogAccess::unmarshal_str, &LogAccess::set_client_req_unmapped_url_host); - global_field_list.add(field, false); - field_symbol_hash.emplace("cquuh", field); - - field = new LogField("client_req_url_scheme", "cqus", LogField::Type::STRING, &LogAccess::marshal_client_req_url_scheme, - &LogAccess::unmarshal_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("cqus", field); - - field = new LogField("client_req_url_scheme", "pqus", LogField::Type::STRING, &LogAccess::marshal_client_req_url_scheme, - &LogAccess::unmarshal_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("pqus", field); - - field = new LogField("client_req_url_path", "cqup", LogField::Type::STRING, &LogAccess::marshal_client_req_url_path, - &LogAccess::unmarshal_str, &LogAccess::set_client_req_url_path); - global_field_list.add(field, false); - field_symbol_hash.emplace("cqup", field); - - field = new LogField("client_req_url_path", "pqup", LogField::Type::STRING, &LogAccess::marshal_client_req_url_path, - &LogAccess::unmarshal_str, &LogAccess::set_client_req_url_path); - global_field_list.add(field, false); - field_symbol_hash.emplace("pqup", field); - - field = new LogField("client_req_protocol_version", "cqpv", LogField::Type::STRING, - &LogAccess::marshal_client_req_protocol_version, &LogAccess::unmarshal_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("cqpv", field); - - field = new LogField("server_req_protocol_version", "sqpv", LogField::Type::STRING, - &LogAccess::marshal_server_req_protocol_version, &LogAccess::unmarshal_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("sqpv", field); - - field = new LogField("client_req_header_len", "cqhl", LogField::Type::sINT, &LogAccess::marshal_client_req_header_len, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("cqhl", field); - - field = new LogField("client_req_squid_len", "cqql", LogField::Type::sINT, &LogAccess::marshal_client_req_squid_len, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("cqql", field); + register_field(new LogField("client_auth_user_name", "caun", LogField::Type::STRING, &LogAccess::marshal_client_auth_user_name, + &LogAccess::unmarshal_str)); + + register_field(new LogField("plugin_identity_id", "piid", LogField::Type::sINT, &LogAccess::marshal_plugin_identity_id, + &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("plugin_identity_tag", "pitag", LogField::Type::STRING, &LogAccess::marshal_plugin_identity_tag, + &LogAccess::unmarshal_str)); + + register_field(new LogField("client_req_timestamp_sec", "cqts", LogField::Type::sINT, + &LogAccess::marshal_client_req_timestamp_sec, &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("client_req_timestamp_hex_sec", "cqth", LogField::Type::sINT, + &LogAccess::marshal_client_req_timestamp_sec, &LogAccess::unmarshal_int_to_str_hex)); + + register_field(new LogField("client_req_timestamp_squid", "cqtq", LogField::Type::sINT, + &LogAccess::marshal_client_req_timestamp_ms, &LogAccess::unmarshal_ttmsf)); + + register_field(new LogField("client_req_timestamp_netscape", "cqtn", LogField::Type::sINT, + &LogAccess::marshal_client_req_timestamp_sec, &LogAccess::unmarshal_int_to_netscape_str)); + + register_field(new LogField("client_req_timestamp_date", "cqtd", LogField::Type::sINT, + &LogAccess::marshal_client_req_timestamp_sec, &LogAccess::unmarshal_int_to_date_str)); + + register_field(new LogField("client_req_timestamp_time", "cqtt", LogField::Type::sINT, + &LogAccess::marshal_client_req_timestamp_sec, &LogAccess::unmarshal_int_to_time_str)); + + register_field(new LogField("client_req_http_method", "cqhm", LogField::Type::STRING, &LogAccess::marshal_client_req_http_method, + &LogAccess::unmarshal_str)); + + register_field(new LogField("client_req_url", "cqu", LogField::Type::STRING, &LogAccess::marshal_client_req_url, + &LogAccess::unmarshal_str, &LogAccess::set_client_req_url)); + + register_field(new LogField("client_req_url", "pqu", LogField::Type::STRING, &LogAccess::marshal_client_req_url, + &LogAccess::unmarshal_str, &LogAccess::set_client_req_url)); + + register_field(new LogField("client_req_url_canonical", "cquc", LogField::Type::STRING, &LogAccess::marshal_client_req_url_canon, + &LogAccess::unmarshal_str, &LogAccess::set_client_req_url_canon)); + + register_field(new LogField("client_req_url_canonical", "pquc", LogField::Type::STRING, &LogAccess::marshal_client_req_url_canon, + &LogAccess::unmarshal_str, &LogAccess::set_client_req_url_canon)); + + register_field(new LogField("client_req_unmapped_url_canonical", "cquuc", LogField::Type::STRING, + &LogAccess::marshal_client_req_unmapped_url_canon, &LogAccess::unmarshal_str, + &LogAccess::set_client_req_unmapped_url_canon)); + + register_field(new LogField("client_req_unmapped_url_path", "cquup", LogField::Type::STRING, + &LogAccess::marshal_client_req_unmapped_url_path, &LogAccess::unmarshal_str, + &LogAccess::set_client_req_unmapped_url_path)); + + register_field(new LogField("client_req_unmapped_url_host", "cquuh", LogField::Type::STRING, + &LogAccess::marshal_client_req_unmapped_url_host, &LogAccess::unmarshal_str, + &LogAccess::set_client_req_unmapped_url_host)); + + register_field(new LogField("client_req_url_scheme", "cqus", LogField::Type::STRING, &LogAccess::marshal_client_req_url_scheme, + &LogAccess::unmarshal_str)); + + register_field(new LogField("client_req_url_scheme", "pqus", LogField::Type::STRING, &LogAccess::marshal_client_req_url_scheme, + &LogAccess::unmarshal_str)); + + register_field(new LogField("client_req_url_path", "cqup", LogField::Type::STRING, &LogAccess::marshal_client_req_url_path, + &LogAccess::unmarshal_str, &LogAccess::set_client_req_url_path)); + + register_field(new LogField("client_req_url_path", "pqup", LogField::Type::STRING, &LogAccess::marshal_client_req_url_path, + &LogAccess::unmarshal_str, &LogAccess::set_client_req_url_path)); + + register_field(new LogField("client_req_protocol_version", "cqpv", LogField::Type::STRING, + &LogAccess::marshal_client_req_protocol_version, &LogAccess::unmarshal_str)); + + register_field(new LogField("server_req_protocol_version", "sqpv", LogField::Type::STRING, + &LogAccess::marshal_server_req_protocol_version, &LogAccess::unmarshal_str)); + + register_field(new LogField("client_req_header_len", "cqhl", LogField::Type::sINT, &LogAccess::marshal_client_req_header_len, + &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("client_req_squid_len", "cqql", LogField::Type::sINT, &LogAccess::marshal_client_req_squid_len, + &LogAccess::unmarshal_int_to_str)); // Client request squid length plus TLS handshake bytes received for TLS connections - field = new LogField("client_req_squid_len_tls", "cqqtl", LogField::Type::sINT, &LogAccess::marshal_client_req_squid_len_tls, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("cqqtl", field); - - field = new LogField("cache_lookup_url_canonical", "cluc", LogField::Type::STRING, &LogAccess::marshal_cache_lookup_url_canon, - &LogAccess::unmarshal_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("cluc", field); - - field = - new LogField("cache_key_hash", "ckh", LogField::Type::STRING, &LogAccess::marshal_cache_key_hash, &LogAccess::unmarshal_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("ckh", field); - - field = new LogField("client_sni_server_name", "cssn", LogField::Type::STRING, &LogAccess::marshal_client_sni_server_name, - &LogAccess::unmarshal_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("cssn", field); - - field = new LogField("client_ssl_cert_provided", "cscert", LogField::Type::sINT, &LogAccess::marshal_client_provided_cert, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("cscert", field); - - field = new LogField("proxy_ssl_cert_provided", "pscert", LogField::Type::sINT, &LogAccess::marshal_proxy_provided_cert, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("pscert", field); - - field = - new LogField("process_uuid", "puuid", LogField::Type::STRING, &LogAccess::marshal_process_uuid, &LogAccess::unmarshal_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("puuid", field); - - field = new LogField("process_snowflake_id", "psfid", LogField::Type::STRING, &LogAccess::marshal_process_sfid, - &LogAccess::unmarshal_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("psfid", field); - - field = new LogField("client_req_content_len", "cqcl", LogField::Type::sINT, &LogAccess::marshal_client_req_content_len, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("cqcl", field); - - field = new LogField("client_req_tcp_reused", "cqtr", LogField::Type::sINT, &LogAccess::marshal_client_req_tcp_reused, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("cqtr", field); - - field = new LogField("client_req_is_ssl", "cqssl", LogField::Type::sINT, &LogAccess::marshal_client_req_is_ssl, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("cqssl", field); - - field = new LogField("client_req_ssl_reused", "cqssr", LogField::Type::sINT, &LogAccess::marshal_client_req_ssl_reused, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("cqssr", field); - - field = new LogField("client_req_ssl_resumption_type", "cqssrt", LogField::Type::sINT, - &LogAccess::marshal_client_ssl_resumption_type, &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("cqssrt", field); - - field = new LogField("client_req_is_internal", "cqint", LogField::Type::sINT, &LogAccess::marshal_client_req_is_internal, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("cqint", field); - - field = new LogField("client_req_mptcp", "cqmpt", LogField::Type::sINT, &LogAccess::marshal_client_req_mptcp_state, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("cqmpt", field); - - field = new LogField("client_sec_protocol", "cqssv", LogField::Type::STRING, &LogAccess::marshal_client_security_protocol, - &LogAccess::unmarshal_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("cqssv", field); - - field = new LogField("client_cipher_suite", "cqssc", LogField::Type::STRING, &LogAccess::marshal_client_security_cipher_suite, - &LogAccess::unmarshal_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("cqssc", field); - - field = new LogField("client_curve", "cqssu", LogField::Type::STRING, &LogAccess::marshal_client_security_curve, - &LogAccess::unmarshal_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("cqssu", field); - - field = new LogField("client_group", "cqssg", LogField::Type::STRING, &LogAccess::marshal_client_security_group, - &LogAccess::unmarshal_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("cqssg", field); - - field = new LogField("client_sec_alpn", "cqssa", LogField::Type::STRING, &LogAccess::marshal_client_security_alpn, - &LogAccess::unmarshal_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("cqssa", field); + register_field(new LogField("client_req_squid_len_tls", "cqqtl", LogField::Type::sINT, + &LogAccess::marshal_client_req_squid_len_tls, &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("cache_lookup_url_canonical", "cluc", LogField::Type::STRING, + &LogAccess::marshal_cache_lookup_url_canon, &LogAccess::unmarshal_str)); + + register_field( + new LogField("cache_key_hash", "ckh", LogField::Type::STRING, &LogAccess::marshal_cache_key_hash, &LogAccess::unmarshal_str)); + + register_field(new LogField("client_sni_server_name", "cssn", LogField::Type::STRING, &LogAccess::marshal_client_sni_server_name, + &LogAccess::unmarshal_str)); + + register_field(new LogField("client_ssl_cert_provided", "cscert", LogField::Type::sINT, &LogAccess::marshal_client_provided_cert, + &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("proxy_ssl_cert_provided", "pscert", LogField::Type::sINT, &LogAccess::marshal_proxy_provided_cert, + &LogAccess::unmarshal_int_to_str)); + + register_field( + new LogField("process_uuid", "puuid", LogField::Type::STRING, &LogAccess::marshal_process_uuid, &LogAccess::unmarshal_str)); + + register_field(new LogField("process_snowflake_id", "psfid", LogField::Type::STRING, &LogAccess::marshal_process_sfid, + &LogAccess::unmarshal_str)); + + register_field(new LogField("client_req_content_len", "cqcl", LogField::Type::sINT, &LogAccess::marshal_client_req_content_len, + &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("client_req_tcp_reused", "cqtr", LogField::Type::sINT, &LogAccess::marshal_client_req_tcp_reused, + &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("client_req_is_ssl", "cqssl", LogField::Type::sINT, &LogAccess::marshal_client_req_is_ssl, + &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("client_req_ssl_reused", "cqssr", LogField::Type::sINT, &LogAccess::marshal_client_req_ssl_reused, + &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("client_req_ssl_resumption_type", "cqssrt", LogField::Type::sINT, + &LogAccess::marshal_client_ssl_resumption_type, &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("client_req_is_internal", "cqint", LogField::Type::sINT, &LogAccess::marshal_client_req_is_internal, + &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("client_req_mptcp", "cqmpt", LogField::Type::sINT, &LogAccess::marshal_client_req_mptcp_state, + &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("client_sec_protocol", "cqssv", LogField::Type::STRING, &LogAccess::marshal_client_security_protocol, + &LogAccess::unmarshal_str)); + + register_field(new LogField("client_cipher_suite", "cqssc", LogField::Type::STRING, + &LogAccess::marshal_client_security_cipher_suite, &LogAccess::unmarshal_str)); + + register_field(new LogField("client_curve", "cqssu", LogField::Type::STRING, &LogAccess::marshal_client_security_curve, + &LogAccess::unmarshal_str)); + + register_field(new LogField("client_group", "cqssg", LogField::Type::STRING, &LogAccess::marshal_client_security_group, + &LogAccess::unmarshal_str)); + + register_field(new LogField("client_sec_alpn", "cqssa", LogField::Type::STRING, &LogAccess::marshal_client_security_alpn, + &LogAccess::unmarshal_str)); // TLS handshake bytes - bytes received from client during TLS handshake - field = new LogField("client_tls_handshake_bytes_rx", "cthbr", LogField::Type::sINT, - &LogAccess::marshal_client_tls_handshake_bytes_rx, &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("cthbr", field); + register_field(new LogField("client_tls_handshake_bytes_rx", "cthbr", LogField::Type::sINT, + &LogAccess::marshal_client_tls_handshake_bytes_rx, &LogAccess::unmarshal_int_to_str)); // TLS handshake bytes - bytes sent to client during TLS handshake - field = new LogField("client_tls_handshake_bytes_tx", "cthbt", LogField::Type::sINT, - &LogAccess::marshal_client_tls_handshake_bytes_tx, &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("cthbt", field); + register_field(new LogField("client_tls_handshake_bytes_tx", "cthbt", LogField::Type::sINT, + &LogAccess::marshal_client_tls_handshake_bytes_tx, &LogAccess::unmarshal_int_to_str)); // TLS handshake bytes - total (rx + tx) during TLS handshake - field = new LogField("client_tls_handshake_bytes", "cthb", LogField::Type::sINT, &LogAccess::marshal_client_tls_handshake_bytes, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("cthb", field); + register_field(new LogField("client_tls_handshake_bytes", "cthb", LogField::Type::sINT, + &LogAccess::marshal_client_tls_handshake_bytes, &LogAccess::unmarshal_int_to_str)); Ptr finish_status_map = make_ptr(new LogFieldAliasTable); finish_status_map->init(N_LOG_FINISH_CODE_TYPES, LOG_FINISH_FIN, "FIN", LOG_FINISH_INTR, "INTR", LOG_FINISH_TIMEOUT, "TIMEOUT"); - field = new LogField("client_finish_status_code", "cfsc", LogField::Type::sINT, &LogAccess::marshal_client_finish_status_code, - &LogAccess::unmarshal_finish_status, make_alias_map(finish_status_map)); - global_field_list.add(field, false); - field_symbol_hash.emplace("cfsc", field); + register_field(new LogField("client_finish_status_code", "cfsc", LogField::Type::sINT, + &LogAccess::marshal_client_finish_status_code, &LogAccess::unmarshal_finish_status, + make_alias_map(finish_status_map))); - field = new LogField("client_req_id", "crid", LogField::Type::sINT, &LogAccess::marshal_client_req_id, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("crid", field); + register_field(new LogField("client_req_id", "crid", LogField::Type::sINT, &LogAccess::marshal_client_req_id, + &LogAccess::unmarshal_int_to_str)); - field = new LogField("client_req_uuid", "cruuid", LogField::Type::STRING, &LogAccess::marshal_client_req_uuid, - &LogAccess::unmarshal_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("cruuid", field); + register_field(new LogField("client_req_uuid", "cruuid", LogField::Type::STRING, &LogAccess::marshal_client_req_uuid, + &LogAccess::unmarshal_str)); - field = new LogField("client_rx_error_code", "crec", LogField::Type::STRING, &LogAccess::marshal_client_rx_error_code, - &LogAccess::unmarshal_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("crec", field); + register_field(new LogField("client_rx_error_code", "crec", LogField::Type::STRING, &LogAccess::marshal_client_rx_error_code, + &LogAccess::unmarshal_str)); - field = new LogField("client_tx_error_code", "ctec", LogField::Type::STRING, &LogAccess::marshal_client_tx_error_code, - &LogAccess::unmarshal_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("ctec", field); + register_field(new LogField("client_tx_error_code", "ctec", LogField::Type::STRING, &LogAccess::marshal_client_tx_error_code, + &LogAccess::unmarshal_str)); - field = new LogField("client_request_all_header_fields", "cqah", LogField::Type::STRING, - &LogAccess::marshal_client_req_all_header_fields, &LogUtils::unmarshalMimeHdr); - global_field_list.add(field, false); - field_symbol_hash.emplace("cqah", field); + register_field(new LogField("client_request_all_header_fields", "cqah", LogField::Type::STRING, + &LogAccess::marshal_client_req_all_header_fields, &LogUtils::unmarshalMimeHdr)); // proxy -> client fields - field = new LogField("proxy_resp_content_type", "psct", LogField::Type::STRING, &LogAccess::marshal_proxy_resp_content_type, - &LogAccess::unmarshal_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("psct", field); + register_field(new LogField("proxy_resp_content_type", "psct", LogField::Type::STRING, + &LogAccess::marshal_proxy_resp_content_type, &LogAccess::unmarshal_str)); - field = new LogField("proxy_resp_reason_phrase", "prrp", LogField::Type::STRING, &LogAccess::marshal_proxy_resp_reason_phrase, - &LogAccess::unmarshal_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("prrp", field); + register_field(new LogField("proxy_resp_reason_phrase", "prrp", LogField::Type::STRING, + &LogAccess::marshal_proxy_resp_reason_phrase, &LogAccess::unmarshal_str)); - field = new LogField("proxy_resp_squid_len", "psql", LogField::Type::sINT, &LogAccess::marshal_proxy_resp_squid_len, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("psql", field); + register_field(new LogField("proxy_resp_squid_len", "psql", LogField::Type::sINT, &LogAccess::marshal_proxy_resp_squid_len, + &LogAccess::unmarshal_int_to_str)); // Squid length plus TLS handshake bytes sent for TLS connections - field = new LogField("proxy_resp_squid_len_tls", "psqtl", LogField::Type::sINT, &LogAccess::marshal_proxy_resp_squid_len_tls, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("psqtl", field); - - field = new LogField("proxy_resp_content_len", "pscl", LogField::Type::sINT, &LogAccess::marshal_proxy_resp_content_len, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("pscl", field); - - field = new LogField("proxy_resp_content_len_hex", "psch", LogField::Type::sINT, &LogAccess::marshal_proxy_resp_content_len, - &LogAccess::unmarshal_int_to_str_hex); - global_field_list.add(field, false); - field_symbol_hash.emplace("psch", field); - - field = new LogField("proxy_resp_status_code", "pssc", LogField::Type::sINT, &LogAccess::marshal_proxy_resp_status_code, - &LogAccess::unmarshal_http_status); - global_field_list.add(field, false); - field_symbol_hash.emplace("pssc", field); - - field = new LogField("proxy_response_status_code_setter", "prscs", LogField::Type::STRING, - &LogAccess::marshal_status_plugin_entry, &LogAccess::unmarshal_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("prscs", field); - - field = new LogField("proxy_resp_header_len", "pshl", LogField::Type::sINT, &LogAccess::marshal_proxy_resp_header_len, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("pshl", field); - - field = new LogField("proxy_finish_status_code", "pfsc", LogField::Type::sINT, &LogAccess::marshal_proxy_finish_status_code, - &LogAccess::unmarshal_finish_status, make_alias_map(finish_status_map)); - global_field_list.add(field, false); - field_symbol_hash.emplace("pfsc", field); + register_field(new LogField("proxy_resp_squid_len_tls", "psqtl", LogField::Type::sINT, + &LogAccess::marshal_proxy_resp_squid_len_tls, &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("proxy_resp_content_len", "pscl", LogField::Type::sINT, &LogAccess::marshal_proxy_resp_content_len, + &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("proxy_resp_content_len_hex", "psch", LogField::Type::sINT, + &LogAccess::marshal_proxy_resp_content_len, &LogAccess::unmarshal_int_to_str_hex)); + + register_field(new LogField("proxy_resp_status_code", "pssc", LogField::Type::sINT, &LogAccess::marshal_proxy_resp_status_code, + &LogAccess::unmarshal_http_status)); + + register_field(new LogField("proxy_response_status_code_setter", "prscs", LogField::Type::STRING, + &LogAccess::marshal_status_plugin_entry, &LogAccess::unmarshal_str)); + + register_field(new LogField("proxy_resp_header_len", "pshl", LogField::Type::sINT, &LogAccess::marshal_proxy_resp_header_len, + &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("proxy_finish_status_code", "pfsc", LogField::Type::sINT, + &LogAccess::marshal_proxy_finish_status_code, &LogAccess::unmarshal_finish_status, + make_alias_map(finish_status_map))); Ptr cache_code_map = make_ptr(new LogFieldAliasTable); cache_code_map->init( @@ -753,61 +622,40 @@ Log::init_fields() SQUID_MISS_ERROR, "MISS_ERROR", SQUID_MISS_CACHE_BYPASS, "MISS_CACHE_BYPASS", SQUID_HIT_MISS_INVALID_ASSIGNED_CODE, "INVALID_CODE"); - field = new LogField("cache_result_code", "crc", LogField::Type::sINT, &LogAccess::marshal_cache_result_code, - &LogAccess::unmarshal_cache_code, make_alias_map(cache_code_map)); - global_field_list.add(field, false); - field_symbol_hash.emplace("crc", field); + register_field(new LogField("cache_result_code", "crc", LogField::Type::sINT, &LogAccess::marshal_cache_result_code, + &LogAccess::unmarshal_cache_code, make_alias_map(cache_code_map))); // Reuse the unmarshalling code from crc - field = new LogField("cache_result_subcode", "crsc", LogField::Type::sINT, &LogAccess::marshal_cache_result_subcode, - &LogAccess::unmarshal_cache_code, make_alias_map(cache_subcode_map)); - global_field_list.add(field, false); - field_symbol_hash.emplace("crsc", field); + register_field(new LogField("cache_result_subcode", "crsc", LogField::Type::sINT, &LogAccess::marshal_cache_result_subcode, + &LogAccess::unmarshal_cache_code, make_alias_map(cache_subcode_map))); - field = new LogField("cache_hit_miss", "chm", LogField::Type::sINT, &LogAccess::marshal_cache_hit_miss, - &LogAccess::unmarshal_cache_hit_miss, make_alias_map(cache_hit_miss_map)); - global_field_list.add(field, false); - field_symbol_hash.emplace("chm", field); + register_field(new LogField("cache_hit_miss", "chm", LogField::Type::sINT, &LogAccess::marshal_cache_hit_miss, + &LogAccess::unmarshal_cache_hit_miss, make_alias_map(cache_hit_miss_map))); - field = new LogField("proxy_response_all_header_fields", "psah", LogField::Type::STRING, - &LogAccess::marshal_proxy_resp_all_header_fields, &LogUtils::unmarshalMimeHdr); - global_field_list.add(field, false); - field_symbol_hash.emplace("psah", field); + register_field(new LogField("proxy_response_all_header_fields", "psah", LogField::Type::STRING, + &LogAccess::marshal_proxy_resp_all_header_fields, &LogUtils::unmarshalMimeHdr)); // proxy -> server fields - field = new LogField("proxy_req_header_len", "pqhl", LogField::Type::sINT, &LogAccess::marshal_proxy_req_header_len, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("pqhl", field); - - field = new LogField("proxy_req_squid_len", "pqql", LogField::Type::sINT, &LogAccess::marshal_proxy_req_squid_len, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("pqql", field); - - field = new LogField("proxy_req_content_len", "pqcl", LogField::Type::sINT, &LogAccess::marshal_proxy_req_content_len, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("pqcl", field); - - field = new LogField("proxy_req_server_ip", "pqsi", LogField::Type::IP, &LogAccess::marshal_proxy_req_server_ip, - &LogAccess::unmarshal_ip_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("pqsi", field); - - field = new LogField("proxy_req_server_port", "pqsp", LogField::Type::sINT, &LogAccess::marshal_proxy_req_server_port, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("pqsp", field); - - field = new LogField("next_hop_ip", "nhi", LogField::Type::IP, &LogAccess::marshal_next_hop_ip, &LogAccess::unmarshal_ip_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("nhi", field); - - field = - new LogField("next_hop_port", "nhp", LogField::Type::sINT, &LogAccess::marshal_next_hop_port, &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("nhp", field); + register_field(new LogField("proxy_req_header_len", "pqhl", LogField::Type::sINT, &LogAccess::marshal_proxy_req_header_len, + &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("proxy_req_squid_len", "pqql", LogField::Type::sINT, &LogAccess::marshal_proxy_req_squid_len, + &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("proxy_req_content_len", "pqcl", LogField::Type::sINT, &LogAccess::marshal_proxy_req_content_len, + &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("proxy_req_server_ip", "pqsi", LogField::Type::IP, &LogAccess::marshal_proxy_req_server_ip, + &LogAccess::unmarshal_ip_to_str)); + + register_field(new LogField("proxy_req_server_port", "pqsp", LogField::Type::sINT, &LogAccess::marshal_proxy_req_server_port, + &LogAccess::unmarshal_int_to_str)); + + register_field( + new LogField("next_hop_ip", "nhi", LogField::Type::IP, &LogAccess::marshal_next_hop_ip, &LogAccess::unmarshal_ip_to_str)); + + register_field(new LogField("next_hop_port", "nhp", LogField::Type::sINT, &LogAccess::marshal_next_hop_port, + &LogAccess::unmarshal_int_to_str)); Ptr hierarchy_map = make_ptr(new LogFieldAliasTable); hierarchy_map->init( @@ -833,283 +681,176 @@ Log::init_fields() SquidHierarchyCode::TIMEOUT_TIMEOUT_SSL_PARENT_MISS, "TIMEOUT_TIMEOUT_SSL_PARENT_MISS", SquidHierarchyCode::INVALID_ASSIGNED_CODE, "INVALID_ASSIGNED_CODE"); - field = new LogField("proxy_hierarchy_route", "phr", LogField::Type::sINT, &LogAccess::marshal_proxy_hierarchy_route, - &LogAccess::unmarshal_hierarchy, make_alias_map(hierarchy_map)); - global_field_list.add(field, false); - field_symbol_hash.emplace("phr", field); - - field = - new LogField("proxy_host_name", "phn", LogField::Type::STRING, &LogAccess::marshal_proxy_host_name, &LogAccess::unmarshal_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("phn", field); - - field = - new LogField("proxy_host_ip", "phi", LogField::Type::IP, &LogAccess::marshal_proxy_host_ip, &LogAccess::unmarshal_ip_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("phi", field); - - field = new LogField("proxy_host_port", "php", LogField::Type::sINT, &LogAccess::marshal_proxy_host_port, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("php", field); - - field = new LogField("proxy_req_is_ssl", "pqssl", LogField::Type::sINT, &LogAccess::marshal_proxy_req_is_ssl, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("pqssl", field); - - field = new LogField("proxy_req_ssl_reused", "pqssr", LogField::Type::sINT, &LogAccess::marshal_proxy_req_ssl_reused, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("pqssr", field); - - field = new LogField("proxy_request_all_header_fields", "pqah", LogField::Type::STRING, - &LogAccess::marshal_proxy_req_all_header_fields, &LogUtils::unmarshalMimeHdr); - global_field_list.add(field, false); - field_symbol_hash.emplace("pqah", field); + register_field(new LogField("proxy_hierarchy_route", "phr", LogField::Type::sINT, &LogAccess::marshal_proxy_hierarchy_route, + &LogAccess::unmarshal_hierarchy, make_alias_map(hierarchy_map))); + + register_field( + new LogField("proxy_host_name", "phn", LogField::Type::STRING, &LogAccess::marshal_proxy_host_name, &LogAccess::unmarshal_str)); + + register_field( + new LogField("proxy_host_ip", "phi", LogField::Type::IP, &LogAccess::marshal_proxy_host_ip, &LogAccess::unmarshal_ip_to_str)); + + register_field(new LogField("proxy_host_port", "php", LogField::Type::sINT, &LogAccess::marshal_proxy_host_port, + &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("proxy_req_is_ssl", "pqssl", LogField::Type::sINT, &LogAccess::marshal_proxy_req_is_ssl, + &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("proxy_req_ssl_reused", "pqssr", LogField::Type::sINT, &LogAccess::marshal_proxy_req_ssl_reused, + &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("proxy_request_all_header_fields", "pqah", LogField::Type::STRING, + &LogAccess::marshal_proxy_req_all_header_fields, &LogUtils::unmarshalMimeHdr)); // server -> proxy fields - field = - new LogField("server_host_ip", "shi", LogField::Type::IP, &LogAccess::marshal_server_host_ip, &LogAccess::unmarshal_ip_to_str); - - global_field_list.add(field, false); - field_symbol_hash.emplace("shi", field); - - field = new LogField("server_host_name", "shn", LogField::Type::STRING, &LogAccess::marshal_server_host_name, - &LogAccess::unmarshal_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("shn", field); - - field = new LogField("server_resp_status_code", "sssc", LogField::Type::sINT, &LogAccess::marshal_server_resp_status_code, - &LogAccess::unmarshal_http_status); - global_field_list.add(field, false); - field_symbol_hash.emplace("sssc", field); - - field = new LogField("server_resp_content_len", "sscl", LogField::Type::sINT, &LogAccess::marshal_server_resp_content_len, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("sscl", field); - - field = new LogField("server_resp_header_len", "sshl", LogField::Type::sINT, &LogAccess::marshal_server_resp_header_len, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("sshl", field); - - field = new LogField("server_resp_squid_len", "ssql", LogField::Type::sINT, &LogAccess::marshal_server_resp_squid_len, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("ssql", field); - - field = new LogField("server_resp_http_version", "sshv", LogField::Type::dINT, &LogAccess::marshal_server_resp_http_version, - &LogAccess::unmarshal_http_version); - global_field_list.add(field, false); - field_symbol_hash.emplace("sshv", field); - - field = - new LogField("milestones_csv", "mstsms", LogField::Type::STRING, &LogAccess::marshal_milestones_csv, &LogAccess::unmarshal_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("mstsms", field); - - field = new LogField("server_resp_time", "stms", LogField::Type::sINT, &LogAccess::marshal_server_resp_time_ms, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("stms", field); - - field = new LogField("server_resp_time_hex", "stmsh", LogField::Type::sINT, &LogAccess::marshal_server_resp_time_ms, - &LogAccess::unmarshal_int_to_str_hex); - global_field_list.add(field, false); - field_symbol_hash.emplace("stmsh", field); - - field = new LogField("server_resp_time_fractional", "stmsf", LogField::Type::sINT, &LogAccess::marshal_server_resp_time_ms, - &LogAccess::unmarshal_ttmsf); - global_field_list.add(field, false); - field_symbol_hash.emplace("stmsf", field); - - field = new LogField("server_resp_time_sec", "sts", LogField::Type::sINT, &LogAccess::marshal_server_resp_time_s, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("sts", field); - - field = new LogField("server_transact_count", "sstc", LogField::Type::sINT, &LogAccess::marshal_server_transact_count, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("sstc", field); - - field = new LogField("server_unavailable_retry_count", "surc", LogField::Type::sINT, - &LogAccess::marshal_server_unavailable_retry_count, &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("surc", field); - - field = new LogField("server_simple_retry_count", "ssrc", LogField::Type::sINT, &LogAccess::marshal_server_simple_retry_count, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("ssrc", field); - - field = new LogField("server_connect_attempts", "sca", LogField::Type::sINT, &LogAccess::marshal_server_connect_attempts, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("sca", field); - - field = new LogField("origin_response_all_header_fields", "ssah", LogField::Type::STRING, - &LogAccess::marshal_server_resp_all_header_fields, &LogUtils::unmarshalMimeHdr); - global_field_list.add(field, false); - field_symbol_hash.emplace("ssah", field); - - field = new LogField("cached_resp_status_code", "csssc", LogField::Type::sINT, &LogAccess::marshal_cache_resp_status_code, - &LogAccess::unmarshal_http_status); - global_field_list.add(field, false); - field_symbol_hash.emplace("csssc", field); - - field = new LogField("cached_resp_content_len", "csscl", LogField::Type::sINT, &LogAccess::marshal_cache_resp_content_len, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("csscl", field); - - field = new LogField("cached_resp_header_len", "csshl", LogField::Type::sINT, &LogAccess::marshal_cache_resp_header_len, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("csshl", field); - - field = new LogField("cached_resp_squid_len", "cssql", LogField::Type::sINT, &LogAccess::marshal_cache_resp_squid_len, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("cssql", field); - - field = new LogField("cached_resp_http_version", "csshv", LogField::Type::dINT, &LogAccess::marshal_cache_resp_http_version, - &LogAccess::unmarshal_http_version); - global_field_list.add(field, false); - field_symbol_hash.emplace("csshv", field); - - field = new LogField("cache_origin_response_all_header_fields", "cssah", LogField::Type::STRING, - &LogAccess::marshal_cache_resp_all_header_fields, &LogUtils::unmarshalMimeHdr); - global_field_list.add(field, false); - field_symbol_hash.emplace("cssah", field); - - field = new LogField("client_retry_after_time", "crat", LogField::Type::sINT, &LogAccess::marshal_client_retry_after_time, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("crat", field); + register_field( + new LogField("server_host_ip", "shi", LogField::Type::IP, &LogAccess::marshal_server_host_ip, &LogAccess::unmarshal_ip_to_str)); + + register_field(new LogField("server_host_name", "shn", LogField::Type::STRING, &LogAccess::marshal_server_host_name, + &LogAccess::unmarshal_str)); + + register_field(new LogField("server_resp_status_code", "sssc", LogField::Type::sINT, &LogAccess::marshal_server_resp_status_code, + &LogAccess::unmarshal_http_status)); + + register_field(new LogField("server_resp_content_len", "sscl", LogField::Type::sINT, &LogAccess::marshal_server_resp_content_len, + &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("server_resp_header_len", "sshl", LogField::Type::sINT, &LogAccess::marshal_server_resp_header_len, + &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("server_resp_squid_len", "ssql", LogField::Type::sINT, &LogAccess::marshal_server_resp_squid_len, + &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("server_resp_http_version", "sshv", LogField::Type::dINT, + &LogAccess::marshal_server_resp_http_version, &LogAccess::unmarshal_http_version)); + + register_field(new LogField("milestones_csv", "mstsms", LogField::Type::STRING, &LogAccess::marshal_milestones_csv, + &LogAccess::unmarshal_str)); + + register_field(new LogField("server_resp_time", "stms", LogField::Type::sINT, &LogAccess::marshal_server_resp_time_ms, + &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("server_resp_time_hex", "stmsh", LogField::Type::sINT, &LogAccess::marshal_server_resp_time_ms, + &LogAccess::unmarshal_int_to_str_hex)); + + register_field(new LogField("server_resp_time_fractional", "stmsf", LogField::Type::sINT, &LogAccess::marshal_server_resp_time_ms, + &LogAccess::unmarshal_ttmsf)); + + register_field(new LogField("server_resp_time_sec", "sts", LogField::Type::sINT, &LogAccess::marshal_server_resp_time_s, + &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("server_transact_count", "sstc", LogField::Type::sINT, &LogAccess::marshal_server_transact_count, + &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("server_unavailable_retry_count", "surc", LogField::Type::sINT, + &LogAccess::marshal_server_unavailable_retry_count, &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("server_simple_retry_count", "ssrc", LogField::Type::sINT, + &LogAccess::marshal_server_simple_retry_count, &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("server_connect_attempts", "sca", LogField::Type::sINT, &LogAccess::marshal_server_connect_attempts, + &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("origin_response_all_header_fields", "ssah", LogField::Type::STRING, + &LogAccess::marshal_server_resp_all_header_fields, &LogUtils::unmarshalMimeHdr)); + + register_field(new LogField("cached_resp_status_code", "csssc", LogField::Type::sINT, &LogAccess::marshal_cache_resp_status_code, + &LogAccess::unmarshal_http_status)); + + register_field(new LogField("cached_resp_content_len", "csscl", LogField::Type::sINT, &LogAccess::marshal_cache_resp_content_len, + &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("cached_resp_header_len", "csshl", LogField::Type::sINT, &LogAccess::marshal_cache_resp_header_len, + &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("cached_resp_squid_len", "cssql", LogField::Type::sINT, &LogAccess::marshal_cache_resp_squid_len, + &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("cached_resp_http_version", "csshv", LogField::Type::dINT, + &LogAccess::marshal_cache_resp_http_version, &LogAccess::unmarshal_http_version)); + + register_field(new LogField("cache_origin_response_all_header_fields", "cssah", LogField::Type::STRING, + &LogAccess::marshal_cache_resp_all_header_fields, &LogUtils::unmarshalMimeHdr)); + + register_field(new LogField("client_retry_after_time", "crat", LogField::Type::sINT, &LogAccess::marshal_client_retry_after_time, + &LogAccess::unmarshal_int_to_str)); // cache write fields Ptr cache_write_code_map = make_ptr(new LogFieldAliasTable); cache_write_code_map->init(N_LOG_CACHE_WRITE_TYPES, LOG_CACHE_WRITE_NONE, "-", LOG_CACHE_WRITE_LOCK_MISSED, "WL_MISS", LOG_CACHE_WRITE_LOCK_ABORTED, "INTR", LOG_CACHE_WRITE_ERROR, "ERR", LOG_CACHE_WRITE_COMPLETE, "FIN"); - field = new LogField("cache_write_result", "cwr", LogField::Type::sINT, &LogAccess::marshal_cache_write_code, - &LogAccess::unmarshal_cache_write_code, make_alias_map(cache_write_code_map)); - global_field_list.add(field, false); - field_symbol_hash.emplace("cwr", field); + register_field(new LogField("cache_write_result", "cwr", LogField::Type::sINT, &LogAccess::marshal_cache_write_code, + &LogAccess::unmarshal_cache_write_code, make_alias_map(cache_write_code_map))); - field = new LogField("cache_write_transform_result", "cwtr", LogField::Type::sINT, &LogAccess::marshal_cache_write_transform_code, - &LogAccess::unmarshal_cache_write_code, make_alias_map(cache_write_code_map)); - global_field_list.add(field, false); - field_symbol_hash.emplace("cwtr", field); + register_field(new LogField("cache_write_transform_result", "cwtr", LogField::Type::sINT, + &LogAccess::marshal_cache_write_transform_code, &LogAccess::unmarshal_cache_write_code, + make_alias_map(cache_write_code_map))); // other fields - field = new LogField("transfer_time_ms", "ttms", LogField::Type::sINT, &LogAccess::marshal_transfer_time_ms, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("ttms", field); - - field = new LogField("transfer_time_ms_hex", "ttmsh", LogField::Type::sINT, &LogAccess::marshal_transfer_time_ms, - &LogAccess::unmarshal_int_to_str_hex); - global_field_list.add(field, false); - field_symbol_hash.emplace("ttmsh", field); - - field = new LogField("transfer_time_ms_fractional", "ttmsf", LogField::Type::sINT, &LogAccess::marshal_transfer_time_ms, - &LogAccess::unmarshal_ttmsf); - global_field_list.add(field, false); - field_symbol_hash.emplace("ttmsf", field); - - field = new LogField("transfer_time_sec", "tts", LogField::Type::sINT, &LogAccess::marshal_transfer_time_s, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("tts", field); - - field = new LogField("file_size", "fsiz", LogField::Type::sINT, &LogAccess::marshal_file_size, &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("fsiz", field); - - field = new LogField("client_connection_id", "ccid", LogField::Type::sINT, &LogAccess::marshal_client_http_connection_id, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("ccid", field); - - field = new LogField("client_transaction_id", "ctid", LogField::Type::sINT, &LogAccess::marshal_client_http_transaction_id, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("ctid", field); - - field = new LogField("cache_read_retry_attempts", "crra", LogField::Type::sINT, &LogAccess::marshal_cache_read_retries, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("crra", field); - - field = new LogField("cache_write_retry_attempts", "cwra", LogField::Type::sINT, &LogAccess::marshal_cache_write_retries, - &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("cwra", field); - - field = new LogField("cache_collapsed_connection_success", "cccs", LogField::Type::sINT, - &LogAccess::marshal_cache_collapsed_connection_success, &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("cccs", field); - - field = new LogField("client_transaction_priority_weight", "ctpw", LogField::Type::sINT, - &LogAccess::marshal_client_http_transaction_priority_weight, &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("ctpw", field); - - field = new LogField("client_transaction_priority_dependence", "ctpd", LogField::Type::sINT, - &LogAccess::marshal_client_http_transaction_priority_dependence, &LogAccess::unmarshal_int_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("ctpd", field); - - field = new LogField("proxy_protocol_version", "ppv", LogField::Type::STRING, &LogAccess::marshal_proxy_protocol_version, - &LogAccess::unmarshal_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("ppv", field); - - field = new LogField("proxy_protocol_src_ip", "pps", LogField::Type::IP, &LogAccess::marshal_proxy_protocol_src_ip, - &LogAccess::unmarshal_ip_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("pps", field); - - field = new LogField("proxy_protocol_dst_ip", "ppd", LogField::Type::IP, &LogAccess::marshal_proxy_protocol_dst_ip, - &LogAccess::unmarshal_ip_to_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("ppd", field); - - field = new LogField("proxy_protocol_authority", "ppa", LogField::Type::STRING, &LogAccess::marshal_proxy_protocol_authority, - &LogAccess::unmarshal_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("ppa", field); - - field = new LogField("proxy_protocol_tls_cipher", "pptc", LogField::Type::STRING, &LogAccess::marshal_proxy_protocol_tls_cipher, - &LogAccess::unmarshal_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("pptc", field); - - field = new LogField("proxy_protocol_tls_version", "pptv", LogField::Type::STRING, &LogAccess::marshal_proxy_protocol_tls_version, - &LogAccess::unmarshal_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("pptv", field); - - field = new LogField("proxy_protocol_tls_group", "pptg", LogField::Type::STRING, &LogAccess::marshal_proxy_protocol_tls_group, - &LogAccess::unmarshal_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("pptg", field); - - field = new LogField("version_build_number", "vbn", LogField::Type::STRING, &LogAccess::marshal_version_build_number, - &LogAccess::unmarshal_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("vbn", field); - - field = - new LogField("version_string", "vs", LogField::Type::STRING, &LogAccess::marshal_version_string, &LogAccess::unmarshal_str); - global_field_list.add(field, false); - field_symbol_hash.emplace("vs", field); + register_field(new LogField("transfer_time_ms", "ttms", LogField::Type::sINT, &LogAccess::marshal_transfer_time_ms, + &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("transfer_time_ms_hex", "ttmsh", LogField::Type::sINT, &LogAccess::marshal_transfer_time_ms, + &LogAccess::unmarshal_int_to_str_hex)); + + register_field(new LogField("transfer_time_ms_fractional", "ttmsf", LogField::Type::sINT, &LogAccess::marshal_transfer_time_ms, + &LogAccess::unmarshal_ttmsf)); + + register_field(new LogField("transfer_time_sec", "tts", LogField::Type::sINT, &LogAccess::marshal_transfer_time_s, + &LogAccess::unmarshal_int_to_str)); + + register_field( + new LogField("file_size", "fsiz", LogField::Type::sINT, &LogAccess::marshal_file_size, &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("client_connection_id", "ccid", LogField::Type::sINT, &LogAccess::marshal_client_http_connection_id, + &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("client_transaction_id", "ctid", LogField::Type::sINT, &LogAccess::marshal_client_http_transaction_id, + &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("cache_read_retry_attempts", "crra", LogField::Type::sINT, &LogAccess::marshal_cache_read_retries, + &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("cache_write_retry_attempts", "cwra", LogField::Type::sINT, &LogAccess::marshal_cache_write_retries, + &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("cache_collapsed_connection_success", "cccs", LogField::Type::sINT, + &LogAccess::marshal_cache_collapsed_connection_success, &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("client_transaction_priority_weight", "ctpw", LogField::Type::sINT, + &LogAccess::marshal_client_http_transaction_priority_weight, &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("client_transaction_priority_dependence", "ctpd", LogField::Type::sINT, + &LogAccess::marshal_client_http_transaction_priority_dependence, &LogAccess::unmarshal_int_to_str)); + + register_field(new LogField("proxy_protocol_version", "ppv", LogField::Type::STRING, &LogAccess::marshal_proxy_protocol_version, + &LogAccess::unmarshal_str)); + + register_field(new LogField("proxy_protocol_src_ip", "pps", LogField::Type::IP, &LogAccess::marshal_proxy_protocol_src_ip, + &LogAccess::unmarshal_ip_to_str)); + + register_field(new LogField("proxy_protocol_dst_ip", "ppd", LogField::Type::IP, &LogAccess::marshal_proxy_protocol_dst_ip, + &LogAccess::unmarshal_ip_to_str)); + + register_field(new LogField("proxy_protocol_authority", "ppa", LogField::Type::STRING, + &LogAccess::marshal_proxy_protocol_authority, &LogAccess::unmarshal_str)); + + register_field(new LogField("proxy_protocol_tls_cipher", "pptc", LogField::Type::STRING, + &LogAccess::marshal_proxy_protocol_tls_cipher, &LogAccess::unmarshal_str)); + + register_field(new LogField("proxy_protocol_tls_version", "pptv", LogField::Type::STRING, + &LogAccess::marshal_proxy_protocol_tls_version, &LogAccess::unmarshal_str)); + + register_field(new LogField("proxy_protocol_tls_group", "pptg", LogField::Type::STRING, + &LogAccess::marshal_proxy_protocol_tls_group, &LogAccess::unmarshal_str)); + + register_field(new LogField("version_build_number", "vbn", LogField::Type::STRING, &LogAccess::marshal_version_build_number, + &LogAccess::unmarshal_str)); + + register_field( + new LogField("version_string", "vs", LogField::Type::STRING, &LogAccess::marshal_version_string, &LogAccess::unmarshal_str)); init_status |= FIELDS_INITIALIZED; } diff --git a/src/proxy/logging/LogField.cc b/src/proxy/logging/LogField.cc index e913eed3a92..0a469316de7 100644 --- a/src/proxy/logging/LogField.cc +++ b/src/proxy/logging/LogField.cc @@ -987,9 +987,9 @@ LogField::marshal_fallback_default(char *buf) const /*------------------------------------------------------------------------- LogFieldList - It is ASSUMED that each element on this list has been allocated from the - heap with "new" and that each element is on at most ONE list. To enforce - this, items are copied by default, using the copy ctor. + add() takes ownership of the field, so each element must be allocated from + the heap with "new" and must be on at most ONE list; the list deletes its + elements in clear(). -------------------------------------------------------------------------*/ LogFieldList::LogFieldList() = default; @@ -1010,15 +1010,11 @@ LogFieldList::clear() } void -LogFieldList::add(LogField *field, bool copy) +LogFieldList::add(LogField *field) { ink_assert(field != nullptr); - if (copy) { - m_field_list.enqueue(new LogField(*field)); - } else { - m_field_list.enqueue(field); - } + m_field_list.enqueue(field); if (field->type() == LogField::Type::sINT) { m_marshal_len += INK_MIN_ALIGN; diff --git a/src/proxy/logging/LogFormat.cc b/src/proxy/logging/LogFormat.cc index 378eebb8ad8..c66ab11f2e8 100644 --- a/src/proxy/logging/LogFormat.cc +++ b/src/proxy/logging/LogFormat.cc @@ -490,7 +490,7 @@ LogFormat::parse_symbol_string(const char *symbol_string, LogFieldList *field_li } else { LogField *new_f = new LogField(*f); new_f->set_aggregate_op(aggregate); - field_list->add(new_f, false); + field_list->add(new_f); field_count++; *contains_aggregates = true; Dbg(dbg_ctl_log_agg, "Aggregate field %s(%s) added", sym, name); @@ -527,7 +527,7 @@ LogFormat::parse_symbol_string(const char *symbol_string, LogFieldList *field_li if (valid_fallback) { f = new LogField(original_symbol.c_str(), std::move(parsed->header_fields), std::move(fallback_field), std::move(parsed->fallback_default)); - field_list->add(f, false); + field_list->add(f); field_count++; Dbg(dbg_ctl_log_format, "Field fallback field %s added", original_symbol.c_str()); } @@ -559,7 +559,7 @@ LogFormat::parse_symbol_string(const char *symbol_string, LogFieldList *field_li f->m_slice = slice; Dbg(dbg_ctl_log_slice, "symbol = %s, [%d:%d]", sym, f->m_slice.m_start, f->m_slice.m_end); } - field_list->add(f, false); + field_list->add(f); field_count++; Dbg(dbg_ctl_log_format, "Container field {%s}%s added", name, sym); } @@ -576,7 +576,7 @@ LogFormat::parse_symbol_string(const char *symbol_string, LogFieldList *field_li Dbg(dbg_ctl_log_format, "Regular field symbol: %s", symbol); f = make_regular_field_from_symbol(symbol); if (f != nullptr) { - field_list->add(f, false); + field_list->add(f); field_count++; Dbg(dbg_ctl_log_format, "Regular field %s added", symbol); } else { diff --git a/src/proxy/logging/YamlLogConfig.cc b/src/proxy/logging/YamlLogConfig.cc index 313ca9bfdb8..954f055c80f 100644 --- a/src/proxy/logging/YamlLogConfig.cc +++ b/src/proxy/logging/YamlLogConfig.cc @@ -27,6 +27,7 @@ #include "proxy/logging/LogObject.h" #include "tscore/EnumDescriptor.h" +#include "tsutil/StringCompare.h" #include #include @@ -176,9 +177,9 @@ YamlLogConfig::decodeLogObject(const YAML::Node &node) LogFileFormat file_type = LOG_FILE_ASCII; // default value if (node["mode"]) { std::string mode = node["mode"].as(); - file_type = (0 == strncasecmp(mode.c_str(), "bin", 3) || (1 == mode.size() && mode[0] == 'b') ? + file_type = (ts::iequals(mode, "binary") || ts::iequals(mode, "bin") || ts::iequals(mode, "b") ? LOG_FILE_BINARY : - (0 == strcasecmp(mode.c_str(), "ascii_pipe") ? LOG_FILE_PIPE : LOG_FILE_ASCII)); + (ts::iequals(mode, "ascii_pipe") ? LOG_FILE_PIPE : LOG_FILE_ASCII)); } int obj_rolling_enabled = cfg->rolling_enabled; diff --git a/src/proxy/logging/YamlLogConfigDecoders.cc b/src/proxy/logging/YamlLogConfigDecoders.cc index 0c3b438c35a..cbabf591d4a 100644 --- a/src/proxy/logging/YamlLogConfigDecoders.cc +++ b/src/proxy/logging/YamlLogConfigDecoders.cc @@ -24,6 +24,8 @@ #include "proxy/logging/LogConfig.h" #include "proxy/logging/LogObject.h" +#include "tsutil/StringCompare.h" + #include #include #include @@ -73,9 +75,9 @@ convert>::decode(const Node &node, std::unique_ptr(); - if (!strncasecmp(escape.c_str(), "json", 4)) { + if (ts::iequals(escape, "json")) { escape_type = LOG_ESCAPE_JSON; - } else if (!strncasecmp(escape.c_str(), "none", 4)) { + } else if (ts::iequals(escape, "none")) { escape_type = LOG_ESCAPE_NONE; } else { throw YAML::ParserException(node.Mark(), "invalid 'escape' argument '" + escape + "' for format name '" + name + "'"); diff --git a/src/records/P_RecDefs.h b/src/records/P_RecDefs.h index a712d2dc8b6..3c3e770d8f0 100644 --- a/src/records/P_RecDefs.h +++ b/src/records/P_RecDefs.h @@ -143,5 +143,3 @@ struct RecMessageItr { using RecMessage = RecMessageHdr; using RecDumpEntryCb = void (*)(RecT, void *, int, const char *, int, RecData *); - -using RecMessageRecvCb = RecErrT (*)(RecMessage *, RecMessageT, void *); diff --git a/src/records/P_RecMessage.h b/src/records/P_RecMessage.h index ecc5969c2f3..32c9955288a 100644 --- a/src/records/P_RecMessage.h +++ b/src/records/P_RecMessage.h @@ -43,7 +43,5 @@ RecMessage *RecMessageMarshal_Realloc(RecMessage *msg, const RecRecord *record); int RecMessageUnmarshalFirst(RecMessage *msg, RecMessageItr *itr, RecRecord **record); int RecMessageUnmarshalNext(RecMessage *msg, RecMessageItr *itr, RecRecord **record); -int RecMessageRegisterRecvCb(RecMessageRecvCb recv_cb, void *cookie); - RecMessage *RecMessageReadFromDisk(const char *fpath); int RecMessageWriteToDisk(RecMessage *msg, const char *fpath); diff --git a/src/records/RecCore.cc b/src/records/RecCore.cc index 91dd163717c..36ffa3ea489 100644 --- a/src/records/RecCore.cc +++ b/src/records/RecCore.cc @@ -585,7 +585,7 @@ RecLookupMatchingRecords(unsigned rec_type, const char *match, void (*callback)( if ((rec_type & (RECT_PROCESS | RECT_NODE | RECT_PLUGIN))) { // First find the new metrics, this is a bit of a hack, because we still use the old // librecords callback with a "pseudo" record. - for (auto &&[name, type, val] : ts::Metrics::instance()) { + ts::Metrics::instance().for_each([&](std::string_view name, ts::Metrics::MetricType type, int64_t val) { if (regex.exec(name.data())) { RecRecord tmp{}; @@ -596,7 +596,7 @@ RecLookupMatchingRecords(unsigned rec_type, const char *match, void (*callback)( tmp.data.rec_int = val; callback(&tmp, data); } - } + }); // Finally check string metrics ts::Metrics::StaticString::instance().for_each([&](const std::string &name, const std::string &value) { if (regex.exec(name)) { @@ -617,14 +617,14 @@ RecLookupMatchingRecords(unsigned rec_type, const char *match, void (*callback)( if (rec_type & RECT_HIDDEN_METRIC) { // Opt-in only: hidden metrics are never reachable through RECT_ALL, see RecDefs.h. auto &hidden = ts::Metrics::hidden_instance(); - // Slot 0 of every Storage is the reserved bad_id placeholder, so it exists under the same name - // in both stores. Skip it here, otherwise a query matching it returns two identically named - // records that differ only in value. - auto it = hidden.begin(); - ++it; - for (; it != hidden.end(); ++it) { - auto &&[name, type, val] = *it; + hidden.for_each([&](std::string_view name, ts::Metrics::MetricType type, int64_t val) { + // Slot 0 of every Storage is the reserved bad_id placeholder, so it exists under the same + // name in both stores. Skip it here, otherwise a query matching it returns two identically + // named records that differ only in value. + if (name == ts::Metrics::BAD_ID_NAME) { + return; + } if (regex.exec(name.data())) { RecRecord tmp{}; @@ -641,7 +641,7 @@ RecLookupMatchingRecords(unsigned rec_type, const char *match, void (*callback)( tmp.data.rec_int = val; callback(&tmp, data); } - } + }); } int num_records = g_num_records; @@ -968,11 +968,11 @@ RecDumpRecords(RecT rec_type, RecDumpEntryCb callback, void *edata) // Dump all new metrics as well (no "type" for them) RecData datum; - for (auto &&[name, type, val] : ts::Metrics::instance()) { + ts::Metrics::instance().for_each([&](std::string_view name, Metrics::MetricType type, int64_t val) { datum.rec_int = val; callback(RECT_PLUGIN, edata, true, name.data(), type == Metrics::MetricType::COUNTER ? TS_RECORDDATATYPE_COUNTER : TS_RECORDDATATYPE_INT, &datum); - } + }); ts::Metrics::StaticString::instance().for_each([&](const std::string &name, const std::string &value) { datum.rec_string = const_cast(value.c_str()); diff --git a/src/records/RecMessage.cc b/src/records/RecMessage.cc index df42ea5e23d..251ffabf92b 100644 --- a/src/records/RecMessage.cc +++ b/src/records/RecMessage.cc @@ -33,9 +33,6 @@ #include "tscore/Layout.h" #include "swoc/MemSpan.h" -static RecMessageRecvCb g_recv_cb = nullptr; -static void *g_recv_cookie = nullptr; - //------------------------------------------------------------------------- // RecMessageAlloc //------------------------------------------------------------------------- @@ -224,22 +221,6 @@ RecMessageUnmarshalNext(RecMessage *msg, RecMessageItr *itr, RecRecord **record) return REC_ERR_OKAY; } -//------------------------------------------------------------------------- -// RecMessageRegisterRecvCb -//------------------------------------------------------------------------- - -int -RecMessageRegisterRecvCb(RecMessageRecvCb recv_cb, void *cookie) -{ - if (g_recv_cb) { - return REC_ERR_FAIL; - } - g_recv_cookie = cookie; - g_recv_cb = recv_cb; - - return REC_ERR_OKAY; -} - //------------------------------------------------------------------------- // RecMessageReadFromDisk //------------------------------------------------------------------------- diff --git a/src/records/RecordsConfig.cc b/src/records/RecordsConfig.cc index 3d381bc71f2..63eb0b4c047 100644 --- a/src/records/RecordsConfig.cc +++ b/src/records/RecordsConfig.cc @@ -406,7 +406,7 @@ static constexpr RecordElement RecordsConfig[] = , {RECT_CONFIG, "proxy.config.http.per_server.connection.metric_enabled", RECD_INT, "0", RECU_DYNAMIC, RR_NULL, RECC_STR, "^[0-1]$", RECA_NULL} , - {RECT_CONFIG, "proxy.config.http.per_server.connection.metric_aggregate", RECD_INT, "0", RECU_DYNAMIC, RR_NULL, RECC_STR, "^[0-2]$", RECA_NULL} + {RECT_CONFIG, "proxy.config.http.per_server.connection.metric_aggregate", RECD_INT, "0", RECU_DYNAMIC, RR_NULL, RECC_STR, "^[0-3]$", RECA_NULL} , {RECT_CONFIG, "proxy.config.http.per_server.connection.metric_prefix", RECD_STRING, "", RECU_DYNAMIC, RR_NULL, RECC_NULL, nullptr, RECA_NULL} , @@ -1715,15 +1715,6 @@ matches_bracketed_int_range(std::string_view s) return i == s.size(); // must end exactly here } -// For string literals: deduces N and strips the trailing '\0' -template -consteval bool -matches_bracketed_int_range(const char (&lit)[N]) -{ - // N includes the null terminator - return matches_bracketed_int_range(std::string_view{lit, N - 1}); -} - // Validate all RECC_INT entries in the RecordsConfig array at compile time template consteval bool diff --git a/src/records/unit_tests/test_RecHiddenMetricLookup.cc b/src/records/unit_tests/test_RecHiddenMetricLookup.cc index d1e2d4c4fcb..c4f6e2e3b16 100644 --- a/src/records/unit_tests/test_RecHiddenMetricLookup.cc +++ b/src/records/unit_tests/test_RecHiddenMetricLookup.cc @@ -115,3 +115,59 @@ TEST_CASE("RecLookupMatchingRecords - hidden metrics", "[librecords][RecLookup][ } } } + +TEST_CASE("RecLookupMatchingRecords - unlisted metrics", "[librecords][RecLookup][unlisted]") +{ + const std::string name = "proxy.test.lookup.unlisted_gauge"; + auto *m = ts::Metrics::Gauge::createPtr(name); + + REQUIRE(m != nullptr); + m->store(7); + + auto &metrics = ts::Metrics::instance(); + auto id = metrics.lookup(name); + + REQUIRE(id != ts::Metrics::NOT_FOUND); + REQUIRE(metrics.unlist(id)); + + SECTION("an unlisted metric is not enumerated") + { + std::vector entries; + + REQUIRE(RecLookupMatchingRecords(RECT_ALL, name.c_str(), collect, &entries) == REC_ERR_OKAY); + + for (const auto &e : entries) { + CHECK(e.name != name); + } + } + + SECTION("an unlisted metric is still found by exact name") + { + // RecLookupRecord resolves through Metrics::lookup() rather than iteration, which is what keeps + // logging fields and TSStatFindName working across an unlisting. + std::vector entries; + + REQUIRE(RecLookupRecord(name.c_str(), collect, &entries) == REC_ERR_OKAY); + REQUIRE(entries.size() == 1); + CHECK(entries[0].name == name); + CHECK(entries[0].int_value == 7); + } + + SECTION("relisting puts it back in enumeration") + { + REQUIRE(metrics.relist(id)); + + std::vector entries; + bool found = false; + + REQUIRE(RecLookupMatchingRecords(RECT_ALL, name.c_str(), collect, &entries) == REC_ERR_OKAY); + for (const auto &e : entries) { + if (e.name == name) { + found = true; + CHECK(e.int_value == 7); + } + } + + REQUIRE(found); + } +} diff --git a/src/traffic_cache_tool/CacheTool.cc b/src/traffic_cache_tool/CacheTool.cc index c28c2111c42..12b9a79c536 100644 --- a/src/traffic_cache_tool/CacheTool.cc +++ b/src/traffic_cache_tool/CacheTool.cc @@ -21,6 +21,7 @@ limitations under the License. */ +#include #include #include #include diff --git a/src/traffic_server/SocksProxy.cc b/src/traffic_server/SocksProxy.cc index 7ddd9309d42..aaa0ab2f875 100644 --- a/src/traffic_server/SocksProxy.cc +++ b/src/traffic_server/SocksProxy.cc @@ -117,7 +117,7 @@ struct SocksProxy : public Continuation { int setupHttpRequest(unsigned char *p); int sendResp(bool granted); - void init(NetVConnection *netVC); + void init(NetVConnection *netVC, HttpSessionAccept *http_acceptor); void free(); private: @@ -128,9 +128,10 @@ struct SocksProxy : public Continuation { IOBufferReader *reader = nullptr; Event *timeout = nullptr; - SocksAuthHandler auth_handler = nullptr; - SocksProxyHandler vc_handler = nullptr; - Action *pending_action = nullptr; + SocksAuthHandler auth_handler = nullptr; + SocksProxyHandler vc_handler = nullptr; + Action *pending_action = nullptr; + HttpSessionAccept *http_acceptor = nullptr; unsigned char version = 0; int port = 0; @@ -141,11 +142,12 @@ struct SocksProxy : public Continuation { ClassAllocator socksProxyAllocator("socksProxyAllocator"); void -SocksProxy::init(NetVConnection *netVC) +SocksProxy::init(NetVConnection *netVC, HttpSessionAccept *http_accept) { - mutex = new_ProxyMutex(); - buf = new_MIOBuffer(BUFFER_SIZE_INDEX_32K); - reader = buf->alloc_reader(); + mutex = new_ProxyMutex(); + buf = new_MIOBuffer(BUFFER_SIZE_INDEX_32K); + reader = buf->alloc_reader(); + http_acceptor = http_accept; SCOPED_MUTEX_LOCK(lock, mutex, this_ethread()); @@ -551,14 +553,10 @@ SocksProxy::state_handing_over_http_request(int event, [[maybe_unused]] void *da switch (event) { case VC_EVENT_WRITE_COMPLETE: { - HttpSessionAccept::Options ha_opt; - ts::Metrics::Counter::increment(stats.http_connections); Dbg(dbg_ctl_SocksProxy, "Handing over the HTTP request"); - ha_opt.transport_type = clientVC->attributes; - HttpSessionAccept http_accept(ha_opt); - if (!http_accept.accept(clientVC, buf, reader)) { + if (!http_acceptor->accept(clientVC, buf, reader)) { state = SOCKS_ERROR; } else { state = ALL_DONE; @@ -673,10 +671,10 @@ SocksProxy::setupHttpRequest(unsigned char *p) } static void -new_SocksProxy(NetVConnection *netVC) +new_SocksProxy(NetVConnection *netVC, HttpSessionAccept *http_acceptor) { SocksProxy *proxy = socksProxyAllocator.alloc(); - proxy->init(netVC); + proxy->init(netVC, http_acceptor); } struct SocksAccepter : public Continuation { @@ -688,13 +686,15 @@ struct SocksAccepter : public Continuation { ink_assert(event == NET_EVENT_ACCEPT); // Dbg(dbg_ctl_Socks, "Accepter got ACCEPT event"); - new_SocksProxy(netVC); + new_SocksProxy(netVC, &http_acceptor); return EVENT_CONT; } // There is no state used we dont need a mutex SocksAccepter() : Continuation(nullptr) { SET_HANDLER(&SocksAccepter::mainEvent); } + + HttpSessionAccept http_acceptor; }; void diff --git a/src/tsutil/CMakeLists.txt b/src/tsutil/CMakeLists.txt index 7d0e9a651ca..541419421ce 100644 --- a/src/tsutil/CMakeLists.txt +++ b/src/tsutil/CMakeLists.txt @@ -30,6 +30,7 @@ set(TSUTIL_PUBLIC_HEADERS ${PROJECT_SOURCE_DIR}/include/tsutil/LocalBuffer.h ${PROJECT_SOURCE_DIR}/include/tsutil/PostScript.h ${PROJECT_SOURCE_DIR}/include/tsutil/Strerror.h + ${PROJECT_SOURCE_DIR}/include/tsutil/StringCompare.h ${PROJECT_SOURCE_DIR}/include/tsutil/StringConvert.h ${PROJECT_SOURCE_DIR}/include/tsutil/Regex.h ${PROJECT_SOURCE_DIR}/include/tsutil/TsMutex.h @@ -79,6 +80,7 @@ if(BUILD_TESTING) unit_tests/test_Metrics.cc unit_tests/test_PostScript.cc unit_tests/test_Strerror.cc + unit_tests/test_StringCompare.cc unit_tests/test_StringConvert.cc unit_tests/test_Regex.cc unit_tests/test_ts_meta.cc diff --git a/src/tsutil/Metrics.cc b/src/tsutil/Metrics.cc index 1168c7d2e44..8e2cf3e240a 100644 --- a/src/tsutil/Metrics.cc +++ b/src/tsutil/Metrics.cc @@ -77,6 +77,10 @@ Metrics::Storage::create(std::string_view name, const MetricType type) auto it = _lookups.find(name); if (it != _lookups.end()) { + // Re-creating a name relists it: same slot, same atomic, and whatever value it accumulated + // while it was out of the listing. A name in _lookups always names an allocated slot. + set_listed(it->second, true); + return it->second; } @@ -137,8 +141,8 @@ Metrics::Storage::lookup(Metrics::IdType id, std::string_view *out_name, Metrics } if (out_type) { - // don't trust the passed in id to get the type as it might have been manufactured (i.e. from iterators) - // so get the type from the storage tuple. + // don't trust the passed in id to get the type as it might have been manufactured, so get the + // type from the storage tuple. *out_type = _extractType(std::get<1>(std::get<0>(*blob)[offset])); } @@ -190,18 +194,37 @@ Metrics::Storage::type(IdType id) const return _extractType(id); } -// Iterator implementation -void -Metrics::iterator::next() +bool +Metrics::Storage::set_listed(Metrics::IdType id, bool listed) { - auto [blob, offset] = _metrics._splitID(_it); + if (!_is_allocated(id)) { + return false; + } + + auto [blob_ix, offset] = _splitID(id); + Metrics::NamesAndAtomics *blob = _blobs[blob_ix].get(); + + // Only this bit, so a flag added later is not clobbered by unlisting or relisting. + if (listed) { + std::get<2>(*blob)[offset].fetch_and(static_cast(~UNLISTED), MEMORY_ORDER); + } else { + std::get<2>(*blob)[offset].fetch_or(UNLISTED, MEMORY_ORDER); + } + + return true; +} - if (++offset == MAX_SIZE) { - ++blob; - offset = 0; +bool +Metrics::Storage::listed(Metrics::IdType id) const +{ + if (!_is_allocated(id)) { + return false; } - _it = _makeId(blob, offset, MetricType::COUNTER); + auto [blob_ix, offset] = _splitID(id); + Metrics::NamesAndAtomics *blob = _blobs[blob_ix].get(); + + return (std::get<2>(*blob)[offset].load(MEMORY_ORDER) & UNLISTED) == 0; } namespace details @@ -269,11 +292,43 @@ namespace details if (it == metrics.end()) { metrics.push_back(DerivedMetric{id, {source}, op}); + } else if (std::find(it->derived_from.begin(), it->derived_from.end(), source) == it->derived_from.end()) { + // Already registered sources are skipped so repeated registration is harmless. + it->derived_from.push_back(source); + } + + // Under the lock, and after the source is registered. create() has already relisted, but a + // remove_source that emptied the list concurrently would otherwise be free to unlist after + // this source was added, leaving a metric that is recomputed but never enumerated. + Metrics::instance().relist(id); + } + + void + remove_source(Metrics::IdType id, Metrics::AtomicType *source) + { + if (!source) { return; } - // Already registered sources are skipped so repeated registration is harmless. - if (std::find(it->derived_from.begin(), it->derived_from.end(), source) == it->derived_from.end()) { - it->derived_from.push_back(source); + + std::lock_guard l(metrics_lock); + auto it = std::find_if(metrics.begin(), metrics.end(), [id](DerivedMetric const &m) { return m.metric == id; }); + + if (it == metrics.end()) { + return; + } + + auto src = std::find(it->derived_from.begin(), it->derived_from.end(), source); + + if (src == it->derived_from.end()) { + return; // Not a source of this metric, so nothing about it changes. + } + + it->derived_from.erase(src); + + // Under the lock with the erase, for the reason given in add_source. The entry stays, holding + // no sources: update() skips those, and add_source finds it again if a contributor returns. + if (it->derived_from.empty()) { + Metrics::instance().unlist(id); } } @@ -335,6 +390,19 @@ Metrics::Derived::add_source(std::string_view derived_name, Metrics::MetricType details::DerivativeMetrics::instance().add_source(id, source, op); } +void +Metrics::Derived::remove_source(std::string_view derived_name, Metrics::AtomicType *source) +{ + auto &instance = Metrics::instance(); + auto id = instance.lookup(derived_name); + + if (id == Metrics::NOT_FOUND) { + return; + } + + details::DerivativeMetrics::instance().remove_source(id, source); +} + Metrics::StaticString & Metrics::StaticString::instance() { diff --git a/src/tsutil/unit_tests/test_Metrics.cc b/src/tsutil/unit_tests/test_Metrics.cc index f267097807f..034a790c49e 100644 --- a/src/tsutil/unit_tests/test_Metrics.cc +++ b/src/tsutil/unit_tests/test_Metrics.cc @@ -26,7 +26,6 @@ #include #include #include -#include #include #include #include @@ -40,33 +39,37 @@ TEST_CASE("Metrics", "[libtsapi][Metrics]") { auto &m = Metrics::instance(); - SECTION("iterator") + SECTION("for_each") { - auto [name, type, value] = *m.begin(); - REQUIRE(value == 0); - REQUIRE(type == Metrics::MetricType::COUNTER); - REQUIRE(name == "proxy.process.api.metrics.bad_id"); + std::vector names; + int64_t first_value = -1; + Metrics::MetricType first_type{}; + + m.for_each([&](std::string_view name, Metrics::MetricType type, int64_t value) { + if (names.empty()) { + first_value = value; + first_type = type; + } + names.emplace_back(name); + }); - REQUIRE(m.begin() != m.end()); + // The reserved bad_id occupies the first slot of every store, so it is always visited first. + REQUIRE_FALSE(names.empty()); + REQUIRE(names.front() == Metrics::BAD_ID_NAME); + REQUIRE(first_value == 0); + REQUIRE(first_type == Metrics::MetricType::COUNTER); - // Other test cases share this process-wide store, so the number of metrics already present - // is not knowable here. Assert the delta from creating one metric instead of an absolute - // iterator position. - auto pre_count = std::distance(m.begin(), m.end()); + // Other test cases share this process-wide store, so the number of metrics already present is + // not knowable here. Assert the delta from creating one metric instead of an absolute count. + auto const pre_count = names.size(); - Metrics::Counter::create("iterator.marker"); - REQUIRE(std::distance(m.begin(), m.end()) == pre_count + 1); + Metrics::Counter::create("for_each.marker"); - auto it = m.begin(); - std::advance(it, pre_count); - REQUIRE(it != m.end()); - ++it; - REQUIRE(it == m.end()); + names.clear(); + m.for_each([&](std::string_view name, Metrics::MetricType, int64_t) { names.emplace_back(name); }); - auto it2 = m.begin(); - std::advance(it2, pre_count); - it2++; - REQUIRE(it2 == m.end()); + REQUIRE(names.size() == pre_count + 1); + REQUIRE(names.back() == "for_each.marker"); // creation order, so the newest is last } SECTION("New metric") @@ -295,6 +298,131 @@ TEST_CASE("Metrics derived ops", "[libtsapi][Metrics]") } } +TEST_CASE("Metrics derived remove_source", "[libtsapi][Metrics]") +{ + auto &m = Metrics::instance(); + + SECTION("a derived metric stays listed while any source remains") + { + auto a = Metrics::Gauge::createHiddenPtr("rm.a"); + auto b = Metrics::Gauge::createHiddenPtr("rm.b"); + + Metrics::Derived::add_source("rm.sum", Metrics::MetricType::GAUGE, a); + Metrics::Derived::add_source("rm.sum", Metrics::MetricType::GAUGE, b); + + Metrics::Gauge::store(a, 3); + Metrics::Gauge::store(b, 4); + Metrics::Derived::update_derived(); + + auto const id = m.lookup("rm.sum"); + REQUIRE(id != Metrics::NOT_FOUND); + REQUIRE(m[id].load() == 7); + REQUIRE(m.listed(id)); + + // One contributor drops out. The metric is still someone else's, so it stays listed and now + // reports only what is left. + Metrics::Derived::remove_source("rm.sum", b); + Metrics::Derived::update_derived(); + + CHECK(m.listed(id)); + CHECK(m[id].load() == 3); + + // The last one drops out, so nothing is contributing and the name goes out of the listing. + Metrics::Derived::remove_source("rm.sum", a); + + CHECK_FALSE(m.listed(id)); + } + + SECTION("re-adding a source relists it") + { + auto a = Metrics::Gauge::createHiddenPtr("rm.relist.a"); + + Metrics::Derived::add_source("rm.relist", Metrics::MetricType::GAUGE, a); + + auto const id = m.lookup("rm.relist"); + REQUIRE(id != Metrics::NOT_FOUND); + + Metrics::Derived::remove_source("rm.relist", a); + REQUIRE_FALSE(m.listed(id)); + + Metrics::Gauge::store(a, 11); + Metrics::Derived::add_source("rm.relist", Metrics::MetricType::GAUGE, a); + Metrics::Derived::update_derived(); + + CHECK(m.listed(id)); + CHECK(m[id].load() == 11); + } + + SECTION("removing an unknown source or name is harmless") + { + auto a = Metrics::Gauge::createHiddenPtr("rm.unknown.a"); + + Metrics::Derived::remove_source("rm.no.such.derived", a); + + Metrics::Derived::add_source("rm.unknown", Metrics::MetricType::GAUGE, a); + + auto const id = m.lookup("rm.unknown"); + auto other = Metrics::Gauge::createHiddenPtr("rm.unknown.other"); + + // Not a source of this metric, so it must not empty the list or unlist anything. + Metrics::Derived::remove_source("rm.unknown", other); + + CHECK(m.listed(id)); + } + + SECTION("a concurrent add and remove cannot leave a contributing metric unlisted") + { + constexpr int64_t sentinel = 4242; + constexpr int rounds = 500; + + auto a = Metrics::Gauge::createHiddenPtr("rm.race.a"); + + Metrics::Gauge::store(a, sentinel); + Metrics::Derived::add_source("rm.race", Metrics::MetricType::GAUGE, a); + + auto const id = m.lookup("rm.race"); + REQUIRE(id != Metrics::NOT_FOUND); + + // One add against one remove per round, checked while quiescent: a single racing pair keeps a + // bad outcome visible at the end of its round instead of being papered over by later + // operations. Only the two orderings matter, so more concurrency would not probe anything new. + bool unlisted_with_source = false; + + for (int round = 0; round < rounds && !unlisted_with_source; ++round) { + Metrics::Derived::add_source("rm.race", Metrics::MetricType::GAUGE, a); + + std::atomic ready{0}; + auto start = [&]() { + ready.fetch_add(1); + while (ready.load() < 2) { + std::this_thread::yield(); + } + }; + + std::thread adder([&]() { + start(); + Metrics::Derived::add_source("rm.race", Metrics::MetricType::GAUGE, a); + }); + std::thread remover([&]() { + start(); + Metrics::Derived::remove_source("rm.race", a); + }); + + adder.join(); + remover.join(); + + // update_derived writes only names that still have a source, so the sentinel landing in a + // zeroed slot is how a registered source is observed from outside. + m[id].store(0); + Metrics::Derived::update_derived(); + + unlisted_with_source = (m[id].load() == sentinel) && !m.listed(id); + } + + CHECK_FALSE(unlisted_with_source); + } +} + TEST_CASE("Metrics derived add_source", "[libtsapi][Metrics]") { auto &m = Metrics::instance(); @@ -437,18 +565,12 @@ TEST_CASE("Metrics hidden store", "[libtsapi][Metrics]") // Not visible in the published store, by name or by iteration. REQUIRE(m.lookup("hidden.only") == Metrics::NOT_FOUND); - for (auto &&[name, type, value] : m) { - REQUIRE(name != "hidden.only"); - } + m.for_each([](std::string_view name, Metrics::MetricType, int64_t) { REQUIRE(name != "hidden.only"); }); // Visible in the hidden store. REQUIRE(h.lookup("hidden.only") != Metrics::NOT_FOUND); bool found = false; - for (auto &&[name, type, value] : h) { - if (name == "hidden.only") { - found = true; - } - } + h.for_each([&](std::string_view name, Metrics::MetricType, int64_t) { found |= (name == "hidden.only"); }); REQUIRE(found); } @@ -707,3 +829,214 @@ TEST_CASE("Metrics id lookup is safe against concurrent creation", "[libtsapi][M // would mean the sweep above never left the first one. REQUIRE(hi - lo > Metrics::MAX_SIZE); } + +TEST_CASE("Metrics unlisting", "[libtsapi][Metrics]") +{ + auto &m = Metrics::instance(); + + SECTION("an unlisted metric is skipped by iteration") + { + Metrics::Counter::create("unlisted.iter.before"); + auto target = Metrics::Counter::create("unlisted.iter.target"); + Metrics::Counter::create("unlisted.iter.after"); + + REQUIRE(m.unlist(target)); + + bool saw_before = false, saw_target = false, saw_after = false; + + m.for_each([&](std::string_view name, Metrics::MetricType, int64_t) { + saw_before |= (name == "unlisted.iter.before"); + saw_target |= (name == "unlisted.iter.target"); + saw_after |= (name == "unlisted.iter.after"); + }); + + REQUIRE(saw_before); + REQUIRE_FALSE(saw_target); + REQUIRE(saw_after); + } + + SECTION("creating an unlisted name again relists it") + { + auto p = Metrics::Counter::createPtr("unlisted.resurrect"); + auto id = m.lookup("unlisted.resurrect"); + + Metrics::Counter::increment(p, 5); + REQUIRE(m.unlist(id)); + REQUIRE_FALSE(m.listed(id)); + + // Same name, same id, same atomic, and the mark is gone. + auto p2 = Metrics::Counter::createPtr("unlisted.resurrect"); + REQUIRE(p2 == p); + REQUIRE(m.lookup("unlisted.resurrect") == id); + REQUIRE(m.listed(id)); + + // Visible again, with its value intact. + bool found = false; + m.for_each([&](std::string_view name, Metrics::MetricType, int64_t value) { + if (name == "unlisted.resurrect") { + found = true; + REQUIRE(value == 5); + } + }); + REQUIRE(found); + } + + SECTION("unlist and relist by name") + { + auto id = Metrics::Counter::create("unlisted.byname"); + + REQUIRE(m.unlist("unlisted.byname")); + REQUIRE_FALSE(m.listed(id)); + + REQUIRE(m.relist("unlisted.byname")); + REQUIRE(m.listed(id)); + + bool found = false; + m.for_each([&](std::string_view name, Metrics::MetricType, int64_t) { found |= (name == "unlisted.byname"); }); + REQUIRE(found); + + // A name that was never created cannot be marked. + REQUIRE_FALSE(m.unlist("unlisted.byname.never.created")); + } + + SECTION("an unlisted metric is still resolvable and still counts") + { + auto p = Metrics::Counter::createPtr("unlisted.resolvable"); + auto id = m.lookup("unlisted.resolvable"); + + REQUIRE(m.unlist(id)); + + // Hidden from enumeration is not gone: by name, by id, and through the atomic it is unchanged. + REQUIRE(m.lookup("unlisted.resolvable") == id); + REQUIRE(m.lookup(id) == p); + REQUIRE(m.valid(id)); + REQUIRE(m.name(id) == "unlisted.resolvable"); + REQUIRE(m.type(id) == Metrics::MetricType::COUNTER); + + Metrics::Counter::increment(p, 3); + REQUIRE(Metrics::Counter::load(p) == 3); + } + + SECTION("for_each skips an unlisted first slot") + { + // Slot 0 is the reserved bad_id, so it is the first slot the walk considers. The anchor keeps + // the assertions below from passing on an empty walk. + auto bad_id = m.lookup(Metrics::BAD_ID_NAME); + REQUIRE(bad_id == 0); + + Metrics::Counter::create("unlisted.first.anchor"); + + auto first_name = [&]() { + std::string first; + bool seen = false; + + m.for_each([&](std::string_view name, Metrics::MetricType, int64_t) { + if (!seen) { + first = name; + seen = true; + } + }); + + return first; + }; + + REQUIRE(m.unlist(bad_id)); + auto const while_unlisted = first_name(); + + // Relist before asserting: a failed assertion ends the section, and leaving bad_id unlisted + // would break every later test case that expects to see it. + REQUIRE(m.relist(bad_id)); + auto const while_listed = first_name(); + + REQUIRE_FALSE(while_unlisted.empty()); // the walk did visit something + REQUIRE(while_unlisted != Metrics::BAD_ID_NAME); + REQUIRE(while_listed == Metrics::BAD_ID_NAME); + } + + SECTION("an unlisted run at the end of the store terminates iteration") + { + // Skipping the last slots in the store is the case where the skip loop has nothing unmarked + // left to land on. The anchor is a listed metric of this section's own, so the loop below is + // known to have run without depending on what other sections left in the shared store. + constexpr int COUNT = 8; + std::vector names; + + Metrics::Counter::create("unlisted.tail.anchor"); + + names.reserve(COUNT); + for (int i = 0; i < COUNT; ++i) { + names.push_back("unlisted.tail." + std::to_string(i)); + REQUIRE(m.unlist(Metrics::Counter::create(names[i]))); + } + + bool saw_anchor = false; + + m.for_each([&](std::string_view name, Metrics::MetricType, int64_t) { + saw_anchor |= (name == "unlisted.tail.anchor"); + for (auto const &n : names) { + REQUIRE(name != n); + } + }); + + REQUIRE(saw_anchor); + } + + SECTION("for_each reports the type each metric was created with") + { + // The type comes from the slot's own stored id rather than from the walk's position, which is + // what keeps a gauge from being reported as a counter. + Metrics::Gauge::createPtr("unlisted.typed.gauge"); + Metrics::Counter::createPtr("unlisted.typed.counter"); + + bool saw_gauge = false, saw_counter = false; + + m.for_each([&](std::string_view name, Metrics::MetricType type, int64_t) { + if (name == "unlisted.typed.gauge") { + saw_gauge = true; + REQUIRE(type == Metrics::MetricType::GAUGE); + } else if (name == "unlisted.typed.counter") { + saw_counter = true; + REQUIRE(type == Metrics::MetricType::COUNTER); + } + }); + + REQUIRE(saw_gauge); + REQUIRE(saw_counter); + } + + SECTION("an id that names no allocated slot is neither listed nor unlistable") + { + // Storage::_is_allocated is the gate; this only checks that unlist and listed go through it. + // Blob 100 was never allocated, the largest id names an offset past MAX_SIZE, and create() + // advances after writing so the id one past the last one created is not allocated yet. + auto last = Metrics::Counter::create("unlisted.next.free"); + + for (auto id : {Metrics::IdType{100 << 16}, std::numeric_limits::max(), last + 1}) { + CHECK_FALSE(m.unlist(id)); + CHECK_FALSE(m.listed(id)); + } + } + + SECTION("the hidden store unlists independently") + { + auto &h = Metrics::hidden_instance(); + + Metrics::Counter::createPtr("unlisted.dual"); + Metrics::Counter::createHiddenPtr("unlisted.dual"); + + auto pub_id = m.lookup("unlisted.dual"); + auto hid_id = h.lookup("unlisted.dual"); + + REQUIRE(h.unlist(hid_id)); + REQUIRE_FALSE(h.listed(hid_id)); + REQUIRE(m.listed(pub_id)); + + bool in_published = false, in_hidden = false; + + m.for_each([&](std::string_view name, Metrics::MetricType, int64_t) { in_published |= (name == "unlisted.dual"); }); + h.for_each([&](std::string_view name, Metrics::MetricType, int64_t) { in_hidden |= (name == "unlisted.dual"); }); + + REQUIRE(in_published); + REQUIRE_FALSE(in_hidden); + } +} diff --git a/src/tsutil/unit_tests/test_StringCompare.cc b/src/tsutil/unit_tests/test_StringCompare.cc new file mode 100644 index 00000000000..53ff2696fdf --- /dev/null +++ b/src/tsutil/unit_tests/test_StringCompare.cc @@ -0,0 +1,71 @@ +/** @file + + Unit tests for StringCompare.h. + + @section license License + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#include + +#include +#include + +TEST_CASE("iequals", "[STE]") +{ + SECTION("equal") + { + REQUIRE(ts::iequals("Accept-Encoding", "Accept-Encoding")); + REQUIRE(ts::iequals("", "")); + } + + SECTION("differing case") + { + REQUIRE(ts::iequals("ACCEPT-ENCODING", "accept-encoding")); + REQUIRE(ts::iequals("Accept-Encoding", "aCCePt-eNCoDiNG")); + } + + SECTION("prefix is not equal") + { + REQUIRE_FALSE(ts::iequals("Accept", "Accept-Encoding")); + REQUIRE_FALSE(ts::iequals("Accept-Encoding", "Accept")); + REQUIRE_FALSE(ts::iequals("", "Accept-Encoding")); + REQUIRE_FALSE(ts::iequals("Accept-Encoding", "")); + } + + SECTION("suffix is not equal") + { + REQUIRE_FALSE(ts::iequals("Encoding", "Accept-Encoding")); + REQUIRE_FALSE(ts::iequals("Accept-Encoding", "Encoding")); + } + + SECTION("same length, differing content") + { + REQUIRE_FALSE(ts::iequals("gzip", "zstd")); + REQUIRE_FALSE(ts::iequals("Accept-Encoding", "Accept-Language")); + } + + SECTION("length is honored over NUL termination") + { + // Callers build views from a pointer and a length; the terminator must not decide the result. + static constexpr char raw[] = "Accept-Encoding-and-more"; + + REQUIRE(ts::iequals(std::string_view{raw, 15}, "Accept-Encoding")); + REQUIRE_FALSE(ts::iequals(std::string_view{raw, 16}, "Accept-Encoding")); + } +} diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index aa09cd1cf83..bf2032c7e85 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -38,6 +38,7 @@ endfunction() add_subdirectory(tools/plugins) add_subdirectory(gold_tests/chunked_encoding) add_subdirectory(gold_tests/at_headers/plugins) +add_subdirectory(gold_tests/cache/plugins) add_subdirectory(gold_tests/continuations/plugins) add_subdirectory(gold_tests/jsonrpc/plugins) add_subdirectory(gold_tests/pluginTest/crash_test) diff --git a/tests/gold_tests/cache/compat-cache-key-client-conditional.test.py b/tests/gold_tests/cache/compat-cache-key-client-conditional.test.py new file mode 100644 index 00000000000..4701101f318 --- /dev/null +++ b/tests/gold_tests/cache/compat-cache-key-client-conditional.test.py @@ -0,0 +1,24 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +Test.Summary = ''' +Verify that revalidating a compatibility hit strips the client's own +conditional headers in the configurations where build_request() forwards them, +so the origin cannot answer with a 304 that would leave the object unmigrated. +A HEAD, which has no body to store, keeps them. +''' + +Test.ATSReplayTest(replay_file="replay/compat-cache-key-client-conditional.replay.yaml") diff --git a/tests/gold_tests/cache/compat-cache-key.test.py b/tests/gold_tests/cache/compat-cache-key.test.py new file mode 100644 index 00000000000..3b9e3d9ccd5 --- /dev/null +++ b/tests/gold_tests/cache/compat-cache-key.test.py @@ -0,0 +1,35 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +Test.Summary = ''' +Verify proxy.config.http.cache.try_compat_key_read: objects stored under the +ATS 9.2 cache key are served while fresh, revalidated without conditional +headers so the full response is stored under the current key, and left in place +under the legacy key to age out. Deletes reach both keys, a lost write lock +still serves the legacy object, and a plugin update of one is served but not +stored. +''' + +Test.ContinueOnFail = True + +Test.ATSReplayTest(replay_file="replay/compat-cache-key.replay.yaml") + +# The write lock loss is injected through a remap plugin, so it needs a TS of +# its own. +Test.ATSReplayTest(replay_file="replay/compat-cache-key-write-lock.replay.yaml") + +# A plugin update of the cached object, driven by a global test plugin. +Test.ATSReplayTest(replay_file="replay/compat-cache-key-plugin-update.replay.yaml") diff --git a/tests/gold_tests/cache/plugins/CMakeLists.txt b/tests/gold_tests/cache/plugins/CMakeLists.txt new file mode 100644 index 00000000000..5ef3e710dbb --- /dev/null +++ b/tests/gold_tests/cache/plugins/CMakeLists.txt @@ -0,0 +1,18 @@ +####################### +# +# Licensed to the Apache Software Foundation (ASF) under one or more contributor license +# agreements. See the NOTICE file distributed with this work for additional information regarding +# copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software distributed under the License +# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express +# or implied. See the License for the specific language governing permissions and limitations under +# the License. +# +####################### + +add_autest_plugin(update_cached_object update_cached_object.cc) diff --git a/tests/gold_tests/cache/plugins/update_cached_object.cc b/tests/gold_tests/cache/plugins/update_cached_object.cc new file mode 100644 index 00000000000..f65ba67ef68 --- /dev/null +++ b/tests/gold_tests/cache/plugins/update_cached_object.cc @@ -0,0 +1,136 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// Exercises TSHttpTxnUpdateCachedObject(). On a fresh cache hit whose request +// carries X-Update-Cached-Object, copy that header's value into the cached +// response as X-Cached-Update and ask for the cached object to be updated. + +#include + +#include + +namespace +{ +constexpr char PLUGIN_NAME[] = "update_cached_object"; +constexpr std::string_view TRIGGER_HEADER{"X-Update-Cached-Object"}; +constexpr std::string_view UPDATED_HEADER{"X-Cached-Update"}; + +DbgCtl dbg_ctl{PLUGIN_NAME}; + +std::string_view +header_value(TSMBuffer bufp, TSMLoc hdr_loc, std::string_view name) +{ + TSMLoc field_loc = TSMimeHdrFieldFind(bufp, hdr_loc, name.data(), static_cast(name.size())); + + if (field_loc == TS_NULL_MLOC) { + return {}; + } + + int len = 0; + char const *value = TSMimeHdrFieldValueStringGet(bufp, hdr_loc, field_loc, -1, &len); + + TSHandleMLocRelease(bufp, hdr_loc, field_loc); + return value != nullptr ? std::string_view{value, static_cast(len)} : std::string_view{}; +} + +bool +set_header(TSMBuffer bufp, TSMLoc hdr_loc, std::string_view name, std::string_view value) +{ + TSMLoc field_loc = TS_NULL_MLOC; + + if (TSMimeHdrFieldCreateNamed(bufp, hdr_loc, name.data(), static_cast(name.size()), &field_loc) != TS_SUCCESS) { + return false; + } + + bool const ok = + TSMimeHdrFieldValueStringSet(bufp, hdr_loc, field_loc, -1, value.data(), static_cast(value.size())) == TS_SUCCESS && + TSMimeHdrFieldAppend(bufp, hdr_loc, field_loc) == TS_SUCCESS; + + TSHandleMLocRelease(bufp, hdr_loc, field_loc); + return ok; +} + +void +update_cached_object(TSHttpTxn txnp) +{ + int lookup_status = 0; + + if (TSHttpTxnCacheLookupStatusGet(txnp, &lookup_status) != TS_SUCCESS || lookup_status != TS_CACHE_LOOKUP_HIT_FRESH) { + return; + } + + TSMBuffer req_bufp = nullptr; + TSMLoc req_hdr = TS_NULL_MLOC; + + if (TSHttpTxnClientReqGet(txnp, &req_bufp, &req_hdr) != TS_SUCCESS) { + return; + } + + std::string_view const value = header_value(req_bufp, req_hdr, TRIGGER_HEADER); + + if (!value.empty()) { + TSMBuffer resp_bufp = nullptr; + TSMLoc resp_hdr = TS_NULL_MLOC; + + if (TSHttpTxnCachedRespModifiableGet(txnp, &resp_bufp, &resp_hdr) != TS_SUCCESS) { + TSError("[%s] could not get the modifiable cached response", PLUGIN_NAME); + } else { + if (!set_header(resp_bufp, resp_hdr, UPDATED_HEADER, value)) { + TSError("[%s] could not set %.*s", PLUGIN_NAME, static_cast(UPDATED_HEADER.size()), UPDATED_HEADER.data()); + } else if (TSHttpTxnUpdateCachedObject(txnp) != TS_SUCCESS) { + TSError("[%s] TSHttpTxnUpdateCachedObject failed", PLUGIN_NAME); + } else { + Dbg(dbg_ctl, "requested cached object update with %.*s", static_cast(value.size()), value.data()); + } + TSHandleMLocRelease(resp_bufp, TS_NULL_MLOC, resp_hdr); + } + } + + TSHandleMLocRelease(req_bufp, TS_NULL_MLOC, req_hdr); +} + +int +handle_event(TSCont /* contp ATS_UNUSED */, TSEvent event, void *edata) +{ + auto txnp = static_cast(edata); + + if (event == TS_EVENT_HTTP_CACHE_LOOKUP_COMPLETE) { + update_cached_object(txnp); + } + + TSHttpTxnReenable(txnp, TS_EVENT_HTTP_CONTINUE); + return 0; +} + +} // namespace + +void +TSPluginInit(int /* argc ATS_UNUSED */, char const * /* argv ATS_UNUSED */[]) +{ + TSPluginRegistrationInfo info; + info.plugin_name = const_cast(PLUGIN_NAME); + info.vendor_name = const_cast("Apache"); + info.support_email = const_cast("dev@trafficserver.apache.org"); + + if (TSPluginRegister(&info) != TS_SUCCESS) { + TSError("[%s] Plugin registration failed", PLUGIN_NAME); + return; + } + + TSHttpHookAdd(TS_HTTP_CACHE_LOOKUP_COMPLETE_HOOK, TSContCreate(handle_event, nullptr)); +} diff --git a/tests/gold_tests/cache/replay/compat-cache-key-client-conditional.replay.yaml b/tests/gold_tests/cache/replay/compat-cache-key-client-conditional.replay.yaml new file mode 100644 index 00000000000..81453e1eeb3 --- /dev/null +++ b/tests/gold_tests/cache/replay/compat-cache-key-client-conditional.replay.yaml @@ -0,0 +1,227 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +# +# A compatibility-key object cannot consume a 304, so a GET that revalidates +# one asks the origin for the full response and stores it under the current +# key. build_request() already removes the client's conditionals for a request +# it expects to cache, but it deliberately forwards them in two cases: when the +# request does not look cacheable, and when +# proxy.config.http.cache.when_to_revalidate is 4. In those the origin could +# still answer 304, which would leave the object unmigrated, so +# issue_revalidate() drops them for a legacy object. A HEAD keeps them: its 200 +# has no body to store, so there is nothing to migrate. +# +# This exercises the when_to_revalidate case, which is a global switch and so +# needs a TS of its own rather than sharing one with compat-cache-key. +# +# As in compat-cache-key.replay.yaml, "/cc;" addresses the same key ATS 9.2 +# produced for "/cc", which is how a legacy object gets into the cache. +# + +meta: + version: "1.0" + +autest: + description: 'Verify client-supplied conditional headers on a stale compatibility hit' + + dns: + name: 'dns-compat-cache-key-client-conditional' + + server: + name: 'proxy-verifier-server' + + client: + name: 'proxy-verifier-client' + + ats: + name: 'ts-for-proxy-verifier' + process_config: + enable_cache: true + + records_config: + proxy.config.diags.debug.enabled: 1 + proxy.config.diags.debug.tags: 'http|http_cache' + proxy.config.http.insert_age_in_response: 0 + proxy.config.http.cache.try_compat_key_read: 1 + # Forward the client's conditionals rather than stripping them. + proxy.config.http.cache.when_to_revalidate: 4 + + remap_config: + - from: "http://example.com/" + to: "http://backend.example.com:{SERVER_HTTP_PORT}/" + + metric_checks: + # Two compatibility hits: the stale object migrated in test 2 and the + # stale HEAD in test 5. + - metric: proxy.process.http.cache.compat_key_reads + value: 2 + +sessions: +- transactions: + + # + # Test 1: Prime the cache under the legacy key. + # + - client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /cc; + headers: + fields: + - [ uuid, 1-prime-legacy-key ] + - [ Host, example.com ] + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Length, 16 ] + - [ Cache-Control, max-age=1 ] + - [ ETag, '"legacy-object"' ] + + proxy-response: + status: 200 + + # + # Test 2: The client's own conditional is stripped for the migration. + # + # The object is stale, so this compatibility hit is revalidated. The client + # sent an ETag of its own and when_to_revalidate 4 would forward it, + # which would let the origin answer 304 -- a response this transaction cannot + # apply to a legacy-key object. + # + - client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /cc + headers: + fields: + - [ uuid, 2-client-conditional-stripped ] + - [ Host, example.com ] + - [ If-None-Match, '"something-the-client-has"' ] + # Outlive the max-age=1 above so the object is stale. + delay: 2s + + proxy-request: + headers: + fields: + - [ If-None-Match, { as: absent } ] + - [ If-Modified-Since, { as: absent } ] + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Length, 16 ] + - [ Cache-Control, max-age=300 ] + - [ ETag, '"migrated-object"' ] + + proxy-response: + status: 200 + + # + # Test 3: The object really did migrate to the current key. + # + # Had the origin been allowed to answer 304, nothing would have been stored + # under the current key and this would go back to the origin instead. + # + - client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /cc + headers: + fields: + - [ uuid, 3-migrated-to-current-key ] + - [ Host, example.com ] + delay: 100ms + + proxy-request: + expect: absent + + server-response: + status: 400 + reason: "Bad Request" + headers: + fields: + - [ Content-Length, 0 ] + + proxy-response: + status: 200 + + # + # Test 4-5: a HEAD keeps the client's conditional. + # + # A HEAD of a stale legacy object revalidates as usual. Its response carries + # no body to store, so there is nothing to migrate and no reason to give up + # the 304. The conditional headers are forwarded as they are. + # + - client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /head; + headers: + fields: + - [ uuid, 4-prime-head-target ] + - [ Host, example.com ] + delay: 100ms + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Length, 16 ] + - [ Cache-Control, max-age=1 ] + - [ ETag, '"head-object"' ] + + proxy-response: + status: 200 + + - client-request: + method: "HEAD" + version: "1.1" + scheme: "http" + url: /head + headers: + fields: + - [ uuid, 5-head-keeps-conditional ] + - [ Host, example.com ] + - [ If-None-Match, '"head-object"' ] + # Outlive the max-age=1 above so the object is stale. + delay: 2s + + proxy-request: + headers: + fields: + - [ If-None-Match, { value: '"head-object"', as: equal } ] + + server-response: + status: 304 + reason: "Not Modified" + headers: + fields: + - [ ETag, '"head-object"' ] + + proxy-response: + status: 304 diff --git a/tests/gold_tests/cache/replay/compat-cache-key-plugin-update.replay.yaml b/tests/gold_tests/cache/replay/compat-cache-key-plugin-update.replay.yaml new file mode 100644 index 00000000000..3f666887ad0 --- /dev/null +++ b/tests/gold_tests/cache/replay/compat-cache-key-plugin-update.replay.yaml @@ -0,0 +1,227 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# +# A plugin update of a cached object (TSHttpTxnUpdateCachedObject) cannot be +# stored for an object found under the 9.2 key: the write would be a create on +# the current key, which the cache aborts on a header-only close. ATS takes the +# failed-update path for it instead, so the client receives the updated headers +# but the object is left as it was. +# +# The update_cached_object test plugin copies X-Update-Cached-Object from the +# request into the cached response as X-Cached-Update, on a fresh hit only. +# +# As in compat-cache-key.replay.yaml, "/legacy;" addresses the same key ATS 9.2 +# produced for "/legacy", which is how a legacy object gets into the cache. +# + +meta: + version: "1.0" + + blocks: + # Every request after the priming ones is answered from the cache. + - origin_not_expected: &origin_not_expected + proxy-request: + expect: absent + + server-response: + status: 400 + reason: "Bad Request" + headers: + fields: + - [ Content-Length, 0 ] + +autest: + description: 'Verify a plugin update of a compatibility-key object is served but not stored' + + dns: + name: 'dns-compat-cache-key-plugin-update' + + server: + name: 'origin-compat-cache-key-plugin-update' + + client: + name: 'client-compat-cache-key-plugin-update' + + ats: + name: 'ts-compat-cache-key-plugin-update' + process_config: + enable_cache: true + + plugin_config: + - name: "update_cached_object.so" + + copy_custom_plugin: + - "plugins/.libs/update_cached_object.so" + + records_config: + proxy.config.diags.debug.enabled: 1 + proxy.config.diags.debug.tags: 'http_trans|update_cached_object' + proxy.config.http.insert_age_in_response: 0 + proxy.config.http.cache.try_compat_key_read: 1 + + remap_config: + - from: "http://example.com/" + to: "http://backend.example.com:{SERVER_HTTP_PORT}/" + + metric_checks: + # The update request and the check after it in tests 5 and 6. + - metric: proxy.process.http.cache.compat_key_reads + value: 2 + + log_validation: + traffic_out: + excludes: + - expression: "[Ff]atal|failed assertion" + description: 'Verify ATS does not abort on the refused update' + contains: + - expression: "requested cached object update with current-key-update" + description: 'Verify the plugin updated the object under the current key' + - expression: "requested cached object update with legacy-key-update" + description: 'Verify the plugin updated the object under the legacy key' + +sessions: +- transactions: + + # + # Tests 1-3: control. Under the current key the update is stored. + # + - client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /current + headers: + fields: + - [ uuid, 1-prime-current-key ] + - [ Host, example.com ] + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Length, 16 ] + - [ Cache-Control, max-age=300 ] + + proxy-response: + status: 200 + + - client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /current + headers: + fields: + - [ uuid, 2-update-current-key ] + - [ Host, example.com ] + - [ X-Update-Cached-Object, current-key-update ] + delay: 100ms + + <<: *origin_not_expected + + proxy-response: + status: 200 + headers: + fields: + - [ X-Cached-Update, { value: current-key-update, as: equal } ] + + - client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /current + headers: + fields: + - [ uuid, 3-current-key-update-stored ] + - [ Host, example.com ] + delay: 100ms + + <<: *origin_not_expected + + proxy-response: + status: 200 + headers: + fields: + - [ X-Cached-Update, { value: current-key-update, as: equal } ] + + # + # Tests 4-6: under the legacy key the update is served but not stored. + # + - client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /legacy; + headers: + fields: + - [ uuid, 4-prime-legacy-key ] + - [ Host, example.com ] + delay: 100ms + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Length, 16 ] + - [ Cache-Control, max-age=300 ] + + proxy-response: + status: 200 + + # Answered with the updated headers rather than the error an invalid update + # gets. + - client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /legacy + headers: + fields: + - [ uuid, 5-update-legacy-key ] + - [ Host, example.com ] + - [ X-Update-Cached-Object, legacy-key-update ] + delay: 100ms + + <<: *origin_not_expected + + proxy-response: + status: 200 + headers: + fields: + - [ X-Cached-Update, { value: legacy-key-update, as: equal } ] + + # Still the unmodified legacy object. + - client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /legacy + headers: + fields: + - [ uuid, 6-legacy-key-update-not-stored ] + - [ Host, example.com ] + delay: 100ms + + <<: *origin_not_expected + + proxy-response: + status: 200 + headers: + fields: + - [ X-Cached-Update, { as: absent } ] diff --git a/tests/gold_tests/cache/replay/compat-cache-key-write-lock.replay.yaml b/tests/gold_tests/cache/replay/compat-cache-key-write-lock.replay.yaml new file mode 100644 index 00000000000..be50bb0ad06 --- /dev/null +++ b/tests/gold_tests/cache/replay/compat-cache-key-write-lock.replay.yaml @@ -0,0 +1,146 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# +# A stale compatibility hit is revalidated unconditionally but stays a hit, so +# the stale-serving fallbacks keep working for it. Here the transaction that +# would migrate the object loses the cache write lock, and with +# proxy.config.http.cache.open_write_fail_action 2 (STALE_ON_REVALIDATE) it +# must serve the legacy object rather than go to the origin. +# +# As in compat-cache-key.replay.yaml, "/stale;" addresses the same key ATS 9.2 +# produced for "/stale", which is how a legacy object gets into the cache. The +# write lock loss is injected the same way as in +# cache-write-lock-stale-serve.replay.yaml. +# + +meta: + version: "1.0" + + blocks: + # The stale object has to be served out of the cache, so this response should + # never be seen. Negative revalidating is disabled below so that a 500 from + # the origin is passed through rather than masked by the cached object. + - origin_not_expected: &origin_not_expected + server-response: + status: 500 + reason: "Internal Server Error" + headers: + fields: + - [ Content-Length, 16 ] + - [ X-Response, origin ] + +autest: + description: 'Verify a lost write lock on a stale compatibility hit serves the legacy object' + + dns: + name: 'dns-compat-cache-key-write-lock' + + server: + name: 'origin-compat-cache-key-write-lock' + + client: + name: 'client-compat-cache-key-write-lock' + + ats: + name: 'ts-compat-cache-key-write-lock' + process_config: + enable_cache: true + + copy_to_config_dir: + - 'cache-write-lock-fail-write.conf' + + records_config: + proxy.config.diags.debug.enabled: 1 + proxy.config.diags.debug.tags: 'http_cache|http_trans|http_match' + proxy.config.http.insert_age_in_response: 0 + proxy.config.http.cache.try_compat_key_read: 1 + # STALE_ON_REVALIDATE: serve stale when the write lock is lost. + proxy.config.http.cache.open_write_fail_action: 2 + proxy.config.http.cache.max_open_write_retry_timeout: 0 + proxy.config.http.negative_revalidating_enabled: 0 + + remap_config: + - from: "http://example.com/" + to: "http://backend.example.com:{SERVER_HTTP_PORT}/" + plugins: + - name: header_rewrite.so + args: ['cache-write-lock-fail-write.conf'] + + metric_checks: + - metric: proxy.process.http.cache.compat_key_reads + value: 1 + + log_validation: + traffic_out: + excludes: + - expression: "[Ff]atal|failed assertion" + description: 'Verify ATS does not abort when the write lock is lost' + contains: + - expression: "object under the compatibility key, revalidating unconditionally" + description: 'Verify the legacy object was revalidated rather than treated as a miss' + - expression: "cache_serve_stale_on_write_lock_fail" + description: 'Verify the legacy object was served because the write lock was lost' + +sessions: +- transactions: + + # Populate the legacy key with an object that is stale one second later. + - client-request: + method: GET + version: '1.1' + url: /stale; + headers: + fields: + - [ Host, example.com ] + - [ uuid, prime-legacy-key ] + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Length, 16 ] + - [ Cache-Control, "max-age=1" ] + - [ X-Response, cached ] + + proxy-response: + status: 200 + headers: + fields: + - [ X-Response, { value: cached, as: equal } ] + + # The compatibility lookup finds the stale object and ATS prepares to + # revalidate it, but the header below denies the write lock. + - client-request: + delay: 2s + + method: GET + version: '1.1' + url: /stale + headers: + fields: + - [ Host, example.com ] + - [ uuid, revalidate-without-write-lock ] + - [ X-Fail-Cache-Write, '1' ] + + <<: *origin_not_expected + + proxy-response: + status: 200 + headers: + fields: + - [ X-Response, { value: cached, as: equal } ] diff --git a/tests/gold_tests/cache/replay/compat-cache-key.replay.yaml b/tests/gold_tests/cache/replay/compat-cache-key.replay.yaml new file mode 100644 index 00000000000..79d11af11e2 --- /dev/null +++ b/tests/gold_tests/cache/replay/compat-cache-key.replay.yaml @@ -0,0 +1,632 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# +# ATS 9.2 hashed a cache key as path + ";" + params. The params segment was +# removed from the URL parser, so ";params" now lives inside the path and the +# separator is no longer added -- which is why every 9.2 object misses after an +# upgrade and why proxy.config.http.cache.try_compat_key_read exists. +# +# That same equivalence gives this test a way to create a 9.2-key object using +# only the running proxy: the 9.2 key of "/obj" is byte-identical to the current +# key of "/obj;". Priming the cache with "/obj;" therefore leaves an object that +# only a compatibility lookup for "/obj" can find. +# + +meta: + version: "1.0" + +autest: + description: 'Verify 92x compatibility cache lookups serve and migrate to the current key' + + dns: + name: 'dns-compat-cache-key' + + server: + name: 'proxy-verifier-server' + + client: + name: 'proxy-verifier-client' + + ats: + name: 'ts-for-proxy-verifier' + process_config: + enable_cache: true + + records_config: + proxy.config.diags.debug.enabled: 1 + proxy.config.diags.debug.tags: 'http|http_cache|url_cachekey' + proxy.config.http.insert_age_in_response: 0 + proxy.config.http.cache.try_compat_key_read: 1 + # Honour Cache-Control in the client request, which is how these + # transactions force revalidation without waiting for an object to age. + proxy.config.http.cache.ignore_client_cc_max_age: 0 + + remap_config: + - from: "http://example.com/" + to: "http://backend.example.com:{SERVER_HTTP_PORT}/" + + metric_checks: + # Seven compatibility hits: the fresh serve, the stale revalidation that + # migrates the object, the PUT in test 8, the migration in test 10, the + # no-cache revalidation in test 12, and the range revalidation and the + # hit after it in test 13. + - metric: proxy.process.http.cache.compat_key_reads + value: 7 + +sessions: +- transactions: + + # + # Test 1: Prime the cache under the legacy key. + # + # "/migrate;" hashes to exactly the key ATS 9.2 produced for "/migrate", so + # after this transaction the cache holds what looks like a 9.2 object. + # + - client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /migrate; + headers: + fields: + - [ uuid, 1-prime-legacy-key ] + - [ Host, example.com ] + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Length, 16 ] + - [ Cache-Control, max-age=300 ] + - [ ETag, '"legacy-object"' ] + + proxy-response: + status: 200 + + # + # Test 2: A fresh compatibility hit is served from cache. + # + # The canonical lookup for "/migrate" misses and the compatibility lookup + # finds the object primed above. The origin must not be contacted. + # + - client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /migrate + headers: + fields: + - [ uuid, 2-compat-hit-fresh ] + - [ Host, example.com ] + delay: 100ms + + proxy-request: + expect: absent + + server-response: + status: 400 + reason: "Bad Request" + headers: + fields: + - [ Content-Length, 0 ] + + proxy-response: + status: 200 + + # + # Test 3: A stale compatibility hit is revalidated unconditionally. + # + # A 304 cannot be applied to a legacy-key object, because the write that would + # carry it is a create on the canonical key rather than an update of the + # legacy vector. ATS therefore asks for the full response: no conditional + # headers appear even though the cached object has an ETag, and the 200 is + # stored under the canonical key. + # + - client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /migrate + headers: + fields: + - [ uuid, 3-compat-migrate ] + - [ Host, example.com ] + # Ask for a revalidation rather than sleeping until the object ages out. + # Freshness is decided before any of the compatibility handling, so the + # path under test is the same either way. + - [ Cache-Control, max-age=0 ] + delay: 100ms + + proxy-request: + headers: + fields: + - [ If-None-Match, { as: absent } ] + - [ If-Modified-Since, { as: absent } ] + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Length, 16 ] + - [ Cache-Control, max-age=300 ] + - [ ETag, '"migrated-object"' ] + + proxy-response: + status: 200 + + # + # Test 4: The migrated object is now a canonical-key hit. + # + - client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /migrate + headers: + fields: + - [ uuid, 4-canonical-hit ] + - [ Host, example.com ] + delay: 100ms + + proxy-request: + expect: absent + + server-response: + status: 400 + reason: "Bad Request" + headers: + fields: + - [ Content-Length, 0 ] + + proxy-response: + status: 200 + + # + # Test 5: The legacy copy is left in place. + # + # "/migrate;" addresses the legacy key directly. Migration does not delete it: + # nothing at this layer reports that the new object reached disk, so removing + # it here would lose the object whenever that write later failed. It ages out + # instead, and stops being read as soon as the current key resolves -- which + # is what lets compat_key_reads decay to zero. + # + # Still being in the cache is observable: the object is stale by now, so it + # revalidates conditionally with the ETag it was stored with. + # + - client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /migrate; + headers: + fields: + - [ uuid, 5-legacy-key-retained ] + - [ Host, example.com ] + # Force the revalidation that reveals the stored ETag. + - [ Cache-Control, max-age=0 ] + delay: 100ms + + proxy-request: + headers: + fields: + - [ If-None-Match, { value: '"legacy-object"', as: equal } ] + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Length, 16 ] + - [ Cache-Control, max-age=300 ] + + proxy-response: + status: 200 + + # + # Test 6: A path that already carries ";params" gets no compatibility lookup. + # + # Its current key and its 9.2 key are the same string, so a second lookup + # would only repeat the one that just missed. + # + - client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /keep;jsessionid=abc + headers: + fields: + - [ uuid, 6-params-path-miss ] + - [ Host, example.com ] + delay: 100ms + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Length, 16 ] + - [ Cache-Control, max-age=300 ] + + proxy-response: + status: 200 + + # + # Test 7: ... and it still caches normally under the current key. + # + # Neither of these two transactions takes a compatibility path, so the + # running compat_key_reads count is still 2 after them. The remaining two + # of the four asserted in metric_checks come from tests 8 and 10. + # + - client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /keep;jsessionid=abc + headers: + fields: + - [ uuid, 7-params-path-hit ] + - [ Host, example.com ] + delay: 100ms + + proxy-request: + expect: absent + + server-response: + status: 400 + reason: "Bad Request" + headers: + fields: + - [ Content-Length, 0 ] + + proxy-response: + status: 200 + + # + # Test 8: a PUT keeps the client's precondition. + # + # Only a GET is revalidated unconditionally; a PUT takes the delete path with + # its request untouched. If-Match is the client's concurrency guard: + # dropping it would let the origin overwrite whatever is there now rather than + # only the version the client saw. + # + - client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /put; + headers: + fields: + - [ uuid, 8a-prime-put-target ] + - [ Host, example.com ] + delay: 100ms + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Length, 16 ] + - [ Cache-Control, max-age=300 ] + - [ ETag, '"v1"' ] + + proxy-response: + status: 200 + + - client-request: + method: "PUT" + version: "1.1" + scheme: "http" + url: /put + headers: + fields: + - [ uuid, 8b-put-keeps-if-match ] + - [ Host, example.com ] + - [ Content-Length, 0 ] + - [ If-Match, '"v1"' ] + delay: 100ms + + proxy-request: + headers: + fields: + - [ If-Match, { value: '"v1"', as: equal } ] + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Length, 0 ] + + proxy-response: + status: 200 + + # + # Test 9-11: a purge has to reach both keys. + # + # After a migration the object exists under both the current and the legacy + # key. Removing only the one the purge looked up would leave the other to be + # served afterwards, which for a purge is the whole point of the operation. + # + - client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /purge; + headers: + fields: + - [ uuid, 9-prime-purge-target ] + - [ Host, example.com ] + delay: 100ms + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Length, 16 ] + - [ Cache-Control, max-age=300 ] + - [ ETag, '"legacy-purge"' ] + + proxy-response: + status: 200 + + - client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /purge + headers: + fields: + - [ uuid, 10-migrate-purge-target ] + - [ Host, example.com ] + - [ Cache-Control, max-age=0 ] + delay: 100ms + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Length, 16 ] + - [ Cache-Control, max-age=300 ] + - [ ETag, '"migrated-purge"' ] + + proxy-response: + status: 200 + + - client-request: + method: "PURGE" + version: "1.1" + scheme: "http" + url: /purge + headers: + fields: + - [ uuid, 11a-purge ] + - [ Host, example.com ] + - [ Content-Length, 0 ] + delay: 100ms + + # A purge is answered from the cache and never reaches the origin. + proxy-request: + expect: absent + + server-response: + status: 400 + reason: "Bad Request" + headers: + fields: + - [ Content-Length, 0 ] + + proxy-response: + status: 200 + + - client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /purge + headers: + fields: + - [ uuid, 11b-purged-content-is-gone ] + - [ Host, example.com ] + delay: 100ms + + # Reaching the origin at all is the assertion: a legacy copy that survived + # the purge is still fresh, so it would have been served from cache and the + # origin would never have been contacted. + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Length, 16 ] + - [ Cache-Control, max-age=300 ] + - [ ETag, '"refetched"' ] + + proxy-response: + status: 200 + + # + # Test 12: A fresh compatibility hit that must still be revalidated. + # + # Cache-Control: no-cache on the stored response makes every hit revalidate, + # fresh or not. That revalidation needs the full response just as a stale one + # does, or the object would never leave the legacy key and every request for + # it would go back to the origin. + # + - client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /nocache; + headers: + fields: + - [ uuid, 12a-prime-no-cache-target ] + - [ Host, example.com ] + delay: 100ms + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Length, 16 ] + - [ Cache-Control, "max-age=300, no-cache" ] + - [ ETag, '"legacy-no-cache"' ] + - [ Last-Modified, "Mon, 01 Jan 2024 00:00:00 GMT" ] + + proxy-response: + status: 200 + + - client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /nocache + headers: + fields: + - [ uuid, 12b-fresh-no-cache-unconditional ] + - [ Host, example.com ] + delay: 100ms + + proxy-request: + headers: + fields: + - [ If-None-Match, { as: absent } ] + - [ If-Modified-Since, { as: absent } ] + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Length, 16 ] + - [ Cache-Control, max-age=300 ] + - [ ETag, '"migrated-no-cache"' ] + + proxy-response: + status: 200 + + - client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /nocache + headers: + fields: + - [ uuid, 12c-no-cache-migrated ] + - [ Host, example.com ] + delay: 100ms + + proxy-request: + expect: absent + + server-response: + status: 400 + reason: "Bad Request" + headers: + fields: + - [ Content-Length, 0 ] + + proxy-response: + status: 200 + + # + # Test 13: A range request revalidates conditionally and keeps the object. + # + # A 206 is never stored, so an unconditional range request would draw one + # even for an unchanged object and turn the revalidation into a delete. The + # conditional request lets the origin answer 304 instead, and the object stays + # under the legacy key for the next full GET to migrate. + # + - client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /range; + headers: + fields: + - [ uuid, 13a-prime-range-target ] + - [ Host, example.com ] + delay: 100ms + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Length, 16 ] + - [ Cache-Control, max-age=300 ] + - [ ETag, '"legacy-range"' ] + + proxy-response: + status: 200 + + - client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /range + headers: + fields: + - [ uuid, 13b-range-revalidates-conditionally ] + - [ Host, example.com ] + - [ Range, bytes=0-3 ] + - [ Cache-Control, max-age=0 ] + delay: 100ms + + proxy-request: + headers: + fields: + - [ If-None-Match, { value: '"legacy-range"', as: equal } ] + + server-response: + status: 304 + reason: "Not Modified" + headers: + fields: + - [ ETag, '"legacy-range"' ] + + proxy-response: + status: 206 + + - client-request: + method: "GET" + version: "1.1" + scheme: "http" + url: /range + headers: + fields: + - [ uuid, 13c-range-target-retained ] + - [ Host, example.com ] + delay: 100ms + + # Served from the legacy copy: the range request did not delete it. + proxy-request: + expect: absent + + server-response: + status: 400 + reason: "Bad Request" + headers: + fields: + - [ Content-Length, 0 ] + + proxy-response: + status: 200 diff --git a/tests/gold_tests/h2/h2origin.test.py b/tests/gold_tests/h2/h2origin.test.py index 4bd98b60005..0545cfcfdce 100644 --- a/tests/gold_tests/h2/h2origin.test.py +++ b/tests/gold_tests/h2/h2origin.test.py @@ -47,6 +47,9 @@ 'proxy.config.http.server_session_sharing.pool': 'thread', 'proxy.config.http.server_session_sharing.match': 'ip,sni,cert', 'proxy.config.ssl.client.verify.server.policy': 'PERMISSIVE', + # Flush log buffers promptly so that the squid.log entry for the last + # transaction appears while the test is still waiting for it. + 'proxy.config.log.max_secs_per_buffer': 1, }) ts.Disk.remap_config.AddLines( @@ -79,18 +82,10 @@ tr.Processes.Default.StartBefore(server_expect) tr.Processes.Default.ReturnCode = 0 -tr = Test.AddTestRun("Wait for the squid.log to be written") -timeout = 30 -watcher = tr.Processes.Process("watcher") -watcher.Command = f"sleep {timeout}" -watcher.Ready = When.FileContains(ts.Disk.squid_log.Name, r'16 http/2 http/2') -watcher.TimeOut = timeout +tr = Test.AddAwaitFileContainsTestRun("Wait for the squid.log to be written", ts.Disk.squid_log.Name, r'16 http/2 http/2') + tr.StillRunningAfter = ts tr.StillRunningAfter = server -tr.TimeOut = timeout -tr.Processes.Default.StartBefore(watcher) -tr.Processes.Default.Command = 'echo await_squid_log' -tr.Processes.Default.ReturnCode = 0 # UUIDs 1-4 should be http/1.1 clients and H2 origin # UUIDs 5-11 and 15-16 should be http/2 clients and H2 origins diff --git a/tests/gold_tests/origin_connection/per_server_connection_max.test.py b/tests/gold_tests/origin_connection/per_server_connection_max.test.py index 05f2c5587c1..1724a23b40e 100644 --- a/tests/gold_tests/origin_connection/per_server_connection_max.test.py +++ b/tests/gold_tests/origin_connection/per_server_connection_max.test.py @@ -36,6 +36,11 @@ # scheduling jitter and the traffic_ctl round trip rather than racing the tick. _STAT_SYNC_WAIT_SECONDS: int = 2 +# How long to wait after changing an overridable record at runtime before driving traffic that +# should see the new value. http_config_cb schedules the reconfigure one second out, so a request +# made immediately after traffic_ctl returns is still served by the previous HttpConfigParams. +_CONFIG_APPLY_WAIT_SECONDS: int = 5 + # The records.yaml settings every ATS instance in this file needs for the waits above to hold. _STAT_SYNC_RECORDS: dict = { 'proxy.config.raw_stat_sync_interval_ms': _STAT_SYNC_INTERVAL_MS, @@ -134,7 +139,7 @@ def _test_metrics(self) -> None: # A 'port' match has one group per address:port and no hostname, so no aggregate should be # registered for it at all. tr.Processes.Default.Streams.All += Testers.ExcludesExpression( - 'per_server.current_connection_max.', 'A non-"both" match type must not register a hostname aggregate.') + 'per_server.current_connection.max.', 'A non-"both" match type must not register a hostname aggregate.') def run(self) -> None: """Configure the TestRun.""" @@ -151,11 +156,11 @@ def run(self) -> None: class ConnectMethodTest: """Test our max origin connection behavior with CONNECT traffic. - Also covers the two aggregate-publishing modes of + Also covers two of the aggregate-publishing modes of proxy.config.http.per_server.connection.metric_aggregate: - - 2 (AGGREGATE_ONLY): only the per hostname aggregate is published; the per group metrics - stay hidden and are visible only with --include-hidden. - - 1 (AGGREGATE_GROUP): the per hostname aggregate is published, and the per group metrics + - 3 (AGGREGATE_SUM): the per hostname sums and max are published; the per group metrics stay + hidden and are visible only with --include-hidden. + - 1 (AGGREGATE_GROUP): the per hostname sums and max are published, and the per group metrics are also mirrored into the published store. The match here defaults to 'both' and there is exactly one group for this hostname, so the @@ -166,7 +171,7 @@ class ConnectMethodTest: _process_counter: int = 0 _client_counter: int = 0 - def __init__(self, max_conn, metric_aggregate=2) -> None: + def __init__(self, max_conn, metric_aggregate=3) -> None: """Configure the server processes in preparation for the TestRun.""" self._metric_aggregate = metric_aggregate self._configure_dns() @@ -233,12 +238,13 @@ def _test_metrics(self, blocked) -> None: tr.Processes.Default.Streams.All += Testers.ContainsExpression( f'per_server.total_connection.{group_name} 5', 'The per group metric should be published at AGGREGATE_GROUP.') else: - # AGGREGATE_ONLY keeps the per group metrics hidden, so none of the three per group - # names may appear in a normal query. current_connection_max is not among them: it only + # AGGREGATE_SUM keeps the per group metrics hidden, so none of the three per group + # names may appear in a normal query. current_connection.max is not among them: it only # ever exists as a hostname aggregate, never per group. for counter in ('current_connection', 'total_connection', 'blocked_connection'): tr.Processes.Default.Streams.All += Testers.ExcludesExpression( - f'per_server.{counter}.{group_name} ', f'per_server.{counter}.{group_name} must stay hidden at AGGREGATE_ONLY.') + f'per_server.{counter}.{group_name} ', + f'per_server.{counter}.{group_name} must stay hidden at metric_aggregate {self._metric_aggregate}.') # The per group metrics must be visible with --include-hidden at either level. This is also # the end to end test for that traffic_ctl option. @@ -294,7 +300,7 @@ class MultiGroupAggregateTest: distinct groups sharing one host aggregate. The two groups are given different concurrency so the SUM and the MAX are distinguishable from each other. - current_connection and current_connection_max are instantaneous gauges recomputed from the live + current_connection and current_connection.max are instantaneous gauges recomputed from the live per group values every ~5s, so they rise and fall with traffic rather than remembering a peak. Observing a non-zero value therefore requires holding connections open across a sync tick. The most robust assertion, and the one that actually distinguishes this instantaneous behavior from @@ -340,9 +346,9 @@ def _configure_trafficserver(self) -> None: 'proxy.config.diags.debug.enabled': 1, 'proxy.config.diags.debug.tags': 'http|dns|hostdb|conn_track', 'proxy.config.http.per_server.connection.metric_enabled': 1, - # Aggregates only: the per group metrics stay hidden, which is what this test is - # about reading through the aggregate. - 'proxy.config.http.per_server.connection.metric_aggregate': 2, + # Sums and max, per group metrics hidden: this test is about reading the group + # behavior through the hostname aggregate. + 'proxy.config.http.per_server.connection.metric_aggregate': 3, 'proxy.config.http.per_server.connection.match': 'both', }) self._ts.Disk.remap_config.AddLines( @@ -381,9 +387,16 @@ def _test_metrics_while_held(self) -> None: 'While held open, the host aggregate current_connection should be the SUM of the ' 'currently open connections across both groups.') tr.Processes.Default.Streams.All += Testers.ContainsExpression( - f'per_server.current_connection_max.multi.origin.com {group_max}', - 'While held open, current_connection_max should be the largest single group current ' + f'per_server.current_connection.max.multi.origin.com {group_max}', + 'While held open, current_connection.max should be the largest single group current ' 'count (MAX), not the sum across the two groups.') + # The per group names end in the address, so anything matching this is a group metric and + # not the hostname aggregate. Every other assertion in this file is a ContainsExpression, + # which cannot catch a metric that should not be there at all. + tr.Processes.Default.Streams.All += Testers.ExcludesExpression( + r'per_server\.\w+_connection\.multi\.origin\.com\.\d', + 'At metric_aggregate 3 the per group metrics must stay hidden, leaving only the ' + 'hostname aggregates published.') def _test_metrics_after_drain(self) -> None: """After traffic drains and a further sync tick passes, both live gauges must read 0. @@ -403,8 +416,8 @@ def _test_metrics_after_drain(self) -> None: 'per_server.current_connection.multi.origin.com 0', 'Once all connections close, the host aggregate current_connection must drain to 0.') tr.Processes.Default.Streams.All += Testers.ContainsExpression( - 'per_server.current_connection_max.multi.origin.com 0', - 'Once all connections close, current_connection_max must also come back down to 0: it ' + 'per_server.current_connection.max.multi.origin.com 0', + 'Once all connections close, current_connection.max must also come back down to 0: it ' 'is a live gauge, not a monotone peak.') def run(self) -> None: @@ -513,15 +526,15 @@ def run(self) -> None: class AggregateOnlyWithoutHostAggregateTest: """Verify metric_aggregate 2 still publishes per group metrics when there is no aggregate. - metric_aggregate 2 (AGGREGATE_ONLY) normally leaves the per group metrics hidden and publishes - only the per hostname aggregate. That aggregate exists only under match 'both', which is the - only match type with more than one group per hostname (Group::host_metric_name returns empty - for the others). With match 'port' there is therefore nothing for the aggregate to stand in - for, so the per group metrics have to be published regardless, or level 2 would report nothing - at all for this group. + metric_aggregate 2 (AGGREGATE_MAX) normally leaves the per group metrics hidden and publishes + only the per hostname max. That aggregate exists only under match 'both', which is the only + match type with more than one group per hostname (Group::host_metric_name returns empty for + the others). With match 'port' there is therefore nothing for the aggregate to stand in for, + so the per group metrics have to be published regardless, or level 2 would report nothing at + all for this group. - Every other test in this file that sets metric_aggregate 2 uses match 'both', so without this - case a regression that dropped the fallback would leave the suite green. + Every other test in this file that suppresses the per group metrics uses match 'both', so + without this case a regression that dropped the fallback would leave the suite green. """ def __init__(self) -> None: @@ -580,9 +593,125 @@ def run(self) -> None: self._test_metrics() +class AggregateRetractionTest: + """Verify that raising metric_aggregate to 2 withdraws already published per group metrics. + + metric_aggregate 2 (AGGREGATE_MAX) publishes the per hostname max and nothing else, so this + also covers that the hostname sums are not published at that level. + + metric_aggregate is dynamic, but the publication decision is made in the ConnectionTracker + Group constructor, and a published metric name is never removed from the metric store. Before + the store grew a tombstone, a name published while the setting was 0 kept reporting for the + life of the process no matter what the setting was changed to, which is exactly what was seen + in production: per group and per hostname metrics side by side at metric_aggregate 2. + + Origin keep alive is disabled so each request opens and closes its own upstream connection. + That returns the group count to zero, which erases the group, so the next request constructs a + fresh one and re-evaluates the setting. A group that never goes idle would keep whatever was in + effect when it was created. + """ + + def __init__(self) -> None: + """Configure the processes for the test.""" + self._dns = _dns + self._server = Test.MakeHttpBinServer("retract_server") + self._configure_trafficserver() + + def _configure_trafficserver(self) -> None: + """Configure an ATS that starts out publishing the per group metrics.""" + self._ts = Test.MakeATSProcess("retract_ts") + self._ts.Disk.records_config.update( + { + **_STAT_SYNC_RECORDS, + 'proxy.config.dns.nameservers': f"127.0.0.1:{self._dns.Variables.Port}", + 'proxy.config.dns.resolv_conf': 'NULL', + 'proxy.config.http.per_server.connection.metric_enabled': 1, + # Start with the per group metrics published, then raise it at runtime below. + 'proxy.config.http.per_server.connection.metric_aggregate': 0, + 'proxy.config.http.per_server.connection.match': 'both', + # Force the upstream connection closed after each transaction so the group is + # erased and the next request rebuilds it. + 'proxy.config.http.keep_alive_enabled_out': 0, + }) + self._ts.Disk.remap_config.AddLine( + f"map http://retract.origin.com/ http://retract.origin.com:{self._server.Variables.Port}/") + + def _curl(self, tr) -> None: + """Drive one request through the remap rule.""" + tr.MakeCurlCommand(f"-v --fail -s -x 127.0.0.1:{self._ts.Variables.port} 'http://retract.origin.com/get'", ts=self._ts) + tr.Processes.Default.ReturnCode = 0 + tr.StillRunningAfter = self._ts + + def run(self) -> None: + """Publish the per group metrics, raise the setting, then verify they are withdrawn.""" + tr = Test.AddTestRun("Drive traffic with the per group metrics published") + _use_shared_dns(tr) + tr.Processes.Default.StartBefore(self._server) + tr.Processes.Default.StartBefore(self._ts) + self._curl(tr) + + tr = Test.AddTestRun("Verify the per group metrics are published at metric_aggregate 0") + tr.Processes.Default.Command = f'sleep {_STAT_SYNC_WAIT_SECONDS}; traffic_ctl metric match per_server' + tr.Processes.Default.ReturnCode = 0 + tr.Processes.Default.Env = self._ts.Env + tr.Processes.Default.TimeOut = _STAT_SYNC_WAIT_SECONDS + 30 + tr.Processes.Default.Streams.All = Testers.ContainsExpression( + r'per_server\.current_connection\.retract\.origin\.com\.\d', + 'At metric_aggregate 0 the per group metric is published under its own name. Without ' + 'this the retraction below would be vacuous.') + tr.StillRunningAfter = self._ts + + tr = Test.AddTestRun("Raise metric_aggregate to 2") + # http_config_cb schedules the reconfigure a second after the record changes + # (HttpConfig.cc), so the new HttpConfigParams is not in place the instant traffic_ctl + # returns. Without this wait the next request is served by the old configuration and + # rebuilds the group under the old setting, which looks exactly like a failure to retract. + tr.Processes.Default.Command = ( + 'traffic_ctl config set proxy.config.http.per_server.connection.metric_aggregate 2 && ' + 'traffic_ctl config reload && ' + f'sleep {_CONFIG_APPLY_WAIT_SECONDS}') + tr.Processes.Default.ReturnCode = 0 + tr.Processes.Default.Env = self._ts.Env + tr.Processes.Default.TimeOut = _CONFIG_APPLY_WAIT_SECONDS + 30 + tr.StillRunningAfter = self._ts + + tr = Test.AddTestRun("Verify the new metric_aggregate is in effect") + tr.Processes.Default.Command = 'traffic_ctl config get proxy.config.http.per_server.connection.metric_aggregate' + tr.Processes.Default.ReturnCode = 0 + tr.Processes.Default.Env = self._ts.Env + tr.Processes.Default.TimeOut = 30 + tr.Processes.Default.Streams.All = Testers.ContainsExpression( + r'metric_aggregate: 2', 'The record must carry the new value before behavior is asserted against it.') + tr.StillRunningAfter = self._ts + + tr = Test.AddTestRun("Drive traffic again so the group is rebuilt under the new setting") + self._curl(tr) + + tr = Test.AddTestRun("Verify the per group metrics were withdrawn") + tr.Processes.Default.Command = f'sleep {_STAT_SYNC_WAIT_SECONDS}; traffic_ctl metric match per_server' + tr.Processes.Default.ReturnCode = 0 + tr.Processes.Default.Env = self._ts.Env + tr.Processes.Default.TimeOut = _STAT_SYNC_WAIT_SECONDS + 30 + tr.Processes.Default.Streams.All = Testers.ExcludesExpression( + r'per_server\.\w+_connection\.retract\.origin\.com\.\d', + 'Once metric_aggregate is 2 and the group has been rebuilt, the per group metrics must ' + 'no longer be published, even though they were published earlier in this process.') + tr.Processes.Default.Streams.All += Testers.ContainsExpression( + r'per_server\.current_connection\.max\.retract\.origin\.com', + 'The hostname max stands in for the withdrawn per group metrics.') + # metric_aggregate 2 is the max and nothing else, so the hostname sums must not appear + # either. '\.com ' with the trailing space matches the aggregate names, whose value follows + # the hostname directly; the max is 'current_connection.max.' and does not match. + tr.Processes.Default.Streams.All += Testers.ExcludesExpression( + r'per_server\.\w+_connection\.retract\.origin\.com ', + 'At metric_aggregate 2 only the max is published: the hostname sums must be absent.') + tr.StillRunningAfter = self._ts + + PerServerConnectionMaxTest().run() -ConnectMethodTest(3, metric_aggregate=2).run(blocked=2, gold_file="gold/two_503_congested.gold") +ConnectMethodTest(3, metric_aggregate=3).run(blocked=2, gold_file="gold/two_503_congested.gold") ConnectMethodTest(0, metric_aggregate=1).run(blocked=0, gold_file="gold/two_200_ok.gold") MultiGroupAggregateTest().run() MetricOverrideTest().run() AggregateOnlyWithoutHostAggregateTest().run() +AggregateRetractionTest().run() diff --git a/tests/gold_tests/pluginTest/compress/compress-vary-accept.config b/tests/gold_tests/pluginTest/compress/compress-vary-accept.config new file mode 100644 index 00000000000..34c9f1be12e --- /dev/null +++ b/tests/gold_tests/pluginTest/compress/compress-vary-accept.config @@ -0,0 +1,4 @@ +cache false +compressible-content-type text/* +supported-algorithms gzip +minimum-content-length 0 diff --git a/tests/gold_tests/pluginTest/compress/compress-vary-accept.test.py b/tests/gold_tests/pluginTest/compress/compress-vary-accept.test.py new file mode 100644 index 00000000000..02c95f5bdb8 --- /dev/null +++ b/tests/gold_tests/pluginTest/compress/compress-vary-accept.test.py @@ -0,0 +1,27 @@ +''' +Verify compress appends Accept-Encoding to an existing origin Vary header. +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +Test.Summary = ''' +Verify compress appends Accept-Encoding to an origin Vary header whose value is +a prefix of Accept-Encoding. +''' + +Test.SkipUnless(Condition.PluginExists('compress.so')) + +Test.ATSReplayTest(replay_file="replay/compress-vary-accept.replay.yaml") diff --git a/tests/gold_tests/pluginTest/compress/replay/compress-vary-accept.replay.yaml b/tests/gold_tests/pluginTest/compress/replay/compress-vary-accept.replay.yaml new file mode 100644 index 00000000000..a40d17a95a4 --- /dev/null +++ b/tests/gold_tests/pluginTest/compress/replay/compress-vary-accept.replay.yaml @@ -0,0 +1,117 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +meta: + version: "1.0" + + blocks: + - compressible-body: &compressible-body + content: + encoding: plain + size: 1024 + +autest: + description: 'Verify compress appends Accept-Encoding to an existing origin Vary header' + + server: + name: 'server' + + client: + name: 'client' + + ats: + name: 'ts' + + copy_to_config_dir: + - 'compress-vary-accept.config' + + records_config: + proxy.config.diags.debug.enabled: 1 + proxy.config.diags.debug.tags: 'compress' + + # The pparam resolves relative to the ATS config dir, where + # copy_to_config_dir places it. + remap_config: + - from: "http://compress.example/" + to: "http://127.0.0.1:{SERVER_HTTP_PORT}/" + plugins: + - name: 'compress.so' + args: ['compress-vary-accept.config'] + +sessions: +- transactions: + + ############################################################################# + # An origin Vary value that is a prefix of "Accept-Encoding" must not be + # mistaken for it: compress has to append Accept-Encoding alongside Accept. + ############################################################################# + - client-request: + method: "GET" + version: "1.1" + url: /vary-accept + headers: + fields: + - [ Host, compress.example ] + - [ Accept-Encoding, gzip ] + - [ uuid, vary-accept ] + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Type, text/javascript ] + - [ Vary, Accept ] + - [ Content-Length, 1024 ] + <<: *compressible-body + + proxy-response: + status: 200 + headers: + fields: + - [ Content-Encoding, { value: gzip, as: equal } ] + - [ Vary, { value: "Accept, Accept-Encoding", as: equal } ] + + ############################################################################# + # Control: an origin that already varies on Accept-Encoding is left alone, + # so the value is not duplicated. + ############################################################################# + - client-request: + method: "GET" + version: "1.1" + url: /vary-accept-encoding + headers: + fields: + - [ Host, compress.example ] + - [ Accept-Encoding, gzip ] + - [ uuid, vary-accept-encoding ] + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Type, text/javascript ] + - [ Vary, Accept-Encoding ] + - [ Content-Length, 1024 ] + <<: *compressible-body + + proxy-response: + status: 200 + headers: + fields: + - [ Content-Encoding, { value: gzip, as: equal } ] + - [ Vary, { value: "Accept-Encoding", as: equal } ] diff --git a/tests/gold_tests/slow_post/partial_post_client.py b/tests/gold_tests/slow_post/partial_post_client.py new file mode 100644 index 00000000000..fd964f6d7f6 --- /dev/null +++ b/tests/gold_tests/slow_post/partial_post_client.py @@ -0,0 +1,74 @@ +#!/usr/bin/env python3 +"""Send a partial POST to trigger the abort_tunnel code path. + +Sends POST headers claiming a large Content-Length but only sends a small +chunk of body data. When a request transform plugin is active, this causes +ATS to call abort_tunnel() while the transform entry is still in the vc_table. +""" + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import socket +import sys + + +def main() -> int: + """Run the client.""" + host = sys.argv[1] if len(sys.argv) > 1 else '127.0.0.1' + port = int(sys.argv[2]) if len(sys.argv) > 2 else 8080 + + request = ( + 'POST / HTTP/1.1\r\n' + 'Host: quick.server.com\r\n' + 'Content-Type: application/octet-stream\r\n' + 'Content-Length: 100000\r\n' + '\r\n').encode() + + partial_body = b'x' * 4096 + + sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + sock.settimeout(5) + try: + sock.connect((host, port)) + sock.sendall(request + partial_body) + print(f'Sent POST headers (Content-Length: 100000) + {len(partial_body)} bytes') + + try: + response = sock.recv(4096) + except ConnectionError: + print('Connection reset (expected for partial POST)') + return 0 + except socket.timeout: + print('ERROR: timeout waiting for response', file=sys.stderr) + return 1 + + if response: + first_line = response.split(b'\r\n')[0].decode(errors='replace') + print(first_line) + if first_line.startswith('HTTP/1.1'): + return 0 + print('ERROR: unexpected response', file=sys.stderr) + return 1 + else: + print('Connection closed (expected for partial POST)') + return 0 + finally: + sock.close() + + +if __name__ == '__main__': + sys.exit(main()) diff --git a/tests/gold_tests/slow_post/quick_server.test.py b/tests/gold_tests/slow_post/quick_server.test.py index fe250407987..b96ae5e92dd 100644 --- a/tests/gold_tests/slow_post/quick_server.test.py +++ b/tests/gold_tests/slow_post/quick_server.test.py @@ -1,4 +1,8 @@ -"""Verify ATS handles a server that replies before receiving a full request.""" +"""Verify ATS handles a server that replies before receiving a full request. + +Also verifies ATS does not leak the TransformVConnection when abort_tunnel() +is called with a request transform plugin active (use_request_transform=True). +""" # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file @@ -27,24 +31,27 @@ class QuickServerTest: """Verify that ATS doesn't delay responses behind slow posts.""" _init_file = '__init__.py' - _http_utils = 'http_utils.py' _slow_post_client = 'slow_post_client.py' + _partial_post_client = 'partial_post_client.py' _quick_server = 'quick_server.py' _dns_counter = 0 _server_counter = 0 _ts_counter = 0 - def __init__(self, abort_request: bool, drain_request: bool, abort_response_headers: bool): + def __init__(self, abort_request: bool, drain_request: bool, abort_response_headers: bool, use_request_transform: bool = False): """Initialize the test. :param drain_request: Whether the server should drain the request body. :param abort_request: Whether the client should abort the request body. - before disconnecting. + :param abort_response_headers: Whether the server should abort response headers. + :param use_request_transform: Whether to install a request transform plugin + hooked at TS_HTTP_READ_REQUEST_HDR_HOOK and use a partial POST client. """ self._should_drain_request = drain_request self._should_abort_request = abort_request self._should_abort_response_headers = abort_response_headers + self._use_request_transform = use_request_transform def _configure_dns(self, tr: 'TestRun') -> None: """Configure the DNS. @@ -89,13 +96,17 @@ def _configure_traffic_server(self, tr: 'TestRun'): 'proxy.config.dns.nameservers': f'127.0.0.1:{self._dns.Variables.Port}', 'proxy.config.dns.resolv_conf': 'NULL', }) + if self._use_request_transform: + Test.PrepareTestPlugin( + os.path.join(Test.Variables.AtsTestPluginsDir, 'tunnel_transform.so'), self._ts, plugin_args='request_hdr') def run(self): """Run the test.""" tr = Test.AddTestRun( f'Aborting request: {self._should_abort_request}, ' f'Draining request: {self._should_drain_request}, ' - f'Aborting response headers: {self._should_abort_response_headers}') + f'Aborting response headers: {self._should_abort_response_headers}, ' + f'Request transform: {self._use_request_transform}') self._configure_dns(tr) self._configure_server(tr) @@ -105,22 +116,30 @@ def run(self): http_utils = os.path.join(tools_dir, 'http_utils.py') tr.Setup.CopyAs(self._init_file, Test.RunDirectory) tr.Setup.CopyAs(http_utils, Test.RunDirectory) - tr.Setup.CopyAs(self._slow_post_client, Test.RunDirectory) tr.Setup.CopyAs(self._quick_server, Test.RunDirectory) - client_command = (f'{sys.executable} {self._slow_post_client} ' - '127.0.0.1 ' - f'{self._ts.Variables.port} ') - if not self._should_abort_request: - client_command += '--finish-request ' - p = tr.Processes.Default - p.Command = client_command - if self._should_abort_request or self._should_abort_response_headers: - p.Streams.All += Testers.ExcludesExpression('HTTP/1.1 200 OK', 'Verify response was received') + if self._use_request_transform: + tr.Setup.CopyAs(self._partial_post_client, Test.RunDirectory) + p = tr.Processes.Default + p.Command = (f'{sys.executable} {self._partial_post_client} ' + f'127.0.0.1 {self._ts.Variables.port}') + p.ReturnCode = 0 + p.Streams.All += Testers.ContainsExpression('HTTP/1.1 200 OK', 'Verify client received the server response') else: - p.Streams.All += Testers.ContainsExpression('HTTP/1.1 200 OK', 'Verify response was received') + tr.Setup.CopyAs(self._slow_post_client, Test.RunDirectory) + client_command = (f'{sys.executable} {self._slow_post_client} ' + '127.0.0.1 ' + f'{self._ts.Variables.port} ') + if not self._should_abort_request: + client_command += '--finish-request ' + p = tr.Processes.Default + p.Command = client_command + if self._should_abort_request or self._should_abort_response_headers: + p.Streams.All += Testers.ExcludesExpression('HTTP/1.1 200 OK', 'Verify response was received') + else: + p.Streams.All += Testers.ContainsExpression('HTTP/1.1 200 OK', 'Verify response was received') + p.ReturnCode = 0 - p.ReturnCode = 0 self._ts.StartBefore(self._dns) self._ts.StartBefore(self._server) p.StartBefore(self._ts) @@ -132,3 +151,10 @@ def run(self): for abort_response_headers in [True, False]: test = QuickServerTest(abort_request, drain_request, abort_response_headers) test.run() + +# Partial POST with a request transform plugin: exercises the abort_tunnel() +# cleanup path for TransformVConnection entries in the vc_table. +# Note: this is a resource leak (not a memory leak) — the VC pointer remains +# reachable via post_transform_info so ASAN/LSAN cannot detect a regression. +# This run is a "doesn't crash" smoke test; the fix itself is the guard. +QuickServerTest(abort_request=False, drain_request=False, abort_response_headers=False, use_request_transform=True).run() diff --git a/tests/tools/plugins/tunnel_transform.cc b/tests/tools/plugins/tunnel_transform.cc index 8b08acd70d6..cd1c454d473 100644 --- a/tests/tools/plugins/tunnel_transform.cc +++ b/tests/tools/plugins/tunnel_transform.cc @@ -1,6 +1,12 @@ /** @file - An example program that does a null transform of response body content. + Null transform plugin for request and/or response body content. + + Modes (selected by plugin argument): + (default) — hook at TS_HTTP_TUNNEL_START_HOOK, add both request + and response transforms (original behavior) + request_hdr — hook at TS_HTTP_READ_REQUEST_HDR_HOOK, add only the + request transform @section license License @@ -21,6 +27,7 @@ limitations under the License. */ +#include #include #include #include @@ -31,10 +38,11 @@ static const char PLUGIN_TAG[] = PLUGIN_NAME; static DbgCtl plugin_ctl{PLUGIN_TAG}; -static int stat_ua_bytes_sent = 0; // number of bytes seen by the transform from UA to OS -static int stat_os_bytes_sent = 0; // number of bytes seen by the transform from OS to UA -static int stat_error = 0; -static int stat_test_done = 0; +static int stat_ua_bytes_sent = 0; // number of bytes seen by the transform from UA to OS +static int stat_os_bytes_sent = 0; // number of bytes seen by the transform from OS to UA +static int stat_error = 0; +static int stat_test_done = 0; +static bool request_hdr_mode = false; // when true, hook at READ_REQUEST_HDR and request transform only typedef struct { TSVIO output_vio; @@ -99,7 +107,10 @@ handle_transform(TSCont contp, bool forward) data->output_buffer = TSIOBufferCreate(); data->output_reader = TSIOBufferReaderAlloc(data->output_buffer); Dbg(plugin_ctl, "\tWriting %" PRId64 " bytes on VConn", TSVIONBytesGet(input_vio)); - data->output_vio = TSVConnWrite(output_conn, contp, data->output_reader, INT64_MAX); + // A request transform must report a real content length — INT64_MAX + // makes state_request_wait_for_transform_read() fail the transaction. + int64_t nbytes = (request_hdr_mode && TSVIONBytesGet(input_vio) > 0) ? TSVIONBytesGet(input_vio) : INT64_MAX; + data->output_vio = TSVConnWrite(output_conn, contp, data->output_reader, nbytes); TSContDataSet(contp, data); } @@ -267,10 +278,12 @@ static void transform_add(TSHttpTxn txnp) { Dbg(plugin_ctl, "Entering transform_add()"); - TSVConn connp = TSTransformCreate(forward_null_transform, txnp); - TSVConn rev_connp = TSTransformCreate(reverse_null_transform, txnp); + TSVConn connp = TSTransformCreate(forward_null_transform, txnp); TSHttpTxnHookAdd(txnp, TS_HTTP_REQUEST_TRANSFORM_HOOK, connp); - TSHttpTxnHookAdd(txnp, TS_HTTP_RESPONSE_TRANSFORM_HOOK, rev_connp); + if (!request_hdr_mode) { + TSVConn rev_connp = TSTransformCreate(reverse_null_transform, txnp); + TSHttpTxnHookAdd(txnp, TS_HTTP_RESPONSE_TRANSFORM_HOOK, rev_connp); + } } static int @@ -280,8 +293,10 @@ transform_plugin(TSCont /* contp ATS_UNUSED */, TSEvent event, void *edata) Dbg(plugin_ctl, "Entering transform_plugin()"); switch (event) { + case TS_EVENT_HTTP_READ_REQUEST_HDR: case TS_EVENT_HTTP_TUNNEL_START: - Dbg(plugin_ctl, "\tEvent is TS_EVENT_HTTP_TUNNEL_START"); + Dbg(plugin_ctl, "\tEvent is %s", + event == TS_EVENT_HTTP_TUNNEL_START ? "TS_EVENT_HTTP_TUNNEL_START" : "TS_EVENT_HTTP_READ_REQUEST_HDR"); transform_add(txnp); TSHttpTxnReenable(txnp, TS_EVENT_HTTP_CONTINUE); return 0; @@ -301,7 +316,7 @@ handleMsg(TSCont /* cont ATS_UNUSED */, TSEvent event, void * /* edata ATS_UNUSE } void -TSPluginInit(int /* argc ATS_UNUSED */, const char ** /* argv ATS_UNUSED */) +TSPluginInit(int argc, const char **argv) { TSPluginRegistrationInfo info; @@ -315,6 +330,11 @@ TSPluginInit(int /* argc ATS_UNUSED */, const char ** /* argv ATS_UNUSED */) goto Lerror; } + if (argc > 1 && strcmp(argv[1], "request_hdr") == 0) { + request_hdr_mode = true; + } + Dbg(plugin_ctl, "mode: %s", request_hdr_mode ? "request_hdr" : "tunnel_start"); + stat_ua_bytes_sent = TSStatCreate("tunnel_transform.ua.bytes_sent", TS_RECORDDATATYPE_INT, TS_STAT_NON_PERSISTENT, TS_STAT_SYNC_SUM); stat_os_bytes_sent = @@ -322,7 +342,11 @@ TSPluginInit(int /* argc ATS_UNUSED */, const char ** /* argv ATS_UNUSED */) stat_error = TSStatCreate("tunnel_transform.error", TS_RECORDDATATYPE_INT, TS_STAT_NON_PERSISTENT, TS_STAT_SYNC_SUM); stat_test_done = TSStatCreate("tunnel_transform.test.done", TS_RECORDDATATYPE_INT, TS_STAT_NON_PERSISTENT, TS_STAT_SYNC_SUM); - TSHttpHookAdd(TS_HTTP_TUNNEL_START_HOOK, TSContCreate(transform_plugin, nullptr)); + if (request_hdr_mode) { + TSHttpHookAdd(TS_HTTP_READ_REQUEST_HDR_HOOK, TSContCreate(transform_plugin, nullptr)); + } else { + TSHttpHookAdd(TS_HTTP_TUNNEL_START_HOOK, TSContCreate(transform_plugin, nullptr)); + } TSLifecycleHookAdd(TS_LIFECYCLE_MSG_HOOK, TSContCreate(handleMsg, TSMutexCreate())); return; diff --git a/tools/changelog/changelog.py b/tools/changelog/changelog.py new file mode 100644 index 00000000000..f0794db3cbf --- /dev/null +++ b/tools/changelog/changelog.py @@ -0,0 +1,514 @@ +#!/usr/bin/env python3 +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Generate a changelog from merged PRs in a GitHub milestone. + +Usage: + uv run --project tools/changelog python tools/changelog/changelog.py \ + -o apache -r trafficserver -m 10.2.0 + +Output is written to stdout in the format used by CHANGELOG-* files: + + Changes with Apache Traffic Server 10.2.0 + #11945 - Make directory operations methods on `Directory` + #12026 - Static link opentelemetry-cpp libraries to otel_tracer plugin + ... + +To generate a changelog file for a release: + + uv run --project tools/changelog python tools/changelog/changelog.py \ + -o apache -r trafficserver -m 10.2.0 > CHANGELOG-10.2.0 + +Use --doc to include extra metadata (merge commit SHA, PR body, labels) for each +PR, useful for generating release documentation. With --from-git the body is the +commit message body rather than a PR body: + + uv run --project tools/changelog python tools/changelog/changelog.py \ + -o apache -r trafficserver -m 10.2.0 --doc --format yaml > changelog.yaml + +For security releases whose fixes land directly on a release branch without +public pull requests (and therefore never appear in a milestone), use +--from-git to source the changelog from a git commit range. This is merged +with the milestone PRs (deduplicated by PR number) so the result includes both +the direct-to-branch commits and any milestone PRs. The -m value is still used +as the version label in the output: + + uv run --project tools/changelog python tools/changelog/changelog.py \ + -o apache -r trafficserver -m 10.1.4 \ + --from-git 10.1.3..upstream/10.1.x + +Requires a GitHub token via GH_TOKEN env var or -a flag to avoid rate limits. +""" + +import argparse +import json +import os +import re +import subprocess +import sys + +import httpx +import yaml + +API_URL = "https://api.github.com" + + +def gh_cli_available() -> bool: + try: + subprocess.run(["gh", "--version"], capture_output=True, check=True) + return True + except (FileNotFoundError, subprocess.CalledProcessError): + return False + + +def changelog_via_gh(owner: str, repo: str, milestone: str, verbose: bool, doc: bool) -> list[dict]: + """Use the gh CLI to fetch milestone PRs (avoids API rate limits).""" + milestone_id = None + result = subprocess.run( + ["gh", "api", f"/repos/{owner}/{repo}/milestones?state=all", "--paginate"], + capture_output=True, + text=True, + ) + if result.returncode != 0: + print(f"gh api error: {result.stderr}", file=sys.stderr) + sys.exit(1) + + milestones = json.loads(result.stdout) + for ms in milestones: + if ms["title"] == milestone: + milestone_id = ms["number"] + break + + if milestone_id is None: + print(f"Milestone not found: {milestone}", file=sys.stderr) + sys.exit(1) + + print(f"Looking for issues from Milestone {milestone}", file=sys.stderr) + + changelog = [] + page = 1 + while True: + print(f"Page {page}", file=sys.stderr) + result = subprocess.run( + [ + "gh", + "api", + f"/repos/{owner}/{repo}/issues?milestone={milestone_id}&state=closed&page={page}&per_page=100", + ], + capture_output=True, + text=True, + ) + if result.returncode != 0: + print(f"gh api error: {result.stderr}", file=sys.stderr) + sys.exit(1) + + issues = json.loads(result.stdout) + if not issues: + break + + for issue in issues: + number = issue["number"] + title = issue["title"] + if verbose: + print(f"Issue #{number} - {title} ", end="", file=sys.stderr) + + if "pull_request" not in issue: + if verbose: + print("not a PR.", file=sys.stderr) + continue + + if not _gh_is_merged(owner, repo, number): + if verbose: + print("not merged.", file=sys.stderr) + continue + + if verbose: + print("added.", file=sys.stderr) + + entry: dict = {"number": number, "title": title} + if doc: + labels = [label["name"] for label in issue.get("labels", [])] + entry["labels"] = labels + pr_detail = subprocess.run( + ["gh", "api", f"/repos/{owner}/{repo}/pulls/{number}"], + capture_output=True, + text=True, + ) + if pr_detail.returncode != 0: + print(f"gh api error fetching PR #{number}: {pr_detail.stderr.strip()}", file=sys.stderr) + sys.exit(1) + pr_data = json.loads(pr_detail.stdout) + entry["sha"] = pr_data.get("merge_commit_sha", "") + entry["body"] = pr_data.get("body", "") or "" + changelog.append(entry) + + page += 1 + + return changelog + + +def changelog_via_api( + owner: str, + repo: str, + milestone: str, + token: str | None, + verbose: bool, + doc: bool, +) -> list[dict]: + """Use httpx to call the GitHub REST API directly.""" + headers = { + "Accept": "application/vnd.github.v3+json", + "User-Agent": "ATS-Changelog-Tool", + } + if token: + headers["Authorization"] = f"Bearer {token}" + + with httpx.Client(base_url=API_URL, headers=headers, timeout=30) as client: + milestone_id = _lookup_milestone(client, owner, repo, milestone) + if milestone_id is None: + print(f"Milestone not found: {milestone}", file=sys.stderr) + sys.exit(1) + + print(f"Looking for issues from Milestone {milestone}", file=sys.stderr) + + changelog = [] + page = 1 + while True: + print(f"Page {page}", file=sys.stderr) + resp = client.get( + f"/repos/{owner}/{repo}/issues", + params={ + "milestone": milestone_id, + "state": "closed", + "page": page, + "per_page": 100, + }, + ) + _check_rate_limit(resp) + resp.raise_for_status() + issues = resp.json() + + if not issues: + break + + for issue in issues: + number = issue["number"] + title = issue["title"] + if verbose: + print(f"Issue #{number} - {title} ", end="", file=sys.stderr) + + if "pull_request" not in issue: + if verbose: + print("not a PR.", file=sys.stderr) + continue + + if not _is_merged(client, owner, repo, number): + if verbose: + print("not merged.", file=sys.stderr) + continue + + if verbose: + print("added.", file=sys.stderr) + + entry: dict = {"number": number, "title": title} + if doc: + labels = [label["name"] for label in issue.get("labels", [])] + entry["labels"] = labels + pr_resp = client.get(f"/repos/{owner}/{repo}/pulls/{number}") + _check_rate_limit(pr_resp) + pr_resp.raise_for_status() + pr_data = pr_resp.json() + entry["sha"] = pr_data.get("merge_commit_sha", "") + entry["body"] = pr_data.get("body", "") or "" + changelog.append(entry) + + page += 1 + + return changelog + + +def changelog_via_git(git_range: str, verbose: bool, doc: bool) -> list[dict]: + """Build the changelog from a git commit range. + + Used for security releases whose fixes are committed directly to a release + branch without public PRs, so they never appear in a GitHub milestone. Each + commit becomes an entry; a trailing "(#N)" in the subject is captured as the + PR number when present, but is not required. + """ + field_sep = "\x1f" + record_sep = "\x1e" + fmt = f"%H{field_sep}%s{field_sep}%b{record_sep}" + result = subprocess.run( + ["git", "log", "--no-merges", "--reverse", f"--pretty=format:{fmt}", git_range], + capture_output=True, + text=True, + ) + if result.returncode != 0: + print(f"git log error: {result.stderr.strip()}", file=sys.stderr) + sys.exit(1) + + print(f"Reading commits from git range {git_range}", file=sys.stderr) + + changelog = [] + for record in result.stdout.split(record_sep): + record = record.strip("\n") + if not record: + continue + sha, subject, body = record.split(field_sep, 2) + # Only a trailing "(#N)" is a PR number. A bare "#N" anywhere in the subject + # is usually an issue reference, and treating it as a PR number would merge + # unrelated entries in merge_changelogs(). + match = re.search(r"\(#(\d+)\)$", subject) + number = int(match.group(1)) if match else None + if verbose: + label = f"#{number}" if number else sha[:12] + print(f"{label} - {subject} added.", file=sys.stderr) + + entry: dict = {"number": number, "title": subject} + if doc: + entry["sha"] = sha + entry["body"] = body.strip() + changelog.append(entry) + + return changelog + + +def merge_changelogs(milestone_entries: list[dict], git_entries: list[dict]) -> list[dict]: + """Union the milestone-sourced and git-sourced entries, deduplicated by PR number. + + Git commits keep their chronological order and carry commit-derived metadata. + Where a git commit's PR number matches a milestone PR, the milestone labels are + grafted onto the git entry. Milestone PRs with no corresponding commit in the + git range (e.g. folded into a combined backport) are appended, sorted by number. + """ + ms_by_number = {e["number"]: e for e in milestone_entries} + + # A PR can appear more than once in a git range -- a revert and reapply, or the + # same commit cherry-picked twice -- and both commits carry the same trailing + # "(#N)". Keep the first, so the surviving entry is the chronological one. + deduped: list[dict] = [] + seen: set[int] = set() + for ge in git_entries: + num = ge.get("number") + if num is not None: + if num in seen: + continue + seen.add(num) + deduped.append(ge) + + for ge in deduped: + num = ge.get("number") + if num in ms_by_number and ms_by_number[num].get("labels") and "labels" not in ge: + ge["labels"] = ms_by_number[num]["labels"] + + extras = [e for e in milestone_entries if e["number"] not in seen] + extras.sort(key=lambda x: x["number"]) + + return deduped + extras + + +def _lookup_milestone(client: httpx.Client, owner: str, repo: str, title: str) -> int | None: + page = 1 + while True: + resp = client.get(f"/repos/{owner}/{repo}/milestones", params={"state": "all", "per_page": 100, "page": page}) + _check_rate_limit(resp) + resp.raise_for_status() + data = resp.json() + if not data: + break + for ms in data: + if ms["title"] == title: + return ms["number"] + page += 1 + return None + + +def _http_status(response: str) -> int | None: + """Status code from the status line that `gh api --include` prints first.""" + match = re.match(r"HTTP/[\d.]+\s+(\d{3})", response.split("\n", 1)[0]) + return int(match.group(1)) if match else None + + +def _gh_is_merged(owner: str, repo: str, pr_number: int) -> bool: + """Whether pr_number is merged, via the gh CLI. + + gh exits non-zero for any HTTP error, so without the status line --include + supplies, a real 404 is indistinguishable from 403 rate limiting or a 5xx. + Anything but 204/404 is fatal; skipping would drop merged PRs and still exit 0. + """ + result = subprocess.run( + ["gh", "api", "--include", f"/repos/{owner}/{repo}/pulls/{pr_number}/merge"], + capture_output=True, + text=True, + ) + status = _http_status(result.stdout) + if status == 204: + return True + if status == 404: + return False + + print( + f"gh api error checking whether PR #{pr_number} is merged " + f"(HTTP {status if status else 'unknown'}): {result.stderr.strip()}", + file=sys.stderr, + ) + sys.exit(1) + + +def _is_merged(client: httpx.Client, owner: str, repo: str, pr_number: int) -> bool: + resp = client.get(f"/repos/{owner}/{repo}/pulls/{pr_number}/merge") + if resp.status_code == 204: + return True + if resp.status_code == 404: + return False + _check_rate_limit(resp) + resp.raise_for_status() + return False + + +def _check_rate_limit(resp: httpx.Response) -> None: + """Exit if @a resp is a rate-limit response, leaving other errors to the caller. + + A 403 alone does not identify a rate limit -- GitHub also uses it for a missing + token or insufficient scopes, where "try using an auth token" is wrong and hides + the real cause. A primary limit zeroes x-ratelimit-remaining, a secondary limit + sends retry-after, and a 429 is always a limit. Everything else falls through to + the raise_for_status() that follows each call site. + """ + if resp.status_code not in (403, 429): + return + if (resp.status_code == 429 or resp.headers.get("x-ratelimit-remaining") == "0" or "retry-after" in resp.headers): + print( + "You have exceeded your rate limit. Try using an auth token.", + file=sys.stderr, + ) + sys.exit(2) + + +def main() -> None: + parser = argparse.ArgumentParser(description="Generate changelog from merged PRs in a GitHub milestone.") + parser.add_argument("-o", "--owner", required=True, help="Repository owner") + parser.add_argument("-r", "--repo", required=True, help="Repository name") + parser.add_argument("-m", "--milestone", required=True, help="Milestone title") + parser.add_argument( + "-a", + "--auth", + default=None, + help="GitHub auth token. Discouraged: it is visible in ps output and shell history. Prefer GH_TOKEN.", + ) + parser.add_argument("-v", "--verbose", action="store_true", help="Verbose output") + parser.add_argument( + "--doc", + action="store_true", + help="Include extra metadata (merge SHA, PR body, labels) for documentation", + ) + parser.add_argument( + "--format", + choices=["text", "yaml"], + default="text", + help="Output format (default: text)", + ) + parser.add_argument( + "--use-gh", + action="store_true", + help="Use gh CLI instead of direct API calls (avoids rate limits)", + ) + parser.add_argument( + "--from-git", + metavar="RANGE", + default=None, + help="Also source commits from a git range (e.g. 10.1.3..upstream/10.1.x) and merge them " + "with the milestone PRs, deduplicated by PR number. Use for security releases whose fixes " + "land directly on a branch without public PRs. -m is still used as the version label.", + ) + args = parser.parse_args() + + token = args.auth or os.environ.get("GH_TOKEN") + + if not token and not args.use_gh: + print( + "WARNING: No GitHub token provided. Unauthenticated requests are limited\n" + "to 60 per hour, which is usually not enough to generate a full changelog.\n" + "\n" + "Provide a token via -a or the GH_TOKEN environment variable.\n" + "\n" + "To create a token:\n" + " 1. Go to https://github.com/settings/tokens\n" + " 2. Click 'Generate new token' -> 'Generate new token (classic)'\n" + " 3. Select the 'public_repo' scope (sufficient for public repositories)\n" + " 4. For fine-grained tokens, grant read-only access to Issues and\n" + " Pull Requests on the target repository\n" + "\n" + "Alternatively, use --use-gh to use the gh CLI with its existing auth.\n", + file=sys.stderr, + ) + + if args.use_gh: + if not gh_cli_available(): + print("gh CLI not found. Install it or omit --use-gh.", file=sys.stderr) + sys.exit(1) + changelog = changelog_via_gh(args.owner, args.repo, args.milestone, args.verbose, args.doc) + else: + changelog = changelog_via_api( + args.owner, + args.repo, + args.milestone, + token, + args.verbose, + args.doc, + ) + + if args.from_git: + git_changelog = changelog_via_git(args.from_git, args.verbose, args.doc) + changelog = merge_changelogs(changelog, git_changelog) + + if changelog: + # Milestone-only entries all carry a PR number and sort by it. A merged + # (--from-git) result preserves git's chronological order with milestone-only + # PRs appended, so it must not be re-sorted. + if not args.from_git: + changelog.sort(key=lambda x: x["number"]) + + if args.format == "yaml": + output = { + "milestone": args.milestone, + "owner": args.owner, + "repo": args.repo, + "entries": changelog, + } + yaml.safe_dump(output, sys.stdout, default_flow_style=False, sort_keys=False, allow_unicode=True) + else: + print(f"Changes with Apache Traffic Server {args.milestone}") + for entry in changelog: + if entry.get("number"): + print(f" #{entry['number']} - {entry['title']}") + else: + print(f" {entry['title']}") + if args.doc: + if entry.get("sha"): + print(f" SHA: {entry['sha']}") + if entry.get("labels"): + print(f" Labels: {', '.join(entry['labels'])}") + if entry.get("body"): + body_lines = entry["body"].strip().split("\n") + print(f" Body:") + for line in body_lines: + print(f" {line}") + + +if __name__ == "__main__": + main() diff --git a/tools/changelog/pyproject.toml b/tools/changelog/pyproject.toml new file mode 100644 index 00000000000..87dc6a4daec --- /dev/null +++ b/tools/changelog/pyproject.toml @@ -0,0 +1,30 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[project] +name = "changelog" +version = "0.1.0" +description = "Generate changelog from GitHub milestones for Apache Traffic Server" +requires-python = ">=3.11" +license = "Apache-2.0" +dependencies = [ + "httpx>=0.27", + "pyyaml>=6.0", +] + +# No [build-system] and no [project.scripts]: this is a uv virtual project, used +# only to pin the dependencies for the in-tree script. Run it as +# `uv run --project tools/changelog python tools/changelog/changelog.py`. diff --git a/tools/changelog/uv.lock b/tools/changelog/uv.lock new file mode 100644 index 00000000000..a08778737c9 --- /dev/null +++ b/tools/changelog/uv.lock @@ -0,0 +1,150 @@ +version = 1 +revision = 3 +requires-python = ">=3.11" + +[[package]] +name = "anyio" +version = "4.13.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "idna" }, + { name = "typing-extensions", marker = "python_full_version < '3.13'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/19/14/2c5dd9f512b66549ae92767a9c7b330ae88e1932ca57876909410251fe13/anyio-4.13.0.tar.gz", hash = "sha256:334b70e641fd2221c1505b3890c69882fe4a2df910cba14d97019b90b24439dc", size = 231622, upload-time = "2026-03-24T12:59:09.671Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/da/42/e921fccf5015463e32a3cf6ee7f980a6ed0f395ceeaa45060b61d86486c2/anyio-4.13.0-py3-none-any.whl", hash = "sha256:08b310f9e24a9594186fd75b4f73f4a4152069e3853f1ed8bfbf58369f4ad708", size = 114353, upload-time = "2026-03-24T12:59:08.246Z" }, +] + +[[package]] +name = "certifi" +version = "2026.2.25" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/af/2d/7bf41579a8986e348fa033a31cdd0e4121114f6bce2457e8876010b092dd/certifi-2026.2.25.tar.gz", hash = "sha256:e887ab5cee78ea814d3472169153c2d12cd43b14bd03329a39a9c6e2e80bfba7", size = 155029, upload-time = "2026-02-25T02:54:17.342Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/9a/3c/c17fb3ca2d9c3acff52e30b309f538586f9f5b9c9cf454f3845fc9af4881/certifi-2026.2.25-py3-none-any.whl", hash = "sha256:027692e4402ad994f1c42e52a4997a9763c646b73e4096e4d5d6db8af1d6f0fa", size = 153684, upload-time = "2026-02-25T02:54:15.766Z" }, +] + +[[package]] +name = "changelog" +version = "0.1.0" +source = { virtual = "." } +dependencies = [ + { name = "httpx" }, + { name = "pyyaml" }, +] + +[package.metadata] +requires-dist = [ + { name = "httpx", specifier = ">=0.27" }, + { name = "pyyaml", specifier = ">=6.0" }, +] + +[[package]] +name = "h11" +version = "0.16.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/01/ee/02a2c011bdab74c6fb3c75474d40b3052059d95df7e73351460c8588d963/h11-0.16.0.tar.gz", hash = "sha256:4e35b956cf45792e4caa5885e69fba00bdbc6ffafbfa020300e549b208ee5ff1", size = 101250, upload-time = "2025-04-24T03:35:25.427Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/04/4b/29cac41a4d98d144bf5f6d33995617b185d14b22401f75ca86f384e87ff1/h11-0.16.0-py3-none-any.whl", hash = "sha256:63cf8bbe7522de3bf65932fda1d9c2772064ffb3dae62d55932da54b31cb6c86", size = 37515, upload-time = "2025-04-24T03:35:24.344Z" }, +] + +[[package]] +name = "httpcore" +version = "1.0.9" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "certifi" }, + { name = "h11" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/06/94/82699a10bca87a5556c9c59b5963f2d039dbd239f25bc2a63907a05a14cb/httpcore-1.0.9.tar.gz", hash = "sha256:6e34463af53fd2ab5d807f399a9b45ea31c3dfa2276f15a2c3f00afff6e176e8", size = 85484, upload-time = "2025-04-24T22:06:22.219Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7e/f5/f66802a942d491edb555dd61e3a9961140fd64c90bce1eafd741609d334d/httpcore-1.0.9-py3-none-any.whl", hash = "sha256:2d400746a40668fc9dec9810239072b40b4484b640a8c38fd654a024c7a1bf55", size = 78784, upload-time = "2025-04-24T22:06:20.566Z" }, +] + +[[package]] +name = "httpx" +version = "0.28.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "anyio" }, + { name = "certifi" }, + { name = "httpcore" }, + { name = "idna" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/b1/df/48c586a5fe32a0f01324ee087459e112ebb7224f646c0b5023f5e79e9956/httpx-0.28.1.tar.gz", hash = "sha256:75e98c5f16b0f35b567856f597f06ff2270a374470a5c2392242528e3e3e42fc", size = 141406, upload-time = "2024-12-06T15:37:23.222Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/2a/39/e50c7c3a983047577ee07d2a9e53faf5a69493943ec3f6a384bdc792deb2/httpx-0.28.1-py3-none-any.whl", hash = "sha256:d909fcccc110f8c7faf814ca82a9a4d816bc5a6dbfea25d6591d6985b8ba59ad", size = 73517, upload-time = "2024-12-06T15:37:21.509Z" }, +] + +[[package]] +name = "idna" +version = "3.11" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/6f/6d/0703ccc57f3a7233505399edb88de3cbd678da106337b9fcde432b65ed60/idna-3.11.tar.gz", hash = "sha256:795dafcc9c04ed0c1fb032c2aa73654d8e8c5023a7df64a53f39190ada629902", size = 194582, upload-time = "2025-10-12T14:55:20.501Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/0e/61/66938bbb5fc52dbdf84594873d5b51fb1f7c7794e9c0f5bd885f30bc507b/idna-3.11-py3-none-any.whl", hash = "sha256:771a87f49d9defaf64091e6e6fe9c18d4833f140bd19464795bc32d966ca37ea", size = 71008, upload-time = "2025-10-12T14:55:18.883Z" }, +] + +[[package]] +name = "pyyaml" +version = "6.0.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/05/8e/961c0007c59b8dd7729d542c61a4d537767a59645b82a0b521206e1e25c2/pyyaml-6.0.3.tar.gz", hash = "sha256:d76623373421df22fb4cf8817020cbb7ef15c725b9d5e45f17e189bfc384190f", size = 130960, upload-time = "2025-09-25T21:33:16.546Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/6d/16/a95b6757765b7b031c9374925bb718d55e0a9ba8a1b6a12d25962ea44347/pyyaml-6.0.3-cp311-cp311-macosx_10_13_x86_64.whl", hash = "sha256:44edc647873928551a01e7a563d7452ccdebee747728c1080d881d68af7b997e", size = 185826, upload-time = "2025-09-25T21:31:58.655Z" }, + { url = "https://files.pythonhosted.org/packages/16/19/13de8e4377ed53079ee996e1ab0a9c33ec2faf808a4647b7b4c0d46dd239/pyyaml-6.0.3-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:652cb6edd41e718550aad172851962662ff2681490a8a711af6a4d288dd96824", size = 175577, upload-time = "2025-09-25T21:32:00.088Z" }, + { url = "https://files.pythonhosted.org/packages/0c/62/d2eb46264d4b157dae1275b573017abec435397aa59cbcdab6fc978a8af4/pyyaml-6.0.3-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:10892704fc220243f5305762e276552a0395f7beb4dbf9b14ec8fd43b57f126c", size = 775556, upload-time = "2025-09-25T21:32:01.31Z" }, + { url = "https://files.pythonhosted.org/packages/10/cb/16c3f2cf3266edd25aaa00d6c4350381c8b012ed6f5276675b9eba8d9ff4/pyyaml-6.0.3-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:850774a7879607d3a6f50d36d04f00ee69e7fc816450e5f7e58d7f17f1ae5c00", size = 882114, upload-time = "2025-09-25T21:32:03.376Z" }, + { url = "https://files.pythonhosted.org/packages/71/60/917329f640924b18ff085ab889a11c763e0b573da888e8404ff486657602/pyyaml-6.0.3-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b8bb0864c5a28024fac8a632c443c87c5aa6f215c0b126c449ae1a150412f31d", size = 806638, upload-time = "2025-09-25T21:32:04.553Z" }, + { url = "https://files.pythonhosted.org/packages/dd/6f/529b0f316a9fd167281a6c3826b5583e6192dba792dd55e3203d3f8e655a/pyyaml-6.0.3-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:1d37d57ad971609cf3c53ba6a7e365e40660e3be0e5175fa9f2365a379d6095a", size = 767463, upload-time = "2025-09-25T21:32:06.152Z" }, + { url = "https://files.pythonhosted.org/packages/f2/6a/b627b4e0c1dd03718543519ffb2f1deea4a1e6d42fbab8021936a4d22589/pyyaml-6.0.3-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:37503bfbfc9d2c40b344d06b2199cf0e96e97957ab1c1b546fd4f87e53e5d3e4", size = 794986, upload-time = "2025-09-25T21:32:07.367Z" }, + { url = "https://files.pythonhosted.org/packages/45/91/47a6e1c42d9ee337c4839208f30d9f09caa9f720ec7582917b264defc875/pyyaml-6.0.3-cp311-cp311-win32.whl", hash = "sha256:8098f252adfa6c80ab48096053f512f2321f0b998f98150cea9bd23d83e1467b", size = 142543, upload-time = "2025-09-25T21:32:08.95Z" }, + { url = "https://files.pythonhosted.org/packages/da/e3/ea007450a105ae919a72393cb06f122f288ef60bba2dc64b26e2646fa315/pyyaml-6.0.3-cp311-cp311-win_amd64.whl", hash = "sha256:9f3bfb4965eb874431221a3ff3fdcddc7e74e3b07799e0e84ca4a0f867d449bf", size = 158763, upload-time = "2025-09-25T21:32:09.96Z" }, + { url = "https://files.pythonhosted.org/packages/d1/33/422b98d2195232ca1826284a76852ad5a86fe23e31b009c9886b2d0fb8b2/pyyaml-6.0.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:7f047e29dcae44602496db43be01ad42fc6f1cc0d8cd6c83d342306c32270196", size = 182063, upload-time = "2025-09-25T21:32:11.445Z" }, + { url = "https://files.pythonhosted.org/packages/89/a0/6cf41a19a1f2f3feab0e9c0b74134aa2ce6849093d5517a0c550fe37a648/pyyaml-6.0.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:fc09d0aa354569bc501d4e787133afc08552722d3ab34836a80547331bb5d4a0", size = 173973, upload-time = "2025-09-25T21:32:12.492Z" }, + { url = "https://files.pythonhosted.org/packages/ed/23/7a778b6bd0b9a8039df8b1b1d80e2e2ad78aa04171592c8a5c43a56a6af4/pyyaml-6.0.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9149cad251584d5fb4981be1ecde53a1ca46c891a79788c0df828d2f166bda28", size = 775116, upload-time = "2025-09-25T21:32:13.652Z" }, + { url = "https://files.pythonhosted.org/packages/65/30/d7353c338e12baef4ecc1b09e877c1970bd3382789c159b4f89d6a70dc09/pyyaml-6.0.3-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:5fdec68f91a0c6739b380c83b951e2c72ac0197ace422360e6d5a959d8d97b2c", size = 844011, upload-time = "2025-09-25T21:32:15.21Z" }, + { url = "https://files.pythonhosted.org/packages/8b/9d/b3589d3877982d4f2329302ef98a8026e7f4443c765c46cfecc8858c6b4b/pyyaml-6.0.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ba1cc08a7ccde2d2ec775841541641e4548226580ab850948cbfda66a1befcdc", size = 807870, upload-time = "2025-09-25T21:32:16.431Z" }, + { url = "https://files.pythonhosted.org/packages/05/c0/b3be26a015601b822b97d9149ff8cb5ead58c66f981e04fedf4e762f4bd4/pyyaml-6.0.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:8dc52c23056b9ddd46818a57b78404882310fb473d63f17b07d5c40421e47f8e", size = 761089, upload-time = "2025-09-25T21:32:17.56Z" }, + { url = "https://files.pythonhosted.org/packages/be/8e/98435a21d1d4b46590d5459a22d88128103f8da4c2d4cb8f14f2a96504e1/pyyaml-6.0.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:41715c910c881bc081f1e8872880d3c650acf13dfa8214bad49ed4cede7c34ea", size = 790181, upload-time = "2025-09-25T21:32:18.834Z" }, + { url = "https://files.pythonhosted.org/packages/74/93/7baea19427dcfbe1e5a372d81473250b379f04b1bd3c4c5ff825e2327202/pyyaml-6.0.3-cp312-cp312-win32.whl", hash = "sha256:96b533f0e99f6579b3d4d4995707cf36df9100d67e0c8303a0c55b27b5f99bc5", size = 137658, upload-time = "2025-09-25T21:32:20.209Z" }, + { url = "https://files.pythonhosted.org/packages/86/bf/899e81e4cce32febab4fb42bb97dcdf66bc135272882d1987881a4b519e9/pyyaml-6.0.3-cp312-cp312-win_amd64.whl", hash = "sha256:5fcd34e47f6e0b794d17de1b4ff496c00986e1c83f7ab2fb8fcfe9616ff7477b", size = 154003, upload-time = "2025-09-25T21:32:21.167Z" }, + { url = "https://files.pythonhosted.org/packages/1a/08/67bd04656199bbb51dbed1439b7f27601dfb576fb864099c7ef0c3e55531/pyyaml-6.0.3-cp312-cp312-win_arm64.whl", hash = "sha256:64386e5e707d03a7e172c0701abfb7e10f0fb753ee1d773128192742712a98fd", size = 140344, upload-time = "2025-09-25T21:32:22.617Z" }, + { url = "https://files.pythonhosted.org/packages/d1/11/0fd08f8192109f7169db964b5707a2f1e8b745d4e239b784a5a1dd80d1db/pyyaml-6.0.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:8da9669d359f02c0b91ccc01cac4a67f16afec0dac22c2ad09f46bee0697eba8", size = 181669, upload-time = "2025-09-25T21:32:23.673Z" }, + { url = "https://files.pythonhosted.org/packages/b1/16/95309993f1d3748cd644e02e38b75d50cbc0d9561d21f390a76242ce073f/pyyaml-6.0.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:2283a07e2c21a2aa78d9c4442724ec1eb15f5e42a723b99cb3d822d48f5f7ad1", size = 173252, upload-time = "2025-09-25T21:32:25.149Z" }, + { url = "https://files.pythonhosted.org/packages/50/31/b20f376d3f810b9b2371e72ef5adb33879b25edb7a6d072cb7ca0c486398/pyyaml-6.0.3-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ee2922902c45ae8ccada2c5b501ab86c36525b883eff4255313a253a3160861c", size = 767081, upload-time = "2025-09-25T21:32:26.575Z" }, + { url = "https://files.pythonhosted.org/packages/49/1e/a55ca81e949270d5d4432fbbd19dfea5321eda7c41a849d443dc92fd1ff7/pyyaml-6.0.3-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a33284e20b78bd4a18c8c2282d549d10bc8408a2a7ff57653c0cf0b9be0afce5", size = 841159, upload-time = "2025-09-25T21:32:27.727Z" }, + { url = "https://files.pythonhosted.org/packages/74/27/e5b8f34d02d9995b80abcef563ea1f8b56d20134d8f4e5e81733b1feceb2/pyyaml-6.0.3-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0f29edc409a6392443abf94b9cf89ce99889a1dd5376d94316ae5145dfedd5d6", size = 801626, upload-time = "2025-09-25T21:32:28.878Z" }, + { url = "https://files.pythonhosted.org/packages/f9/11/ba845c23988798f40e52ba45f34849aa8a1f2d4af4b798588010792ebad6/pyyaml-6.0.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:f7057c9a337546edc7973c0d3ba84ddcdf0daa14533c2065749c9075001090e6", size = 753613, upload-time = "2025-09-25T21:32:30.178Z" }, + { url = "https://files.pythonhosted.org/packages/3d/e0/7966e1a7bfc0a45bf0a7fb6b98ea03fc9b8d84fa7f2229e9659680b69ee3/pyyaml-6.0.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:eda16858a3cab07b80edaf74336ece1f986ba330fdb8ee0d6c0d68fe82bc96be", size = 794115, upload-time = "2025-09-25T21:32:31.353Z" }, + { url = "https://files.pythonhosted.org/packages/de/94/980b50a6531b3019e45ddeada0626d45fa85cbe22300844a7983285bed3b/pyyaml-6.0.3-cp313-cp313-win32.whl", hash = "sha256:d0eae10f8159e8fdad514efdc92d74fd8d682c933a6dd088030f3834bc8e6b26", size = 137427, upload-time = "2025-09-25T21:32:32.58Z" }, + { url = "https://files.pythonhosted.org/packages/97/c9/39d5b874e8b28845e4ec2202b5da735d0199dbe5b8fb85f91398814a9a46/pyyaml-6.0.3-cp313-cp313-win_amd64.whl", hash = "sha256:79005a0d97d5ddabfeeea4cf676af11e647e41d81c9a7722a193022accdb6b7c", size = 154090, upload-time = "2025-09-25T21:32:33.659Z" }, + { url = "https://files.pythonhosted.org/packages/73/e8/2bdf3ca2090f68bb3d75b44da7bbc71843b19c9f2b9cb9b0f4ab7a5a4329/pyyaml-6.0.3-cp313-cp313-win_arm64.whl", hash = "sha256:5498cd1645aa724a7c71c8f378eb29ebe23da2fc0d7a08071d89469bf1d2defb", size = 140246, upload-time = "2025-09-25T21:32:34.663Z" }, + { url = "https://files.pythonhosted.org/packages/9d/8c/f4bd7f6465179953d3ac9bc44ac1a8a3e6122cf8ada906b4f96c60172d43/pyyaml-6.0.3-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:8d1fab6bb153a416f9aeb4b8763bc0f22a5586065f86f7664fc23339fc1c1fac", size = 181814, upload-time = "2025-09-25T21:32:35.712Z" }, + { url = "https://files.pythonhosted.org/packages/bd/9c/4d95bb87eb2063d20db7b60faa3840c1b18025517ae857371c4dd55a6b3a/pyyaml-6.0.3-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:34d5fcd24b8445fadc33f9cf348c1047101756fd760b4dacb5c3e99755703310", size = 173809, upload-time = "2025-09-25T21:32:36.789Z" }, + { url = "https://files.pythonhosted.org/packages/92/b5/47e807c2623074914e29dabd16cbbdd4bf5e9b2db9f8090fa64411fc5382/pyyaml-6.0.3-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:501a031947e3a9025ed4405a168e6ef5ae3126c59f90ce0cd6f2bfc477be31b7", size = 766454, upload-time = "2025-09-25T21:32:37.966Z" }, + { url = "https://files.pythonhosted.org/packages/02/9e/e5e9b168be58564121efb3de6859c452fccde0ab093d8438905899a3a483/pyyaml-6.0.3-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:b3bc83488de33889877a0f2543ade9f70c67d66d9ebb4ac959502e12de895788", size = 836355, upload-time = "2025-09-25T21:32:39.178Z" }, + { url = "https://files.pythonhosted.org/packages/88/f9/16491d7ed2a919954993e48aa941b200f38040928474c9e85ea9e64222c3/pyyaml-6.0.3-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c458b6d084f9b935061bc36216e8a69a7e293a2f1e68bf956dcd9e6cbcd143f5", size = 794175, upload-time = "2025-09-25T21:32:40.865Z" }, + { url = "https://files.pythonhosted.org/packages/dd/3f/5989debef34dc6397317802b527dbbafb2b4760878a53d4166579111411e/pyyaml-6.0.3-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:7c6610def4f163542a622a73fb39f534f8c101d690126992300bf3207eab9764", size = 755228, upload-time = "2025-09-25T21:32:42.084Z" }, + { url = "https://files.pythonhosted.org/packages/d7/ce/af88a49043cd2e265be63d083fc75b27b6ed062f5f9fd6cdc223ad62f03e/pyyaml-6.0.3-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:5190d403f121660ce8d1d2c1bb2ef1bd05b5f68533fc5c2ea899bd15f4399b35", size = 789194, upload-time = "2025-09-25T21:32:43.362Z" }, + { url = "https://files.pythonhosted.org/packages/23/20/bb6982b26a40bb43951265ba29d4c246ef0ff59c9fdcdf0ed04e0687de4d/pyyaml-6.0.3-cp314-cp314-win_amd64.whl", hash = "sha256:4a2e8cebe2ff6ab7d1050ecd59c25d4c8bd7e6f400f5f82b96557ac0abafd0ac", size = 156429, upload-time = "2025-09-25T21:32:57.844Z" }, + { url = "https://files.pythonhosted.org/packages/f4/f4/a4541072bb9422c8a883ab55255f918fa378ecf083f5b85e87fc2b4eda1b/pyyaml-6.0.3-cp314-cp314-win_arm64.whl", hash = "sha256:93dda82c9c22deb0a405ea4dc5f2d0cda384168e466364dec6255b293923b2f3", size = 143912, upload-time = "2025-09-25T21:32:59.247Z" }, + { url = "https://files.pythonhosted.org/packages/7c/f9/07dd09ae774e4616edf6cda684ee78f97777bdd15847253637a6f052a62f/pyyaml-6.0.3-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:02893d100e99e03eda1c8fd5c441d8c60103fd175728e23e431db1b589cf5ab3", size = 189108, upload-time = "2025-09-25T21:32:44.377Z" }, + { url = "https://files.pythonhosted.org/packages/4e/78/8d08c9fb7ce09ad8c38ad533c1191cf27f7ae1effe5bb9400a46d9437fcf/pyyaml-6.0.3-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:c1ff362665ae507275af2853520967820d9124984e0f7466736aea23d8611fba", size = 183641, upload-time = "2025-09-25T21:32:45.407Z" }, + { url = "https://files.pythonhosted.org/packages/7b/5b/3babb19104a46945cf816d047db2788bcaf8c94527a805610b0289a01c6b/pyyaml-6.0.3-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6adc77889b628398debc7b65c073bcb99c4a0237b248cacaf3fe8a557563ef6c", size = 831901, upload-time = "2025-09-25T21:32:48.83Z" }, + { url = "https://files.pythonhosted.org/packages/8b/cc/dff0684d8dc44da4d22a13f35f073d558c268780ce3c6ba1b87055bb0b87/pyyaml-6.0.3-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a80cb027f6b349846a3bf6d73b5e95e782175e52f22108cfa17876aaeff93702", size = 861132, upload-time = "2025-09-25T21:32:50.149Z" }, + { url = "https://files.pythonhosted.org/packages/b1/5e/f77dc6b9036943e285ba76b49e118d9ea929885becb0a29ba8a7c75e29fe/pyyaml-6.0.3-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:00c4bdeba853cc34e7dd471f16b4114f4162dc03e6b7afcc2128711f0eca823c", size = 839261, upload-time = "2025-09-25T21:32:51.808Z" }, + { url = "https://files.pythonhosted.org/packages/ce/88/a9db1376aa2a228197c58b37302f284b5617f56a5d959fd1763fb1675ce6/pyyaml-6.0.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:66e1674c3ef6f541c35191caae2d429b967b99e02040f5ba928632d9a7f0f065", size = 805272, upload-time = "2025-09-25T21:32:52.941Z" }, + { url = "https://files.pythonhosted.org/packages/da/92/1446574745d74df0c92e6aa4a7b0b3130706a4142b2d1a5869f2eaa423c6/pyyaml-6.0.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:16249ee61e95f858e83976573de0f5b2893b3677ba71c9dd36b9cf8be9ac6d65", size = 829923, upload-time = "2025-09-25T21:32:54.537Z" }, + { url = "https://files.pythonhosted.org/packages/f0/7a/1c7270340330e575b92f397352af856a8c06f230aa3e76f86b39d01b416a/pyyaml-6.0.3-cp314-cp314t-win_amd64.whl", hash = "sha256:4ad1906908f2f5ae4e5a8ddfce73c320c2a1429ec52eafd27138b7f1cbe341c9", size = 174062, upload-time = "2025-09-25T21:32:55.767Z" }, + { url = "https://files.pythonhosted.org/packages/f1/12/de94a39c2ef588c7e6455cfbe7343d3b2dc9d6b6b2f40c4c6565744c873d/pyyaml-6.0.3-cp314-cp314t-win_arm64.whl", hash = "sha256:ebc55a14a21cb14062aa4162f906cd962b28e2e9ea38f9b4391244cd8de4ae0b", size = 149341, upload-time = "2025-09-25T21:32:56.828Z" }, +] + +[[package]] +name = "typing-extensions" +version = "4.15.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/72/94/1a15dd82efb362ac84269196e94cf00f187f7ed21c242792a923cdb1c61f/typing_extensions-4.15.0.tar.gz", hash = "sha256:0cea48d173cc12fa28ecabc3b837ea3cf6f38c6d1136f85cbaaf598984861466", size = 109391, upload-time = "2025-08-25T13:49:26.313Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/18/67/36e9267722cc04a6b9f15c7f3441c2363321a3ea07da7ae0c0707beb2a9c/typing_extensions-4.15.0-py3-none-any.whl", hash = "sha256:f0fa19c6845758ab08074a0cfa8b7aecb71c999ca73d62883bc25cc018c4e548", size = 44614, upload-time = "2025-08-25T13:49:24.86Z" }, +] diff --git a/tools/git/changelog.pl b/tools/git/changelog.pl deleted file mode 100755 index a76632ae6bd..00000000000 --- a/tools/git/changelog.pl +++ /dev/null @@ -1,174 +0,0 @@ -#!/usr/bin/env perl -# -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -use strict; -use warnings; - -use WWW::Curl::Easy; -use JSON; -use Getopt::Std; - -my $url = "https://api.github.com"; - -sub usage() { - print "USAGE: changelog.pl -o owner -r repo -m milestone [-a auth] [-v]\n"; - print "\t-v verbose\n\n"; - print "Example:\n"; - print "\tchangelog.pl -o apache -r trafficserver -m 8.0.6 -a auth_token\n"; - exit(1); -} - -sub rate_fail -{ - print STDERR "You have exceeded your rate limit. Try using an auth token.\n"; - exit 2; -} - -sub milestone_lookup($$$$$) -{ - my($curl, $url, $owner, $repo, $milestone_title) = @_; - - my $endpoint = "/repos/$owner/$repo/milestones"; - my $resp_body; - - $curl->setopt(CURLOPT_WRITEDATA, \$resp_body); - $curl->setopt(CURLOPT_URL, $url . $endpoint); - - my $retcode = $curl->perform(); - if ($retcode == 0 && $curl->getinfo(CURLINFO_HTTP_CODE) == 200) { - my $milestones = from_json($resp_body); - foreach my $milestone (@{$milestones}) { - if ($milestone->{title} eq $milestone_title) { - return $milestone->{number}; - } - } - } elsif ($retcode == 0 && $curl->getinfo(CURLINFO_HTTP_CODE) == 403) { - rate_fail(); - } - - undef; -} - -sub is_merged($$$$$) -{ - my($curl, $url, $owner, $repo, $issue_id) = @_; - - my $endpoint = "/repos/$owner/$repo/pulls/$issue_id/merge"; - my $resp_body; - - $curl->setopt(CURLOPT_WRITEDATA, \$resp_body); - $curl->setopt(CURLOPT_URL, $url . $endpoint); - - my $retcode = $curl->perform(); - if ($retcode == 0 && $curl->getinfo(CURLINFO_HTTP_CODE) == 204) { - return 1; - } elsif ($retcode == 0 && $curl->getinfo(CURLINFO_HTTP_CODE) == 403) { - rate_fail(); - } - - undef; -} - -sub issue_search($$$$$$) -{ - my($curl, $url, $owner, $repo, $milestone_id, $page) = @_; - - my $endpoint = "/repos/$owner/$repo/issues"; - - my $params = "milestone=$milestone_id&state=closed&page=$page"; - - my $resp_body; - - $curl->setopt(CURLOPT_WRITEDATA, \$resp_body); - $curl->setopt(CURLOPT_URL, $url . $endpoint . '?' . $params); - - my $retcode = $curl->perform(); - if ($retcode == 0 && $curl->getinfo(CURLINFO_HTTP_CODE) == 200) { - return from_json($resp_body); - } elsif ($retcode == 0 && $curl->getinfo(CURLINFO_HTTP_CODE) == 403) { - rate_fail(); - } - - undef; -} - -{ - # get the command line arguments - my %opts; - getopts("o:r:m:a:v", \%opts); - usage() if (! defined $opts{o} || ! defined $opts{r} || ! defined $opts{m}); - my $owner = $opts{o}; - my $repo = $opts{r}; - my $milestone = $opts{m}; - my $auth = $opts{a}; - my $verbose = 0; - $verbose = 1 if (defined $opts{v}); - - my $curl = WWW::Curl::Easy->new; - $curl->setopt(CURLOPT_VERBOSE, 1) if $verbose; - $curl->setopt(CURLOPT_HTTPHEADER, ['Accept: application/vnd.github.v3+json', 'User-Agent: Awesome-Octocat-App']); - - if (defined($auth)) { - $curl->setopt(CURLOPT_USERPWD, $auth); - } - - my $milestone_id = milestone_lookup($curl, $url, $owner, $repo, $milestone); - - if (!defined($milestone_id)) { - print STDERR "Milestone not found: $milestone\n"; - exit 1; - } - - my $issues; - my $changelog; - my $page = 1; - - print STDERR "Looking for issues from Milestone $milestone\n"; - - do { - print STDERR "Page $page\n"; - $issues = issue_search($curl, $url, $owner, $repo, $milestone_id, $page); - foreach my $issue (@{$issues}) { - if (defined($issue)) { - print STDERR "Issue #" . $issue->{number} . " - " . $issue->{title} . " "; - - if (!exists($issue->{pull_request})) { - print STDERR "not a PR.\n"; - next; - } - - if (!is_merged($curl, $url, $owner, $repo, $issue->{number})) { - print STDERR "not merged.\n"; - next; - } - - print STDERR "added.\n"; - push @{$changelog}, {number => $issue->{number}, title => $issue->{title}}; - } - } - $page++; - } while (scalar @{$issues}); - - if (defined($changelog)) { - print "Changes with Apache Traffic Server $milestone\n"; - - foreach my $issue (sort {$a->{number} <=> $b->{number}} @{$changelog}) { - print " #$issue->{number} - $issue->{title}\n"; - } - } -} diff --git a/tools/hrw4u/schema/sandbox.schema.json b/tools/hrw4u/schema/sandbox.schema.json index 516064ceee1..5e9b27322f9 100644 --- a/tools/hrw4u/schema/sandbox.schema.json +++ b/tools/hrw4u/schema/sandbox.schema.json @@ -45,7 +45,8 @@ "READ_RESPONSE", "SEND_RESPONSE", "TXN_CLOSE", - "VARS" + "VARS", + "SESSION_VARS" ] }, "uniqueItems": true diff --git a/tools/hrw4u/scripts/hrw4u b/tools/hrw4u/scripts/hrw4u index 91a586a181b..533815a6879 100755 --- a/tools/hrw4u/scripts/hrw4u +++ b/tools/hrw4u/scripts/hrw4u @@ -21,6 +21,7 @@ from __future__ import annotations import argparse import os +from importlib.metadata import PackageNotFoundError, version as _pkg_version from pathlib import Path from typing import Any @@ -31,7 +32,15 @@ from hrw4u.common import run_main from hrw4u.sandbox import SandboxConfig +def _package_version() -> str: + try: + return _pkg_version("apple-hrw4u") + except PackageNotFoundError: + return "unknown" + + def _add_args(parser: argparse.ArgumentParser, output_group: argparse._MutuallyExclusiveGroup) -> None: + parser.add_argument("--version", action="version", version=f"hrw4u {_package_version()}") output_group.add_argument( "--output", choices=["hrw", "hrw4u"], diff --git a/tools/hrw4u/tests/data/sandbox/denied-section-session-vars.ast.txt b/tools/hrw4u/tests/data/sandbox/denied-section-session-vars.ast.txt new file mode 100644 index 00000000000..a194f511720 --- /dev/null +++ b/tools/hrw4u/tests/data/sandbox/denied-section-session-vars.ast.txt @@ -0,0 +1 @@ +(program (programItem (section (sessionVarSection SESSION_VARS { (variables (variablesItem (variableDecl session_marked : bool ;))) }))) (programItem (section REMAP { (sectionBody (statement inbound.req.X-Marked = (value "1") ;)) })) ) diff --git a/tools/hrw4u/tests/data/sandbox/denied-section-session-vars.error.txt b/tools/hrw4u/tests/data/sandbox/denied-section-session-vars.error.txt new file mode 100644 index 00000000000..974a8009f2c --- /dev/null +++ b/tools/hrw4u/tests/data/sandbox/denied-section-session-vars.error.txt @@ -0,0 +1,2 @@ +'SESSION_VARS' is denied by sandbox policy (section) +Feature denied by sandbox policy. Contact platform team. diff --git a/tools/hrw4u/tests/data/sandbox/denied-section-session-vars.input.txt b/tools/hrw4u/tests/data/sandbox/denied-section-session-vars.input.txt new file mode 100644 index 00000000000..153339985a8 --- /dev/null +++ b/tools/hrw4u/tests/data/sandbox/denied-section-session-vars.input.txt @@ -0,0 +1,7 @@ +SESSION_VARS { + session_marked: bool; +} + +REMAP { + inbound.req.X-Marked = "1"; +} diff --git a/tools/hrw4u/tests/data/sandbox/denied-section-session-vars.sandbox.yaml b/tools/hrw4u/tests/data/sandbox/denied-section-session-vars.sandbox.yaml new file mode 100644 index 00000000000..43274b164a0 --- /dev/null +++ b/tools/hrw4u/tests/data/sandbox/denied-section-session-vars.sandbox.yaml @@ -0,0 +1,6 @@ +sandbox: + message: "Feature denied by sandbox policy. Contact platform team." + + deny: + sections: + - SESSION_VARS