Pull from upstream#1
Open
deepakverma wants to merge 491 commits into
Open
Conversation
Fixing the current tests stalling PRs. Issue2507 fundamentally can't play nice either others, especially shared connections and simultaneous, so fixing that case, and adding `SetEmpty`) which has come up a few times as an example of this working.
This is a test-only change. I don't want to risk an upgrade and harvesting PII from anyone who works on our project, so I'm removing Moq immediately. See https://github.com/moq/moq/issues/1372 for details/discussion.
ILogger support - since we support `Action<ConfigurationOptions>` this can be added today e.g. with `o => o.LoggerFactory = myLoggerFactory`:
```cs
var muxer = ConnectionMultiplexer.Connect("localhost", o => o.LoggerFactory = myLoggerFactory);
```
...or we could add sibling overloads to the existing `TextWriter`. I'd love to not create a ^2 matrix every time we do this, though. Starting simpler for that reason. Note: this is on top of #2050 for simplicity, will roll against `main` when merged.
- satisfy infosec scanner that we're not risking pointer weirdness (although in reality, this is always UTF8; can't be hijacked) - prefer EncodingExtensions.GetString when it is available - switch payload to an explicit field so we can use ^^^ via pass-by-ref There is no actual infosec here; if the inbuilt UTF8 implementation is broken, we're already hosed; there is no extension point to replace UTF8
Setting up CodeQL scanning on the repo to see where it's at, and hopefully leave enabled as a safeguard.
Overall changes:
- [x] introduce `Resp2Type` and `Resp3Type` shims (`Resp2Type` has reduced types); existing code using `[Obsolete] Type` uses `Resp2Type` for minimal code impact
- [x] mark existing `Type` as `[Obsolete]`, and proxy to `Resp2Type` for compat
- [x] deal with null handling differences
- [x] deal with `Boolean`, which works very differently (`t`/`f` instead of `1`/`0`)
- [x] deal with `[+|-]{inf|nan}` when parsing doubles (explicitly called out in the RESP3 spec)
- [x] parse new tokens
- [x] `HELLO` handshake
- [x] core message and result handling
- [x] validation and fallback
- [x] prove all return types (see: redis/redis-specifications#15)
- [x] <strike>streamed RESP3</strike> omitting; not implemented by server; can revisit
- [x] deal with pub/sub differences
- [x] check routing of publish etc
- [x] check re-wire of subscription if failed
- [x] check receive notifications
- [x] connection management (i.e. not to spin up in resp3)
- [x] connection fallback spinup (i.e. if we were trying resp3 but failed)
- [x] other
- [x] [undocumented RESP3 delta](redis/redis-doc#2513)
- [x] run core tests in both RESP2 and RESP3
- [x] compensate for tests that expect separate subscription connections
Co-authored-by: Nick Craver <nickcraver@microsoft.com>
Co-authored-by: Nick Craver <nrcraver@gmail.com>
#2525) Sending tracers is not necessary because we just connected to the nodes a few seconds ago. It causes problems because sent tracers do not have enough time to respond.
Occasionally we'd see `chunkLength` errors from `StringWriter` `.ToString()` calls after connecting. I think we've isolated this (via test stress runs) down to a write happening post-lock on the `TextWriterLogger` disposal. This lock in dispose ensures we're not trying to write to a writer we should have fully released at the end of a `.Connect()`/`.ConnectAsync()` call.
Fixes: - `DisconnectAndNoReconnectThrowsConnectionExceptionAsync`: Under load, we're not connecting in 50ms for the initial thing, and stability > speed, so give a little on this test to help out. Average runtime will be higher, but that's way better than sporadically failing. - `Roles`: Don't assume we're in the initial primary/replica setup (we may have failed over) and make it work either way.
* properly categorizing EXPIRETIME and PEXPIRETIME * release notes
This allows connections through Sentinel via RESP3 by adding `HELLO` to the command map. Fixes #2591.
In Linux, we see 15 minute socket stalls due to OS-level TCP retries. What this means is the PhysicalConnection detects no issues on the pipe that's retrying, but is also not receiving data at all leading to long stalls in client applications. The goal here is to detect that we're timing out commands people have issued to the connection but we're getting _NOTHING_ back on the socket at all. In this case, we should assume the socket is dead and issue a reconnect so that we get out of the hung situation much faster. For an initial go at this, we've chosen 4x the timeout interval as a threshold, but could make this configurable if needed.
* add type-forward on IsExternalInit to resolve #2619
* Simplify InternalsVisibleTo * move PublicKey to Directory.Build.props * docs: add release notes entry --------- Co-authored-by: Marc Gravell <marc.gravell@gmail.com>
* honor select disposition in transactions * rearranging if/switch * using server instead of multiplxer * release notes
Co-authored-by: Nick Craver <nrcraver@gmail.com>
This adds a bit of null ref handling (few ifs). Fixes #2576. Overall, this is biting people in the shutdown race, more likely under load, so let's eat the if checks here to prevent it. I decided to go with the specific approach here as to not affect inlining.
* fix #2642 1: don't treat trivial clusters as clusters - Alibaba uses this config 2: report synchronous failures immidiately and accurately * instead of using node count, use explicit tracking of the DB count * release notes
* fix #2653 * Update src/StackExchange.Redis/ServerEndPoint.cs Co-authored-by: Philo <philon@microsoft.com> * release notes * merge fail --------- Co-authored-by: Philo <philon@microsoft.com>
* AMR: use RESP3 and disable the default config channel by default * release notes * support TLS (self-signed) in the test server, use for the AMR test * don't do the AMR post-connection callback * fix test break in Issue883 due to lazy provider lookup * improved release notes * verify config override works as expected * spin up pub/sub on demand (or if pub/sub is active) * build warning cleanup * logging * breakfix * format * AMR test for RESP2
* update CI to 8.8 rc1 * and the dockerfile
* draft support for 8.8 arrays * - mark API as experimental - add keyspace notification tests - tidying - docs * clarify how last-items interacts with ring buffers * fix CI netfx compilation * use ValuePairInterleavedProcessorBase to ensure that jagged vs flat doesn't impact us (RESP2 vs RESP3) * make life even easier * add Array to signature prefix list * stabilize hotkeys CI * fix last-minute ARGREP result change
…faults-provider (#3082) * Don't serialize (ToString() or clone Protocol if it comes from the defaults-provider * field list * release notes
* enable TCP keep-alives * Update ReleaseNotes.md
* 1. test and fix incorrect routing of some "sorted set" and "stream" commands 2. fix via new custom efficient message cores 3. add reusable helpers for common null checks on key/value arrays 4. prep work for likely ZDIFFCARD/ZUNIONCARD * rationalize * prefer async tests * add release note
* - fix snafu with NOCASE (logic inverted) - add IsReversed to simplify reversal logic * PR ref * - support ARINFO FULL - refactors ArrayInfo initialization to be span based, and put the main logic inside ArrayInfo technically this is a break, but: `[Experimental]` * release notes
* SENTINEL: defensive check around ROLE * PR number * CI fail; try updating test-reporter
* bare bones INCREX first stab * tidy INCREX impl * add INCREX integration tests * merge * compensate for 8.8RC1 API and behaviour changes * let's try adding the new files * oops, make sure flags is the final arg * account for last-minute-changes to INCREX * add my expectation assertions as integration tests; multiple failures - I'm asking "server" for guidance * update CI image * update test expectations to implement the reality of the server as-at 8.8 * update CI image to 8.8
Propagate exceptions out of Message.WriteTo so PhysicalBridge's outer write path can record a connection failure, and have HandleWriteException kill the PhysicalConnection via RecordConnectionFailed. Without this, a write that throws partway through a frame (e.g. an OOM during serialization) leaves bytes on the wire while the response queue still considers the slot healthy. The next reply from the server then matches against the wrong in-flight message — the symptom seen in #2883, #2804, and #2919, where commands return values intended for a different caller. HighIntegrity mode mitigates the symptom by detecting the desync after the fact via per-message echo checksums; this change addresses the underlying cause for the write-side variant. Adds three tests: - WriteTo must rethrow non-RedisCommandException out of WriteImpl, so the outer bridge catch can act on it. - RedisCommandException continues to surface unwrapped (it carries its own meaning and is excluded from the WriteTo catch filter). - End-to-end: a Message whose WriteImpl throws faults the awaiter with a RedisConnectionException(InternalFailure) AND raises a ConnectionFailed event, proving the physical connection was torn down.
* investigation into socket failure * 1. make TCP keep-alives an explicit option (default: "on") 2. hard-disable TCP keep-alives on DNS endpoints for non-Windows before .NET 10 * release notes * update CI image to 8.8 * fix typo in runtime test isolation / port selection. * Add OptionKeysAreAllNormalized to ensure all keys are correctly normalized; add tcpKeepAlive to docs * stabilize DeleteLastElementPublishesArrayDeleteBeforeKeyDeleteNotifications * allow more grace while waiting for ArrayTests work; only run that test in DEBUG * reduce CI brittleness by restricting tests that have wide side-effects; they can't run in parallel between runtimes * more CI stabilization; HOTKEYS this time
Parenthesize `??` as `+` binds stronger
* WIP v3 (#3035) * WIP v3 merge attempt * fix refs/externs * nits * ascii hash cleanup * remove unused local function * v3 remove pipelines.sockets.unofficial (#3042) * remove Pipelines.Sockets.Unofficial * wip * remove the last traces of Pipelines.Sockets.Unofficial * fix deps * WIP: deps * fix build deps * rev GitHubActionsTestLogger * - fix delegate [UnsafeAccessor] usage - implement Mono pattern * merge (incomplete) * merge * missed a sync-context await * fix missing configureawait usage * missing VT consumption * checking CI fail * versioning * don't test WriteMode.Sync/Async yet * failing test for #3048 * upstream #3049 into v3 * release notes * fix broken config test * re-add channel handling (merge) * optimize pub/sub check for server-defined channels * prefer WriteBulkString("..."u8) => WriteRaw(RESPu8) * optimize CommandMap by pre-generating all the RESP chunks * Move CI to 8.8 rc1 (#3079) * update CI to 8.8 rc1 * and the dockerfile * Draft support for 8.8 arrays (#3076) * draft support for 8.8 arrays * - mark API as experimental - add keyspace notification tests - tidying - docs * clarify how last-items interacts with ring buffers * fix CI netfx compilation * use ValuePairInterleavedProcessorBase to ensure that jagged vs flat doesn't impact us (RESP2 vs RESP3) * make life even easier * add Array to signature prefix list * stabilize hotkeys CI * fix last-minute ARGREP result change * CI: graceful skip if DEBUG not available * Alternative AwaitableMutex implementation for netfx (#3081) * implement AwaitableMutex for netfx * optimize successful sync take path * words * optimize async path for immediate acquisition * re-implement TCP keep-alive * default RESP3 everywhere (#3084) * default RESP3 everywhere * Retain <v6 RESP2 logic; default to v6 * move ship file entries * improve logging for ReadNotificationAsync * stabilize CI for RetryPolicyFailureCases * merge * broken merge (unshipped.txt) * add all the files * build fix * merge drift * CI: allow more time for Echo * use Task machinery for teardown test * Drag benchmark from respite branch * typo * "classic" * BufferedStreamWriter threading tests and fixes * more clarifications on BufferedStreamWriter * change default mode * investigating parse bug (Garnet SPOP?) * categorise all simulate-connection-failure tests; start working through failures * more simulate-failure tests * update `aoc` test expectation (fast-fail assert in test connect) * optimize ConfigurationOptions fields * Update CI workflow to include v3 branch for packaging * Fix missing Me() key usage * be explicit about test TFMs * merge the sync and async write implementations into a shared implementation that can transition from sync to async if needed * 3.0 release * counting is hard
v3 is now merged into main (v2 branch for maintenance); the version-offset would now lead to weirdness
* - introduce CycleBufferPool as a concept that encapsulates MemoryPool<byte> with the growth logic - support `-d` in the benchmark * - add ConfigurationOptions support - add [SkipLocalsInit] to RESPite * add ResponseArrayPool (#3104) * add ResponseArrayPool * Lease * IMemoryOwner<byte>? --------- Co-authored-by: ITikhonov <ITikhonov@lanit.ru> * refactor API so that MemoryPool<T> can be used throughout, and shared between lease and cycle-buffers * we don't actually need a default CycleBufferPool<T> impl; * fix typo * avoid `Failed` nomenclature in tests - makes it hard to isolate CI problems --------- Co-authored-by: Ivan Tikhonov <pairbit@gmail.com> Co-authored-by: ITikhonov <ITikhonov@lanit.ru>
Updated release notes for version 2.13.17 with fixes and enhancements.
* move KeyIdle[Async]Tests to a server without replication * stabilize Windows CI * don't run the `OBJECT IDLETIME` tests in WSL * skip sentinel tests on Windows CI
* Merge pull request #4 from pairbit/RedisValueSequence Redis value sequence * fix order * fix * PublicAPI.Unshipped * GetCharCount and GetMaxCharCount * add checking OverflowException * comment SequenceCompareTo * implement StartsWith * fix StartsWith * refact * GetHashCode * add Equals, CompareTo, StartsWith for ByteArray or MemoryManager * add ReadOnlySequenceIterator * refac * - GetHashCode - GetChars/GetCharCount - Simplify - normalize .First.Span * TryParse * mark WriteUnifiedSequence(ROS) as redundant for now (and fix 64-bit length) * CompareTo * add integration test (and fix CI) * polishing * avoid the string alloc in GetHashCode (when possible) --------- Co-authored-by: ITikhonov <ITikhonov@lanit.ru> Co-authored-by: Marc Gravell <marc.gravell@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.