Add configurable Windows RCON text conversion and detailed player ID queries

.github/workflows/console-query-tests.yml

@@ -0,0 +1,28 @@
+name: Console and query regression tests
+on:
+  push:
+    paths:
+      - 'Server/Components/Console/**'
+      - 'Server/Components/LegacyNetwork/**'
+      - 'Tests/ConsoleQuery/**'
+      - '.github/workflows/console-query-tests.yml'
+  pull_request:
+    paths:
+      - 'Server/Components/Console/**'
+      - 'Server/Components/LegacyNetwork/**'
+      - 'Tests/ConsoleQuery/**'
+      - '.github/workflows/console-query-tests.yml'
+  workflow_dispatch:
+permissions:
+  contents: read
+jobs:
+  unit:
+    strategy:
+      matrix:
+        os: [ubuntu-latest, windows-latest]
+    runs-on: ${{ matrix.os }}
+    steps:
+      - uses: actions/checkout@v4
+      - run: cmake -S Tests/ConsoleQuery -B build/regression -DCMAKE_BUILD_TYPE=Release
+      - run: cmake --build build/regression --config Release
+      - run: ctest --test-dir build/regression -C Release --output-on-failure

CMakeLists.txt

@@ -157,3 +157,9 @@ if(BUILD_ABI_CHECK_TOOL)
 	message("Configuring abi-check")
 	add_subdirectory(Tools)
 endif()
+
+option(BUILD_CONSOLE_QUERY_TESTS "Build console encoding and player query regression tests" OFF)
+if(BUILD_CONSOLE_QUERY_TESTS)
+	enable_testing()
+	add_subdirectory(Tests/ConsoleQuery)
+endif()

CONSOLE_QUERY.md

@@ -0,0 +1,87 @@
+# Legacy console text and detailed player queries
+
+These changes are source changes proposed for review, not an official open.mp
+release. Build the whole server from one revision; do not mix components from
+this checkout with a different server/SDK revision.
+
+## Windows console and remote RCON encoding
+
+The game protocol sends text bytes without an encoding marker. For a GBK game
+server, Unicode text entered at the Windows console must become GBK before it
+reaches `say` or a script command. Add this to the existing `config.json`:
+
+```json
+"console": { "input_codepage": 936 }
+```
+
+`console.input_codepage` defaults to **65001 (UTF-8)**, preserving existing
+behavior. Use 936 for GBK, 950 for Big5, or 0 for the Windows system ANSI code
+page. Conversion rejects invalid or unrepresentable characters, including emoji
+that GBK cannot encode, instead of executing a changed command. Local command
+replies are converted back to UTF-8 for console/log output. Settings are Windows
+only; POSIX console behavior is unchanged.
+
+Remote RCON defaults to forwarding the original command bytes. Clients that
+already send GBK, including SampCtrl's AUTO/GBK mode, need no server conversion.
+For a client that sends **UTF-8 commands** to a GBK game server, explicitly enable:
+
+```json
+"rcon": { "utf8_command_codepage": 936 }
+```
+
+Merge this property into the existing `rcon` object, preserving its `enable` and
+`password` properties. Zero (the default) disables conversion. A nonzero value
+selects the Windows code page used for the decoded UTF-8 command. Malformed UTF-8
+and lossy conversions return an error and do not dispatch the command. Passwords
+are checked against their original bytes before any conversion; replies retain
+their original byte encoding. In-game RCON and `SendRconCommand` are unchanged.
+
+All remote senders must agree on UTF-8 before enabling this option. Do not enable
+it while SampCtrl is in AUTO/GBK mode: either leave it at 0 or select UTF-8 in the
+app. UTF-8 command input does not change the encoding of server replies, player
+names, configuration strings, or other game messages. A client's response decoder
+must still support the server's reply encoding. There is no reliable automatic
+way to distinguish untagged UTF-8 from every legacy encoding.
+
+## Public detailed player queries
+
+The existing `c` response remains unchanged: it has names and scores but no IDs.
+The legacy `d` response supplies real pool IDs, names, scores and ping when there
+are at most 100 players and every ID fits in a byte. If any ID exceeds 255, `d`
+does not reply; it never truncates an ID or substitutes the row number.
+
+The proposed uppercase `D` extension supplies 16-bit IDs in bounded pages. It
+uses the existing SampCtrl `sc_d16=1` rule as a compatibility capability marker.
+This is an experimental extension for upstream review, not a claim of an
+official standardized protocol. It exposes the same public names/scores plus
+IDs and ping; it contains no IP addresses or credentials. `enable_query=false`
+disables it together with the existing queries. RCON authentication is neither
+required for reading this public information nor bypassed for management actions.
+
+All integers are little-endian. Requests start with the normal 11-byte SAMP query
+header with opcode `D`, followed by:
+
+| Field | Bytes | Meaning |
+| --- | ---: | --- |
+| version | 1 | 1 |
+| generation | 4 | 0 for the first page; reuse the returned generation thereafter |
+| page | 2 | Zero based; generation 0 requires page 0 |
+
+Responses echo the query header, then version(1), generation(4), total(2), page(2),
+pageCount(2), count(2). Each record contains ID(2), nameLength(1), name bytes,
+signed score(4), unsigned ping(4). Name bytes are unchanged, at most 24 bytes.
+
+- The main thread snapshots non-bot players every second, sorted by actual ID.
+- Each page has at most 32 players. At 1000 players there are 32 pages; even with
+  maximum-length names a response is at most 1144 bytes, including the query header.
+- An empty server returns one empty page. Generation 0 is never emitted.
+- The last 16 snapshots are retained, bounding memory and supporting consistent
+  pagination while players change. Bad lengths, versions, pages and expired
+  generations receive no reply. A client should restart at generation 0 after
+  an unavailable page and reject mixed generations or duplicate IDs.
+- A snapshot may be up to one refresh interval old. Management clients must
+  revalidate identity through authenticated commands before acting on an ID
+  that may have been reused after a disconnect.
+
+See [the regression test instructions](Tests/ConsoleQuery/README.md) for local
+unit tests and isolated server integration tests.

Server/Components/Console/console_encoding.hpp

@@ -0,0 +1,77 @@
+/* SPDX-License-Identifier: MPL-2.0
+ * Lossless Windows console command conversion.
+ */
+#pragma once
+#ifdef WIN32
+#include <windows.h>
+#include <limits>
+#include <string>
+#include <string_view>
+#include <utility>
+
+namespace ConsoleEncoding
+{
+// Refuse lossy conversion: a command must not silently acquire '?' or best-fit characters.
+inline bool encode(std::wstring_view input, unsigned codepage, std::string& output)
+{
+	output.clear();
+	if (codepage == 0)
+		codepage = GetACP();
+	if (!IsValidCodePage(codepage) || input.size() > (std::numeric_limits<int>::max)())
+		return false;
+	if (input.empty())
+		return true;
+	const DWORD flags = (codepage == CP_UTF8 || codepage == 54936) ? WC_ERR_INVALID_CHARS : 0;
+	const int length = static_cast<int>(input.size());
+	const int size = WideCharToMultiByte(codepage, flags, input.data(), length, nullptr, 0, nullptr, nullptr);
+	if (size <= 0)
+		return false;
+	std::string result(size, '\0');
+	if (WideCharToMultiByte(codepage, flags, input.data(), length, result.data(), size, nullptr, nullptr) != size)
+		return false;
+	const int wideSize = MultiByteToWideChar(codepage, 0, result.data(), size, nullptr, 0);
+	if (wideSize != length)
+		return false;
+	std::wstring roundTrip(wideSize, L'\0');
+	if (MultiByteToWideChar(codepage, 0, result.data(), size, roundTrip.data(), wideSize) != wideSize || roundTrip != input)
+		return false;
+	output = std::move(result);
+	return true;
+}
+
+inline bool toUtf8(std::string_view input, unsigned codepage, std::string& output)
+{
+	output.clear();
+	if (input.size() > (std::numeric_limits<int>::max)())
+		return false;
+	if (input.empty())
+		return true;
+	const int size = MultiByteToWideChar(codepage, 0, input.data(), static_cast<int>(input.size()), nullptr, 0);
+	if (size <= 0)
+		return false;
+	std::wstring wide(size, L'\0');
+	if (MultiByteToWideChar(codepage, 0, input.data(), static_cast<int>(input.size()), wide.data(), size) != size)
+		return false;
+	return encode(wide, CP_UTF8, output);
+}
+
+// Remote RCON has no encoding marker. Only call this when an administrator
+// explicitly configures UTF-8 input; never guess from the command bytes.
+inline bool fromUtf8(std::string_view input, unsigned codepage, std::string& output)
+{
+	output.clear();
+	if (input.size() > (std::numeric_limits<int>::max)())
+		return false;
+	if (input.empty())
+		return encode({}, codepage, output);
+	const int length = static_cast<int>(input.size());
+	const int size = MultiByteToWideChar(CP_UTF8, MB_ERR_INVALID_CHARS, input.data(), length, nullptr, 0);
+	if (size <= 0)
+		return false;
+	std::wstring wide(size, L'\0');
+	if (MultiByteToWideChar(CP_UTF8, MB_ERR_INVALID_CHARS, input.data(), length, wide.data(), size) != size)
+		return false;
+	return encode(wide, codepage, output);
+}
+}
+#endif

Server/Components/Console/console_impl.hpp

@@ -20,6 +20,7 @@
 #include <network.hpp>
 #include <sdk.hpp>
 #include <thread>
+#include "console_encoding.hpp"
 
 using namespace Impl;
 
@@ -63,7 +64,10 @@ class ConsoleComponent final : public IConsoleComponent, public CoreEventHandler
 	DefaultEventDispatcher<ConsoleEventHandler> eventDispatcher;
 	std::mutex cmdMutex;
 	std::atomic_bool newCmd = false;
-	String cmd;
+	std::wstring cmd;
+#ifdef WIN32
+	inline static thread_local unsigned localReplyCodepage = 0;
+#endif
 	ThreadProcData* threadData;
 	std::thread cinThread;
 	std::thread::native_handle_type nativeThreadHandle;
@@ -154,6 +158,16 @@ class ConsoleComponent final : public IConsoleComponent, public CoreEventHandler
 public:
 	void sendMessage(const ConsoleCommandSenderData& recipient, StringView message) override
 	{
+#ifdef WIN32
+		// Replies to local input use the command's byte encoding; logLnU8 expects UTF-8.
+		String localReply;
+		if (recipient.sender == ConsoleCommandSender::Console && localReplyCodepage != 0
+			&& ConsoleEncoding::toUtf8(std::string_view(message.data(), message.size()), localReplyCodepage, localReply))
+		{
+			core->logLnU8(LogLevel::Message, "%.*s", PRINT_VIEW(localReply));
+			return;
+		}
+#endif
 		// We don't output UTF-8 for ingame commands
 		if (recipient.sender == ConsoleCommandSender::Player)
 		{
@@ -203,6 +217,22 @@ class ConsoleComponent final : public IConsoleComponent, public CoreEventHandler
 		return SemanticVersion(OMP_VERSION_MAJOR, OMP_VERSION_MINOR, OMP_VERSION_PATCH, BUILD_NUMBER);
 	}
 
+	void provideConfiguration(ILogger& logger, IEarlyConfig& config, bool defaults) override
+	{
+#ifdef WIN32
+		// Preserve UTF-8 by default; legacy servers can select their game code page.
+		if (defaults || config.getType("console.input_codepage") == ConfigOptionType_None)
+		{
+			config.setInt("console.input_codepage", CP_UTF8);
+		}
+		if (defaults || config.getType("rcon.utf8_command_codepage") == ConfigOptionType_None)
+		{
+			// Zero leaves remote commands byte-for-byte unchanged.
+			config.setInt("rcon.utf8_command_codepage", 0);
+		}
+#endif
+	}
+
 	void onLoad(ICore* core) override
 	{
 		this->core = core;
@@ -245,7 +275,8 @@ class ConsoleComponent final : public IConsoleComponent, public CoreEventHandler
 			{
 				std::scoped_lock<std::mutex> lock(threadData->component->cmdMutex);
 
-				threadData->component->cmd = std::wstring_convert<std::codecvt_utf8_utf16<wchar_t>, wchar_t>().to_bytes(line);
+				// Keep Unicode intact until the main thread applies the configured game encoding.
+				threadData->component->cmd = line;
 				threadData->component->newCmd = true;
 			}
 			else
@@ -342,10 +373,33 @@ class ConsoleComponent final : public IConsoleComponent, public CoreEventHandler
 		bool expected = true;
 		if (newCmd.compare_exchange_weak(expected, false))
 		{
-			cmdMutex.lock();
-			String command = cmd;
-			cmdMutex.unlock();
-
+			std::wstring line;
+			{
+				std::scoped_lock<std::mutex> lock(cmdMutex);
+				line = cmd;
+			}
+			String command;
+#ifdef WIN32
+			const int* setting = core->getConfig().getInt("console.input_codepage");
+			const unsigned codepage = setting ? static_cast<unsigned>(*setting) : CP_UTF8;
+			if (!ConsoleEncoding::encode(line, codepage, command))
+			{
+				core->logLn(LogLevel::Error, "Local console command was not executed: text cannot be represented in console.input_codepage (%u).", codepage);
+				return;
+			}
+			struct ReplyEncodingScope
+			{
+				unsigned previous;
+				ReplyEncodingScope(unsigned value)
+					: previous(localReplyCodepage)
+				{
+					localReplyCodepage = value;
+				}
+				~ReplyEncodingScope() { localReplyCodepage = previous; }
+			} scope(codepage == 0 ? GetACP() : codepage);
+#else
+			command = std::wstring_convert<std::codecvt_utf8_utf16<wchar_t>, wchar_t>().to_bytes(line);
+#endif
 			send(command);
 		}
 	}

Server/Components/LegacyNetwork/Query/detailed_players.hpp

@@ -0,0 +1,81 @@
+/* SPDX-License-Identifier: MPL-2.0
+ * Bounded, paginated detailed player query encoding.
+ */
+#pragma once
+#include <algorithm>
+#include <cstdint>
+#include <string>
+#include <vector>
+#include <utility>
+
+namespace DetailedPlayersQuery
+{
+constexpr size_t PageSize = 32;
+constexpr size_t MaxPlayers = 1000;
+constexpr size_t MaxNameBytes = 24;
+struct Player
+{
+	uint16_t id;
+	std::string name;
+	int32_t score;
+	uint32_t ping;
+};
+struct Snapshot
+{
+	uint32_t generation;
+	std::vector<std::vector<char>> pages;
+};
+inline void append(std::vector<char>& out, uint32_t value, size_t bytes)
+{
+	for (size_t i = 0; i < bytes; ++i)
+		out.push_back(static_cast<char>((value >> (8 * i)) & 255));
+}
+inline void record(std::vector<char>& out, const Player& player, bool wide)
+{
+	append(out, player.id, wide ? 2 : 1);
+	const auto length = std::min(player.name.size(), MaxNameBytes);
+	append(out, static_cast<uint32_t>(length), 1);
+	out.insert(out.end(), player.name.begin(), player.name.begin() + length);
+	append(out, static_cast<uint32_t>(player.score), 4);
+	append(out, player.ping, 4);
+}
+inline Snapshot buildSnapshot(uint32_t generation, const std::vector<Player>& players)
+{
+	Snapshot result { generation, {} };
+	if (generation == 0 || players.size() > MaxPlayers)
+		return result;
+	const size_t pageCount = std::max(size_t(1), (players.size() + PageSize - 1) / PageSize);
+	for (size_t page = 0; page < pageCount; ++page)
+	{
+		const size_t begin = page * PageSize;
+		const size_t end = std::min(begin + PageSize, players.size());
+		std::vector<char> output(11, 0);
+		output[10] = 'D';
+		append(output, 1, 1);
+		append(output, generation, 4);
+		append(output, static_cast<uint32_t>(players.size()), 2);
+		append(output, static_cast<uint32_t>(page), 2);
+		append(output, static_cast<uint32_t>(pageCount), 2);
+		append(output, static_cast<uint32_t>(end - begin), 2);
+		for (size_t i = begin; i < end; ++i)
+			record(output, players[i], true);
+		result.pages.push_back(std::move(output));
+	}
+	return result;
+}
+inline std::vector<char> buildLegacy(const std::vector<Player>& players)
+{
+	// The legacy protocol cannot represent IDs above 255. Never wrap or invent IDs.
+	if (players.size() > 100 || std::any_of(players.begin(), players.end(), [](const Player& p)
+			{
+				return p.id > 255;
+			}))
+		return {};
+	std::vector<char> output(11, 0);
+	output[10] = 'd';
+	append(output, static_cast<uint32_t>(players.size()), 2);
+	for (const auto& player : players)
+		record(output, player, false);
+	return output;
+}
+}