Stop per-peer streamer threads at shutdown to avoid teardown crash

[jolavillette]

Stop per-peer streamer threads at shutdown to avoid teardown crash

Fix: crash / hang when closing RetroShare (shutdown thread-teardown race)

Symptom

When closing RetroShare, the log would sometimes show this pair of lines, often
repeated in bursts, and the process would fail to exit cleanly (in the worst
case it had to be killed):

E 1783751379.589 void RsMutex::lock()pthread_mutex_lock returned:  error: 22 Invalid argument category: generic

stack trace:
  ./retroshare-gui : RsMemoryManagement::SmallObject::operator new(unsigned long)+0x20
  ./retroshare-gui : RsRawSerialiser::deserialise(void*, unsigned int*)+0x5a
  ./retroshare-gui : RsSerialiser::deserialise(void*, unsigned int*)+0x19a
  ./retroshare-gui : pqistreamer::handleincoming()+0x7ff
  ./retroshare-gui : pqistreamer::tick_recv(unsigned int)+0x54
  ./retroshare-gui : pqithreadstreamer::threadTick()+0x6a
  ./retroshare-gui : ()+0xaccc19
  ./retroshare-gui : RsThread::rsthread_init(void*)+0x44
  /lib/x86_64-linux-gnu/libc.so.6 : ()+0x9caa4
  ...
(EE) allocating 56 bytes of memory that cannot be deleted. This is a bug, except if it happens when closing Retroshare

Both messages actually originate from the same line, SmallObject::operator new
in libretroshare/src/util/smallobject.cc:

void *SmallObject::operator new(size_t size)
{
    RsStackMutex m(_mtx);       // <- "error 22" here (mutex already destroyed)
    if(_allocator._active)      // <- _active == false
        return _allocator.allocate(size);
    else {
        std::cerr << "(EE) allocating ... cannot be deleted ...";  // <- 2nd message
        return malloc(size);
    }
}

Root cause

RsServer::rsGlobalShutDown() (libretroshare/src/rsserver/p3face-config.cc)
stopped the registered service threads and the RsServer tick thread, but never
the per-peer network I/O threads (pqithreadstreamer, one per connection).
These are detached RsTickingThreads that keep looping:

threadTick() -> tick_recv() -> handleincoming() -> RsSerialiser::deserialise() -> new RsItem

Nothing stopped them at shutdown (the only path that does, removePeer(), only
runs when a friend is removed), so they outlived the whole shutdown sequence.

When the process finally exited, the static destructors in smallobject.cc
ran:

1. ~SmallObjectAllocator sets _active = false;
2. ~RsMutex calls pthread_mutex_destroy(&_mtx).

Meanwhile those still-running streamer threads were deserialising an incoming
packet -> SmallObject::operator new -> RsStackMutex(_mtx) on the
already-destroyed mutex -> pthread_mutex_lock returns EINVAL (error 22),
and _active == false triggers the malloc fallback plus the "cannot be
deleted" message.

Net effect: concurrent access to static objects being torn down = undefined
behaviour, a flood of log lines, and a process that would not terminate
cleanly. Databases were already closed before this point, so there was no data
loss, but shutdown was broken.

This is a Linux destruction-ordering race, distinct from the Windows/Qt6
emutls thread-exit issue.

Fix

Add pqipersongrp::fullstopAllThreads(), called from rsGlobalShutDown() right
after fullstop() (so the RsServer tick thread is no longer iterating the peer
list concurrently), to stop all per-peer streamer threads before the static
objects are destroyed.

Two subtle pitfalls, both diagnosed live with gdb, had to be handled:

a) Deadlock. A first version held coreMtx while waiting for the threads.
But a streamer thread that is delivering a received item needs coreMtx to send
its reply (pqihandler::queueOutRsItem). Cycle: main -> holds coreMtx -> waits for streamers; p3rtt thread -> holds the rtt service mutex -> waits for coreMtx; ~50 streamers -> wait for the rtt mutex. Permanent hang.
=> Two phases: signal stop + close sockets while holding coreMtx
(non-blocking), release coreMtx, then wait for the threads to stop.

b) SIGSEGV. A blind (pqiperson*)it->second->pqi cast over the whole mods
map. But mods also contains the pqiloopback (the node's own loopback).
pqiloopback is a sibling of pqiperson (both derive from PQInterface),
not a pqiperson, so the cast reinterpreted unrelated memory -> garbage
mPersonMtx (error 22) then crash. removePeer() never hit this because it
looks up a specific friend id, never the loopback.
=> Use dynamic_cast<pqiperson*> and skip non-persons (the loopback has no
network threads to stop anyway).

Final implementation (libretroshare/src/pqi/pqipersongrp.cc):

int pqipersongrp::fullstopAllThreads()
{
    std::list<pqiperson *> persons;
    {
        RsStackMutex stack(coreMtx);                 // phase 1: under the lock
        for(auto it = mods.begin(); it != mods.end(); ++it)
        {
            // mods also holds non-pqiperson interfaces (e.g. the pqiloopback
            // for our own node) which have no network I/O threads; a blind
            // cast would crash on the loopback.
            pqiperson *p = dynamic_cast<pqiperson *>(it->second->pqi);
            if(!p) continue;
            p->stoplistening();
            p->reset();                              // askForStop() + close sockets (non-blocking)
            persons.push_back(p);
        }
    }                                                // coreMtx released
    for(auto it = persons.begin(); it != persons.end(); ++it)
        (*it)->fullstopthreads();                    // phase 2: wait, WITHOUT coreMtx
    return 1;
}

And the call site in RsServer::rsGlobalShutDown():

fullstop();

// Stop the per-peer network I/O threads (pqithreadstreamer) before the static
// SmallObject allocator/mutex are destroyed. Must run after fullstop() so the
// RsServer tick thread is no longer iterating the peer list concurrently.
if(pqih) pqih->fullstopAllThreads();

Result

At shutdown the streamer threads are now stopped before static teardown: no more
error 22 / cannot be deleted flood, no deadlock, no SIGSEGV, clean exit.