Art Cancro [Wed, 6 Sep 2023 03:05:01 +0000 (23:05 -0400)]
Release version 993 generated by do-release.sh
Art Cancro [Wed, 6 Sep 2023 03:04:50 +0000 (23:04 -0400)]
berkeley_db: cdb_next_item() use DB_REALLOC, not DB_MALLOC
Fixes a 'Brown Paper Bug' that will leak memory ... catastrophically if ctdldump is run on a big db.
Art Cancro [Wed, 6 Sep 2023 03:00:39 +0000 (23:00 -0400)]
ctdldump/ctdlload: document the -y flag
Art Cancro [Wed, 6 Sep 2023 02:55:15 +0000 (22:55 -0400)]
serv_fulltext: better handling of exit by yield vs complete
Art Cancro [Tue, 5 Sep 2023 22:14:40 +0000 (18:14 -0400)]
Richard Stallman is a fucking left-wing communist douchebag.
Art Cancro [Tue, 5 Sep 2023 22:09:38 +0000 (18:09 -0400)]
textclient: don't prompt to change internet email addresses when deleting a user
The 'edit user' and 'delete user' workflows run through the same function since
the process of deleting a user simply edits them and changes their access level
to 0. Add a check for cmd==25 (edit user) before asking the user if they want
to change the email addresses.
Art Cancro [Tue, 5 Sep 2023 03:59:44 +0000 (23:59 -0400)]
Omit the warning about session scheduled for termination but not idle.
Something about this warning makes it reference a bad pointer and crash
during periods of extremely high load. I don't know why, but just leaving
it alone seems to fix the problem.
Art Cancro [Sat, 2 Sep 2023 21:27:59 +0000 (17:27 -0400)]
Release version 992 generated by do-release.sh
Art Cancro [Sat, 2 Sep 2023 03:22:50 +0000 (23:22 -0400)]
only build chkpw and chkpwd if we can
Art Cancro [Sat, 2 Sep 2023 03:15:14 +0000 (23:15 -0400)]
Remove the warning-this-is-unfinished warnings from ctdldump.c and ctdlload.c
Most of these tools are now building successfully on FreeBSD.
Art Cancro [Sat, 2 Sep 2023 02:42:03 +0000 (22:42 -0400)]
Include -lintl where we need it on systems that have it
Art Cancro [Sat, 2 Sep 2023 02:37:20 +0000 (22:37 -0400)]
Include -lresolv only if we have it
Charlie Root [Sat, 2 Sep 2023 02:27:08 +0000 (22:27 -0400)]
loadtest: run with a warning instead of exiting when not all threads connect.
Art Cancro [Fri, 1 Sep 2023 16:10:02 +0000 (16:10 +0000)]
loadtest: pretty display when running with hundreds of threads
Art Cancro [Fri, 1 Sep 2023 15:55:20 +0000 (15:55 +0000)]
fulltext: be smarter about exiting silently if there's nothing to do.
Art Cancro [Fri, 1 Sep 2023 15:07:42 +0000 (15:07 +0000)]
fulltext: after the indexer runs for 60 seconds (tunable), yield the thread
root [Fri, 1 Sep 2023 14:50:57 +0000 (14:50 +0000)]
loadtest: minor changes
Art Cancro [Fri, 1 Sep 2023 03:10:36 +0000 (23:10 -0400)]
Release version 991 generated by do-release.sh
Art Cancro [Fri, 1 Sep 2023 03:10:28 +0000 (23:10 -0400)]
Moved the creation of the data directory.
It was happening too late in the new code and failing on some new installations.
Art Cancro [Thu, 31 Aug 2023 21:35:40 +0000 (12:35 -0900)]
Release version 990 generated by do-release.sh
Art Cancro [Thu, 31 Aug 2023 21:35:33 +0000 (12:35 -0900)]
Release version 989 generated by do-release.sh
Art Cancro [Thu, 31 Aug 2023 21:13:13 +0000 (12:13 -0900)]
On second thought, let's do it this way.
Art Cancro [Thu, 31 Aug 2023 21:08:02 +0000 (12:08 -0900)]
Revert "setup: reflect the fact that the operator should restart the server"
This reverts commit
c7589e995080a55e55c587d65c85a5889beb4194.
Art Cancro [Thu, 31 Aug 2023 21:00:12 +0000 (12:00 -0900)]
setup: reflect the fact that the operator should restart the server
Art Cancro [Thu, 31 Aug 2023 19:10:32 +0000 (10:10 -0900)]
Removed background and restart from citserver.
These functions should be managed by an external supervisor program.
Art Cancro [Thu, 31 Aug 2023 18:48:43 +0000 (09:48 -0900)]
fix compiler warning
Art Cancro [Thu, 31 Aug 2023 18:32:55 +0000 (14:32 -0400)]
Renamed Makefile to GNUmakefile to enforce the use of gmake
Art Cancro [Thu, 31 Aug 2023 04:03:53 +0000 (00:03 -0400)]
Rewrite of the indexer is complete!
No more search hooks. Modules can call each other now and that's a lot simpler.
We also eliminated the index cache, because it made things a lot more complex, and
a modern database is going to mmap most of the index as it gets used anyway.
Art Cancro [Thu, 31 Aug 2023 03:28:30 +0000 (23:28 -0400)]
Removed all the disused CtdlUnregister functions.
We don't call Unregister functions anymore because our objective is now
to close the databases and GTFO when the server needs to shut down.
Art Cancro [Thu, 31 Aug 2023 03:13:11 +0000 (23:13 -0400)]
search function is complete
Art Cancro [Wed, 30 Aug 2023 19:27:39 +0000 (15:27 -0400)]
Wrapping indexer transactions inside a critical section seems to make it not deadlock
Art Cancro [Wed, 30 Aug 2023 16:03:08 +0000 (12:03 -0400)]
Rewrote the indexer
Art Cancro [Wed, 30 Aug 2023 02:49:34 +0000 (22:49 -0400)]
Removed double-fetch-room in the indexer.
I don't want to celebrate prematurely, but we've gotten through 10,000 ops
in every of 10 threads without deadlocking.
Art Cancro [Wed, 30 Aug 2023 02:40:31 +0000 (22:40 -0400)]
free(msglist)
Art Cancro [Wed, 30 Aug 2023 02:30:32 +0000 (22:30 -0400)]
Totally ripping apart the indexer.
And yet, it doesn't seem to be anything 'deep' that's making it deadlock.
If I get desperate I may have to just give it its own thread or something.
Art Cancro [Tue, 29 Aug 2023 21:12:04 +0000 (17:12 -0400)]
serv_fulltext: wordbreaker now returns a libcitadel Array
Art Cancro [Tue, 29 Aug 2023 19:03:34 +0000 (15:03 -0400)]
CtdlForEachMessage() don't process message 0
Art Cancro [Tue, 29 Aug 2023 17:12:09 +0000 (13:12 -0400)]
threads.c: commented out try_critical_section() , I don't want this
Art Cancro [Tue, 29 Aug 2023 15:50:22 +0000 (11:50 -0400)]
loadtest.c: after creating accounts and rooms, log back in using a non privileged socket
Art Cancro [Tue, 29 Aug 2023 15:44:31 +0000 (11:44 -0400)]
Makefile: proper incantation for compiling loadtest
Art Cancro [Tue, 29 Aug 2023 15:36:36 +0000 (11:36 -0400)]
context.c: fixed invalid pointer
Art Cancro [Tue, 29 Aug 2023 05:37:59 +0000 (01:37 -0400)]
Temporarily disable the indexer
Art Cancro [Tue, 29 Aug 2023 03:54:50 +0000 (23:54 -0400)]
no diags
Art Cancro [Tue, 29 Aug 2023 03:53:58 +0000 (23:53 -0400)]
cdb_tick() called periodically
Art Cancro [Tue, 29 Aug 2023 02:38:55 +0000 (22:38 -0400)]
When opening a cursor, always read through to the end.
This should theoretically eliminate the need to ever call cdb_close_cursor()
Art Cancro [Tue, 29 Aug 2023 02:24:45 +0000 (22:24 -0400)]
cosmetic stuff
Art Cancro [Tue, 29 Aug 2023 02:08:52 +0000 (22:08 -0400)]
cleaned that up a bit
Art Cancro [Tue, 29 Aug 2023 02:02:51 +0000 (22:02 -0400)]
loadtest.c: multiple threadds
Art Cancro [Tue, 29 Aug 2023 01:26:35 +0000 (16:26 -0900)]
Generate listsub token with a hash instead of crypt
Art Cancro [Mon, 28 Aug 2023 23:43:38 +0000 (19:43 -0400)]
.gitignore: add dbnothing and loadtest
Art Cancro [Mon, 28 Aug 2023 15:26:54 +0000 (11:26 -0400)]
Makefile: use ${CC} selected by configure
Art Cancro [Mon, 28 Aug 2023 15:19:42 +0000 (11:19 -0400)]
Removed the 'ucred' stuff. It's insecure and it doesn't work on FreeBSD anyway.
Art Cancro [Mon, 28 Aug 2023 15:13:42 +0000 (11:13 -0400)]
FreeBSD compatibility (first of several commits)
Art Cancro [Mon, 28 Aug 2023 01:03:01 +0000 (21:03 -0400)]
Removed extra diagnostics
Art Cancro [Mon, 28 Aug 2023 00:40:34 +0000 (15:40 -0900)]
diagnostics
Art Cancro [Sun, 27 Aug 2023 04:17:42 +0000 (00:17 -0400)]
Found a place where CM_SetAsField() was owning db memory.
Changed that, of course. It was causing a double free.
Art Cancro [Sun, 27 Aug 2023 04:04:57 +0000 (00:04 -0400)]
CM_SetField() no longer accepts a length, just uses strdup()
Art Cancro [Sun, 27 Aug 2023 00:29:07 +0000 (20:29 -0400)]
loadtest.c: when deleting, always make sure we deleted 1 message.
If there are multiple loadtest processes running -- and that's really
the whole idea -- we will often try to delete a message that was already
deleted by another process. That's not an error condition, but it would
cause posts to outnumber deletes and the rooms would grow endlessly. So
if we didn't successfully delete something, we go back and try deleting
another one.
Fun fact: the algorithm used to decide which message to delete is the same
algorithm used in the 'fortune' program.
Art Cancro [Sun, 27 Aug 2023 00:09:59 +0000 (20:09 -0400)]
loadtest.c: added
Art Cancro [Sat, 26 Aug 2023 23:19:54 +0000 (19:19 -0400)]
berkeley_db.c: improvements to transactional data store
https://docs.oracle.com/cd/E17276_01/html/programmer_reference/transapp_env_open.html
The use of the DB_RECOVER flag to dbenv->open() appears to do more than just try to
recover the environment upon startup. It also creates __db.00X files, which are
perhaps holding the environment??? In any case it seems to be more solid this way.
I am also removing the use of any "private" transactions, since they are no longer
needed. Operations which write to the database check to see if the thread's dedicated
transaction pointer is in effect, maintaining that transaction if it is, and creating
a transaction in the same pointer otherwise.
There is a "double free" bug somewhere in here and I'm still trying to figure out what
conditions cause it to happen.
Art Cancro [Fri, 25 Aug 2023 21:08:53 +0000 (12:08 -0900)]
master_cleanup() is now the global shutdown/exit function
Art Cancro [Fri, 25 Aug 2023 03:47:33 +0000 (18:47 -0900)]
db debugging
Art Cancro [Fri, 25 Aug 2023 02:29:04 +0000 (17:29 -0900)]
dbnothing does nothing
Art Cancro [Fri, 25 Aug 2023 00:06:31 +0000 (15:06 -0900)]
dbnothing: temporary utility that just opens and closes the database as a test
Art Cancro [Thu, 24 Aug 2023 23:37:18 +0000 (14:37 -0900)]
ctdldump/ctdlload: fix usage messages
Art Cancro [Thu, 24 Aug 2023 23:23:09 +0000 (14:23 -0900)]
ctdldump/ctdlload: -h now works identical to citserver
Art Cancro [Thu, 24 Aug 2023 23:16:20 +0000 (14:16 -0900)]
ctdlload: use the backend API instead of BDB calls
It works, but it is several orders of magnitude slower. Why?
Art Cancro [Thu, 24 Aug 2023 18:44:37 +0000 (09:44 -0900)]
ctdldump: use the backend API instead of BDB calls
This makes ctdldump backend agnostic, or at least as backend
agnostic as citserver itself is at this point. We will probably
add some calling syntax across the board when we add more
backends, but at least this utility doesn't have to be rewritten
now. This makes me a happy ig. ctdlload will be next.
Art Cancro [Thu, 24 Aug 2023 15:08:02 +0000 (06:08 -0900)]
Makefile: BACKEND_LDFLAGS "created" by configure
Art Cancro [Thu, 24 Aug 2023 14:30:40 +0000 (05:30 -0900)]
Drop support for upgrading sites older than v811 (8.11)
That release was more than 11 years ago. Sites have had ample time to upgrade.
Art Cancro [Thu, 24 Aug 2023 14:09:38 +0000 (05:09 -0900)]
Makefile: move -ldb from hardcoded to a new BACKEND_LDFLAGS
Art Cancro [Thu, 24 Aug 2023 13:55:33 +0000 (04:55 -0900)]
Makefile: restructured so that backends can be compiled into utilities
database.c has been moved from server/ into server/backends/common.
Makefile now separates server objects from backend objects.
This is all in preparation for ctdldump and ctdlload to link the backend
code instead of calling BDB directly.
Art Cancro [Wed, 23 Aug 2023 18:37:10 +0000 (09:37 -0900)]
Remove CTDLUID definition, point to ctdluid symbol instead.
CTDLUID was probably a hard-coded preprocessor definition back in the
ancient times. Now it is not.
Art Cancro [Wed, 23 Aug 2023 14:17:28 +0000 (05:17 -0900)]
cdb_next_item() now returns both key and value
Art Cancro [Tue, 22 Aug 2023 03:54:07 +0000 (18:54 -0900)]
Removed call to db_env->set_cachesize()
The third parameter has always been set to 0
so it wasn't ever doing anything before.
Art Cancro [Tue, 22 Aug 2023 03:14:31 +0000 (18:14 -0900)]
Calls to cdb_fetch()/cdb_next_item() now check ptr for NULL or non-NULL
This is a better practice than checking to see if the data length was 0.
It now allows for extant but zero-length data to be returned.
Art Cancro [Mon, 21 Aug 2023 20:49:58 +0000 (11:49 -0900)]
cdb_free() is no longer needed; removed from backend API
Art Cancro [Mon, 21 Aug 2023 20:45:38 +0000 (11:45 -0900)]
cdb_fetch() and cdb_next_item() now return a struct cdbdata instead of a pointer to one
Art Cancro [Mon, 21 Aug 2023 19:09:45 +0000 (10:09 -0900)]
bdb_decompress_if_necessary() now operates on DBT instead of cdbdata
This allows us to maintain the DB_DBT_REALLOC heuristics.
Art Cancro [Mon, 21 Aug 2023 01:33:14 +0000 (16:33 -0900)]
Remove diagnostic section
Art Cancro [Mon, 21 Aug 2023 01:23:47 +0000 (16:23 -0900)]
Keep track of whether cdb->ptr must be freed by cdb_free()
This would happen, for example, if we decompressed the record.
Art Cancro [Sun, 20 Aug 2023 03:18:42 +0000 (18:18 -0900)]
Apply patch sent by UFarx to support CPPFLAGS
Art Cancro [Sun, 20 Aug 2023 02:42:31 +0000 (17:42 -0900)]
cdb_fetch()/cdb_next_item() return immutable data
************************************************************************
Before now, the Berkeley DB back end was supplied the DB_DBT_MALLOC flag
and it became the caller's responsibility to handle any memory allocated
and returned by cdb_fetch() and cdb_next_item(). These heuristics are
not expected to be compatible with new back ends currently in planning.
cdb_fetch() and cdb_next_item() now return data that is to be considered
immutable (don't write to it or you'll segfault). The caller must still
call cdb_free() to free the `struct cdbdata` itself, but the pointers
inside it point to buffers that the back end will reuse.
After a call to cdb_fetch() or cdb_next_item(), the buffers are
guaranteed to be readable until the next call to one of those functions
on the same table.
Art Cancro [Sat, 19 Aug 2023 21:34:34 +0000 (12:34 -0900)]
serv_imap.c: moved to new CtdlFetchMsgList() API.
This should be the last of them. Next step is to audit all other use of cdb_free().
Art Cancro [Sat, 19 Aug 2023 20:28:56 +0000 (11:28 -0900)]
serv_imap: removed all the CCC and CCCIMAP shorthand
Art Cancro [Thu, 17 Aug 2023 22:52:54 +0000 (13:52 -0900)]
msgbase.c all migrated to new api now
Art Cancro [Thu, 17 Aug 2023 22:32:56 +0000 (13:32 -0900)]
CtdlFetchMsgList() - treat no-record and zero-messages identically
In both cases, we return 0 to the caller and set the array pointer to NULL.
This simplifies the calling convention and eliminates extra code to check for both conditions
Art Cancro [Thu, 17 Aug 2023 22:23:20 +0000 (13:23 -0900)]
The previous commit was actually for CtdlDeleteMessages()
Art Cancro [Thu, 17 Aug 2023 22:20:49 +0000 (13:20 -0900)]
CtdlSaveMsgPointersInRoom() is now lmdb-safe
Art Cancro [Thu, 17 Aug 2023 22:13:33 +0000 (13:13 -0900)]
serv_nntp.c: move to new API
Art Cancro [Thu, 17 Aug 2023 21:25:48 +0000 (12:25 -0900)]
InitialMailCheck() uses new API
Art Cancro [Thu, 17 Aug 2023 21:11:39 +0000 (12:11 -0900)]
msgbase.c: change to new API
Art Cancro [Wed, 16 Aug 2023 18:14:58 +0000 (09:14 -0900)]
msgbase.c: upgrade CtdlSetSeen() to the new CtdlGetMsgList() API
Art Cancro [Wed, 16 Aug 2023 03:45:24 +0000 (18:45 -0900)]
serv_expire: move to new msglists call
Art Cancro [Wed, 16 Aug 2023 03:14:24 +0000 (18:14 -0900)]
DRY fetch of msglists.
A lot of the calls to cdb_fetch(CDB_MSGLISTS...) follow the same design pattern
and many of them are in need of rework to handle how they free their memory. So
I created a new function CtdlGetMsgList() to do it in a consistent way. Three
callers have been migrated to the new syntax.
Art Cancro [Wed, 16 Aug 2023 02:18:31 +0000 (17:18 -0900)]
serv_fulltext.c: copy the data returned by cdb_fetch() instead of taking ownership
We have to do this everywhere. Berkeley DB can be configured to hand off ownership of memory to
the caller, but LMDB cannot.
Art Cancro [Fri, 11 Aug 2023 20:14:09 +0000 (11:14 -0900)]
More renaming of symbols that are local to the berkeley_db module.
dbp[] is now bdb_table[]
dbenv is now bdb_env
Art Cancro [Fri, 11 Aug 2023 15:46:34 +0000 (06:46 -0900)]
Moved cdb_chmod_data() out of the backend API
This function only does file and directory level operations on
the data/ directory and files in it. These operations will be
common across all backends.
Art Cancro [Fri, 11 Aug 2023 15:21:58 +0000 (06:21 -0900)]
Renamed some more functions and variables that are specific to the bdb_ module.
Art Cancro [Fri, 11 Aug 2023 13:21:45 +0000 (04:21 -0900)]
comments cleanup