Using clone with glibc

[Richard C Bilson]

The project that I am working on depends on access to raw kernel
threads (i.e., not through pthreads). On linux, we use clone to obtain
such threads. Traditionally, this was quite straightforward, but
recent versions of glibc have made this increasingly more complicated.
I'm hoping that some of the knowledgeable people on this list can
suggest some ways of dealing with the issues involved.

It seems unavoidable that a program creating threads with clone must
allocate TLS for the new thread, otherwise errno (and perhaps other
internal libc data) is shared unsafely between threads. This requires
setting up a thread pointer (messing with gs on x86, for instance), and
getting dl_allocate_tls to do the heavy lifting. Am I correct in
believing that all of this is necessary?

More problematic, however, are various bits of the pthread descriptor
that libc relies upon having at fixed offsets from the thread pointer.
At the application level, I don't know how big the thread descriptor
is, nor how to initialize it. There doesn't seem to be a function that
does this sort of minimal initialization. Does one exist, or is there
an easier way to use clone threads with glibc?

[Roland McGrath]

Basically, if you want to call libc functions you should do it from a
thread that was set up by libc or libpthread. i.e., if you make your own
threads with clone, only call libc functions from the initial thread.

The interfaces needed to initialize a thread pointer structure with the
fields libc relies on, even when not using libpthread, are private. You
can indeed call _dl_allocate_tls, but it is in the GLIBC_PRIVATE symbol
version set to indicate that its calling conventions might change or
disappear in any future libc version. The only reason it's accessible at
all is because ld.so, libc, and libpthread share this symbol and the way
ELF works that makes it available to you as well.

This is all the case when libc itself uses TLS. If you build a libc DSO
that does not use TLS itself and do not use libpthread at all, then nothing
will use the thread pointer (%gs segment on x86). For example, on Fedora
systems the normal glibc installation includes multiple different
libc+libpthread DSO sets built in different ways. Any NPTL build requires
TLS. When building glibc with Linuxthreads instead, TLS use can be turned
off, and it is in the Fedora builds. On Fedora, you can use
LD_ASSUME_KERNEL=2.4.19 in the environment to make sure you don't get the
NPTL build of libc for that program run.


Thanks,
Roland