Abstract
The talk will be about the CGo-free database/sql driver for SQLite
(modernc.org/sqlite) and the related technologies it relies upon.
Namely the C to Go compiler/transpiler (modernc.org/ccgo/v4) and the runtime
support emulating the C libc (modernc.org/libc).
In the SQLite driver part the talk will depict the testing framework that
verifies new releases of the Go package. It's a combination of manually written
tests in Go and the publicly available suite of Tcl programs from the official
sqlite.org repository.
The ccgo transpiler part will be about the interesting challenges in
reproducing C semantics in Go and how the different solutions were tested and
gradually adjusted. The transpilation is not universally working for any and
all C programs. There are some limitations and I will talk what probably cannot
be done at all, what works only to some extent and where the semantics
intentionally differs. That's the case of the threading model. The transpiled
code models C threads as Go goroutines. It is a trade-off that suits some,
perhaps many C programs well, but in the same time makes other transpiled C
programs possibly misbehave. It is often possible to solve this problem by
patching the original C sources in preparation to the transpilation and some
examples for that can be presented.
The latest ccgo version can now semi-automatically discover the transpiled Go
packages for C libraries that are mentioned in the -l flag in C compilation,
like in -lfoo that links against libfoo.a or libfoo.so. Effectively bringing a
kind-of package manager system for the C to Go transpiled programs. I find this
interesting and I'm not aware of a precedence/example of a similar feature from
elsewhere. The same probably holds with respect to ccgo now having its own
object file format, enabling incrementally putting together the results of
several C translation units. In other words, ccgo is also a linker that can
produce compilable .go files as a result of combining multiple ccgo object
files (extension .o.go) and/or ccgo archives (extension .ago). This enhances
the transpiler possibilities. More so when an already linked Go package can be
used as a linker object as well. This way the C to Go transpilled code does not
have to incorporate all of the original C code, but it can simply import the
reusable parts of it the same way usual Go programs - or dynamically linked C
programs - do.
In the libc part I will again talk about the technical challenges and the
adopted solutions, what I consider interesting about some of them. For
example, some C programs, unfortunately, assume execution environment that is
not existing in the C standard. One particular example in Tcl will be discussed
together how it influenced the Go libc implementation, which was effectively
forced to use a less efficient solution to a particular problem just to emulate
things the way Tcl C code expects. Even when technically it's undefined
behaviour, in C parlance.
I want to bring some attention to the state of libc. It does not support all
platforms/architectures as well as some others. Namely Linux is the platform
that received so far most of the work and that platform is now almost entirely
transpiled from musl-libc. I think it would benefit the Go community if we can
find more modernc.org/libc maintainers knowing well the less supported targets.
The situation is slowly improving and some individuals/companies are stepping
forward and putting in their resources. I want to use the talk to encourage
some/any of the big players in this field to help the progress by partially
allocating people and/or HW or VM builders, particularly when not available or
not working yet reliably in qemu(1).
In the talk's conclusions I will summarize the pros and cons of the
transpilation approach. In short, it's a trade-off between lower run time
performance and easy cross-compilation that Go so nicely provides for CGo-free
code. Which side of the coin is more important is naturally a per-case
decision, but it seems like the transpilation way has proven itself useful.
There are 1000+ known importers of the package as of the time of this writing.
