yann@1: File.........: overview.txt
yann@197: Content......: Overview of how crosstool-NG works.
yann@92: Copyrigth....: (C) 2007 Yann E. MORIN <yann.morin.1998@anciens.enib.fr>
yann@192: License......: Creative Commons Attribution Share Alike (CC-by-sa), v2.5
yann@92: 
yann@1: ________________
yann@1:                /
yann@1: Introduction  /
yann@1: _____________/
yann@1: 
yann@1: crosstool-NG aims at building toolchains. Toolchains are an essential component
yann@1: in a software development project. It will compile, assemble and link the code
yann@1: that is being developped. Some pieces of the toolchain will eventually end up
yann@1: in the resulting binary/ies: static libraries are but an example.
yann@1: 
yann@1: So, a toolchain is a very sensitive piece of software, as any bug in one of the
yann@1: components, or a poorly configured component, can lead to execution problems,
yann@1: ranging from poor performance, to applications ending unexpectedly, to
yann@1: mis-behaving software (which more than often is hard to detect), to hardware
yann@1: damage, or even to human risks (which is more than regretable).
yann@1: 
yann@1: Toolchains are made of different piece of software, each being quite complex
yann@1: and requiring specially crafted options to build and work seamlessly. This
yann@1: is usually not that easy, even in the not-so-trivial case of native toolchains.
yann@1: The work reaches a higher degree of complexity when it comes to cross-
yann@40: compilation, where it can become quite a nightmare...
yann@1: 
yann@40: Some cross-toolchains exist on the internet, and can be used for general
yann@1: development, but they have a number of limitations:
yann@1:   - they can be general purpose, in that they are configured for the majority:
yann@1:     no optimisation for your specific target,
yann@1:   - they can be prepared for a specific target and thus are not easy to use,
yann@1:     nor optimised for, or even supporting your target,
yann@1:   - they often are using ageing components (compiler, C library, etc...) not
yann@1:     supporting special features of your shiny new processor;
yann@1: On the other side, these toolchain offer some advantages:
yann@1:   - they are ready to use and quite easy to install and setup,
yann@1:   - they are proven if used by a wide community.
yann@1: 
yann@1: But once you want to get all the juice out of your specific hardware, you will
yann@197: want to build your own toolchain. This is where crosstool-NG comes into play.
yann@1: 
yann@1: There are also a number of tools that builds toolchains for specific needs,
yann@1: which is not really scalable. Examples are:
yann@1:   - buildroot (buildroot.uclibc.org) whose main puprpose is to build root file
yann@1:     systems, hence the name. But once you have your toolchain with buildroot,
yann@1:     part of it is installed in the root-to-be, so if you want to build a whole
yann@1:     new root, you either have to save the existing one as a template and
yann@1:     restore it later, or restart again from scratch. This is not convenient,
yann@1:   - ptxdist (www.pengutronix.de/software/ptxdist), whose purpose is very
yann@1:     similar to buildroot,
yann@1:   - other projects (openembeded.org for example), which is again used to
yann@1:     build root file systems.
yann@1: 
yann@1: crosstool-NG is really targetted at building toolchains, and only toolchains.
yann@1: It is then up to you to use it the way you want.
yann@1: 
yann@1: ___________
yann@1:           /
yann@1: History  /
yann@1: ________/
yann@1: 
yann@1: crosstool was first 'conceived' by Dan Kegel, which offered it to the community,
yann@1: as a set of scripts, a repository of patches, and some pre-configured, general
yann@1: purpose setup files to be used to configure crosstool. This is available at
yann@203: http://www.kegel.com/crosstool, and the subversion repository is hosted on
yann@203: google at http://code.google.com/p/crosstool/.
yann@1: 
yann@1: At the time of writing, crosstool only supports building with one C library,
yann@1: namely glibc, and one C compiler, gcc; it is cripled with historical support
yann@92: for legacy components, and is some kind of a mess to upgrade. Also, submited
yann@203: patches take a loooong time before they are integrated mainline.
yann@1: 
yann@1: I once managed to add support for uClibc-based toolchains, but it did not make
yann@1: into mainline, mostly because I don't have time to port the patch forward to
yann@1: the new versions, due in part to the big effort it was taking.
yann@1: 
yann@1: So I decided to clean up crosstool in the state it was, re-order the things
yann@1: in place, and add appropriate support for what I needed, that is uClibc
yann@203: support. That was a disaster, as inclusion into mainline is slow as hell,
yann@203: and the changes were so numerous.
yann@1: 
yann@1: The only option left to me was rewrite crosstool from scratch. I decided to go
yann@197: this way, and name the new implementation crosstool-NG, standing for crosstool
yann@197: Next Generation, as many other comunity projects do, and as a wink at the TV
yann@197: series "Star Trek: The Next Generation". ;-)
yann@1: 
yann@168: ____________________________
yann@168:                            /
yann@168: Configuring crosstool-NG  /
yann@168: _________________________/
yann@168: 
yann@168: crosstool-NG is configured the same way you configure your Linux kernel: by
yann@168: using a curses-based menu. It is assumed you now how to handle this.
yann@168: 
yann@168: To enter the menu, type:
yann@192:   ct-ng menuconfig
yann@168: 
yann@203: Almost every config item has a help entry. Read them carefully.
yann@168: 
yann@168: String and number options can refer to environment variables. In such a case,
yann@192: you must use the shell syntax: ${VAR}. You shall neither single- nor double-
yann@192: quote the string options.
yann@168: 
yann@192: There are three environment variables that are computed by crosstool-NG, and
yann@168: that you can use:
yann@168: 
yann@168: CT_TARGET:
yann@168:   It represents the target triplet you are building for. You can use it for
yann@168:   example in the installation/prefix directory, such as:
yann@168:     /opt/x-tools/${CT_TARGET}
yann@168: 
yann@168: CT_TOP_DIR:
yann@182:   The top directory where crosstool-NG is running. You shouldn't need it in
yann@182:   most cases. There is one case where you may need it: if you have local
yann@182:   patches and you store them in your running directory, you can refer to them
yann@168:   by using CT_TOP_DIR, such as:
yann@168:     ${CT_TOP_DIR}/patches.myproject
yann@168: 
yann@168: CT_VERSION:
yann@192:   The version of crosstool-NG you are using. Not much use for you, but it's
yann@168:   there if you need it.
yann@168: 
yann@168: 
yann@168: Interesting config options |
yann@168: ---------------------------*
yann@168: 
yann@168: CT_LOCAL_TARBALLS_DIR:
yann@168:   If you already have sone tarballs in a direcotry, enter it here. That will
yann@197:   speed up the retrieving phase, where crosstool-NG would otherwise download
yann@168:   those tarballs.
yann@168: 
yann@168: CT_PREFIX_DIR:
yann@168:   This is where the toolchain will be installed in (and for now, where it
yann@168:   will run from).
yann@168: 
yann@168: CT_TARGET_VENDOR:
yann@168:   An identifier for your toolchain, will take place in the vendor part of the
yann@168:   target triplet. It shall *not* contain spaces or dashes. Usually, keep it
yann@168:   to a one-word string, or use underscores to separate words if you need.
yann@168:   Avoid dots, commas, and special characters.
yann@168: 
yann@168: CT_TARGET_ALIAS:
yann@168:   An alias for the toolchian. It will be used as a prefix to the toolchain
yann@168:   tools. For example, you will have ${CT_TARGET_ALIAS}-gcc
yann@168: 
yann@246: Also, if you think you don't see enough versions, you can try to enable one of
yann@246: those:
yann@246: 
yann@246: CT_OBSOLETE:
yann@246:   Show obsolete versions or tools. Most of the time, you don't want to base
yann@246:   your toolchain on too old a version (of gcc, for example). But at times, it
yann@246:   can come handy to use such an old version for regression tests. Those old
yann@246:   versions are hidden behind CT_BSOLETE.
yann@246: 
yann@246: CT_EXPERIMENTAL:
yann@246:   Show experimental versions or tools. Again, you might not want to base your
yann@246:   toolchain on too recent tools (eg. gcc) for production. But if you need a
yann@246:   feature present only in a recent version, or a new tool, you can find them
yann@246:   hidden behind CT_EXPERIMENTAL.
yann@246: 
yann@246: CT_BROKEN:
yann@246:   Show broken versions or tools. Some usefull tools are currently broken: they
yann@246:   won't compile, run, or worse, cause defects when running. But if you are
yann@246:   brave enough, you can try and debug them. They are hidden behind CT_BROKEN,
yann@246:   which itself is hiddent behind EXPERIMENTAL.
yann@246: 
yann@276: Re-building an existing toolchain |
yann@276: ----------------------------------+
yann@276: 
yann@276: If you have an existing toolchain, you can re-use the options used to build it
yann@276: to create a new toolchain. That needs a very little bit of effort on your side
yann@276: but is quite easy. The options to build a toolchain are saved in the build log
yann@276: file that is saved within the toolchain. crosstool-NG can extract those options
yann@276: to recreate a new configuration:
yann@276:   ct-ng extractconfig </path/to/your/build.log
yann@276: 
yann@276: will extract those options, prompt you for the new ones, which you can later
yann@276: edit with menuconfig.
yann@276: 
yann@276: Of course, if your build log was compressed, you'd have to use something like:
yann@276:   bzcat /path/to/your/build.log.bz2 |ct-ng extractconfig
yann@276: 
yann@168: ________________________
yann@168:                        /
yann@168: Running crosstool-NG  /
yann@168: _____________________/
yann@1: 
yann@197: crosstool-NG is configured by a configurator presenting a menu-stuctured set of
yann@1: options. These options let you specify the way you want your toolchain built,
yann@1: where you want it installed, what architecture and specific processor it
yann@1: will support, the version of the components you want to use, etc... The
yann@1: value for those options are then stored in a configuration file.
yann@1: 
yann@168: To build the toolchain, simply type:
yann@203:   ct-ng build
yann@135: 
yann@135: This will use the above configuration to retrieve, extract and patch the
yann@135: components, build, install and eventually test your newly built toolchain.
yann@1: 
yann@1: You are then free to add the toolchain /bin directory in your PATH to use
yann@1: it at will.
yann@1: 
yann@135: In any case, you can get some terse help. Just type:
yann@192:   ct-ng help
yann@203: or:
yann@203:   man 1 ct-ng
yann@135: 
yann@135: 
yann@135: Stoping and restarting a build |
yann@135: -------------------------------*
yann@135: 
yann@135: If you want to stop the build after a step you are debugging, you can pass the
yann@135: variable STOP to make:
yann@192:   ct-ng STOP=some_step
yann@135: 
yann@135: Conversely, if you want to restart a build at a specific step you are
yann@135: debugging, you can pass the RESTART variable to make:
yann@192:   ct-ng RESTART=some_step
yann@135: 
yann@136: Alternatively, you can call make with the name of a step to just do that step:
yann@192:   ct-ng libc_headers
yann@136: is equivalent to:
yann@192:   ct-ng RESTART=libs_headers STOP=libc_headers
yann@136: 
yann@136: The shortcuts -step_name and step_name- allow to respectively stop or restart
yann@136: at that step. Thus:
yann@192:   ct-ng -libc_headers        and:    ct-ng libc_headers-
yann@136: are equivalent to:
yann@192:   ct-ng STOP=libc_headers    and:    ct-ng RESTART=libc_headers
yann@136: 
yann@181: To obtain the list of acceptable steps, please call:
yann@192:   ct-ng liststeps
yann@181: 
yann@168: Note that in order to restart a build, you'll have to say 'Y' to the config
yann@168: option CT_DEBUG_CT_SAVE_STEPS, and that the previous build effectively went
yann@168: that far.
yann@92: 
yann@92: 
yann@168: Testing all toolchains at once |
yann@168: -------------------------------*
yann@92: 
yann@168: You can test-build all samples; simply call:
yann@192:   ct-ng regtest
yann@40: 
yann@227: _______________________
yann@227:                       /
yann@227: Using the toolchain  /
yann@227: ____________________/
yann@227: 
yann@227: Using the toolchain is as simple as adding the toolchain's bin directory in
yann@227: your PATH, such as:
yann@227:   export PATH="${PATH}:/your/toolchain/path/bin"
yann@227: 
yann@227: and then using the target triplet to tell the build systems to use your
yann@227: toolchain:
yann@227:   ./configure --target=your-target-triplet
yann@227:   make CC=your-target-triplet-gcc
yann@227:   make CROSS_COMPILE=your-target-triplet-
yann@227:   and so on...
yann@227: 
yann@227: When your root directory is ready, it is still missing some important bits: the
yann@227: toolchain's libraries. To populate your root directory with those libs, just
yann@227: run:
yann@227:   your-target-triplet-populate -s /your/root -d /your/root-populated
yann@227: 
yann@227: This will copy /your/root into /your/root-populated, and put the needed and only
yann@227: the needed libraries there. Thus you don't polute /your/root with any cruft that
yann@227: would no longer be needed should you have to remove stuff. /your/root always
yann@227: contains only those things you install in it.
yann@227: 
yann@227: You can then use /your/root-populated to build up your file system image, a
yann@227: tarball, or to NFS-mount it from your target, or whatever you need.
yann@227: 
yann@40: ___________________
yann@40:                   /
yann@40: Toolchain types  /
yann@40: ________________/
yann@40: 
yann@40: There are four kinds of toolchains you could encounter.
yann@40: 
yann@40: First off, you must understand the following: when it comes to compilers there
yann@40: are up to four machines involved:
yann@40:   1) the machine configuring the toolchain components: the config machine
yann@40:   2) the machine building the toolchain components:    the build machine
yann@40:   3) the machine running the toolchain:                the host machine
yann@203:   4) the machine the toolchain is generating code for: the target machine
yann@40: 
yann@40: We can most of the time assume that the config machine and the build machine
yann@40: are the same. Most of the time, this will be true. The only time it isn't
yann@40: is if you're using distributed compilation (such as distcc). Let's forget
yann@40: this for the sake of simplicity.
yann@40: 
yann@40: So we're left with three machines:
yann@40:  - build
yann@40:  - host
yann@40:  - target
yann@40: 
yann@40: Any toolchain will involve those three machines. You can be as pretty sure of
yann@40: this as "2 and 2 are 4". Here is how they come into play:
yann@40: 
yann@40: 1) build == host == target
yann@40:     This is a plain native toolchain, targetting the exact same machine as the
yann@40:     one it is built on, and running again on this exact same machine. You have
yann@40:     to build such a toolchain when you want to use an updated component, such
yann@40:     as a newer gcc for example.
yann@197:     crosstool-NG calls it "native".
yann@40: 
yann@40: 2) build == host != target
yann@40:     This is a classic cross-toolchain, which is expected to be run on the same
yann@40:     machine it is compiled on, and generate code to run on a second machine,
yann@40:     the target.
yann@197:     crosstool-NG calls it "cross".
yann@40: 
yann@40: 3) build != host == target
yann@40:     Such a toolchain is also a native toolchain, as it targets the same machine
yann@40:     as it runs on. But it is build on another machine. You want such a
yann@40:     toolchain when porting to a new architecture, or if the build machine is
yann@40:     much faster than the host machine.
yann@197:     crosstool-NG calls it "cross-native".
yann@40: 
yann@40: 4) build != host != target
yann@92:     This one is called a canadian-toolchain (*), and is tricky. The three
yann@40:     machines in play are different. You might want such a toolchain if you
yann@40:     have a fast build machine, but the users will use it on another machine,
yann@40:     and will produce code to run on a third machine.
yann@197:     crosstool-NG calls it "canadian".
yann@40: 
yann@197: crosstool-NG can build all these kinds of toolchains (or is aiming at it,
yann@197: anyway!)
yann@40: 
yann@40: (*) The term Canadian Cross came about because at the time that these issues
yann@40:     were all being hashed out, Canada had three national political parties.
yann@40:     http://en.wikipedia.org/wiki/Cross_compiler
yann@40: 
yann@1: _____________
yann@1:             /
yann@1: Internals  /
yann@1: __________/
yann@1: 
yann@92: Internally, crosstool-NG is script-based. To ease usage, the frontend is
yann@92: Makefile-based.
yann@92: 
yann@92: Makefile front-end |
yann@92: -------------------*
yann@92: 
yann@203: The entry point to crosstool-NG is the Makefile script "ct-ng". Calling this
yann@203: script with an action will act exactly as if the Makefile was in the current
yann@203: working directory and make was called with the action as rule. Thus:
yann@203:   ct-ng menuconfig
yann@203: is equivalent to having the Makefile in CWD, and calling:
yann@203:   make menuconfig
yann@203: 
yann@203: Having ct-ng as it is avoids copying the Makefile everywhere, and acts as a
yann@203: traditional command.
yann@203: 
yann@203: ct-ng loads sub- Makefiles from the library directory $(CT_LIB_DIR), as set up
yann@203: at configuration time with ./configure.
yann@203: 
yann@203: ct-ng also search for config files, sub-tools, samples, scripts and patches in
yann@203: that library directory.
yann@92: 
yann@182: Kconfig parser |
yann@182: ---------------*
yann@92: 
yann@92: The kconfig language is a hacked version, vampirised from the toybox project
yann@182: by Rob LANDLEY (http://www.landley.net/code/toybox/), itself coming from the
yann@182: Linux kernel (http://www.linux.org/ http://www.kernel.org/), and (heavily)
yann@182: adapted to my needs.
yann@92: 
yann@203: The kconfig parsers (conf and mconf) are not installed pre-built, but as
yann@203: source files. Thus you can have the directory where crosstool-NG is installed,
yann@203: exported (via NFS or whatever) and have clients with different architectures
yann@203: use the same crosstool-NG installation, and most notably, the same set of
yann@203: patches.
yann@203: 
yann@203: Build scripts |
yann@203: --------------*
yann@203: 
yann@203: To Be Written later...