File.........: overview.txt

 Content......: Overview of how crosstool-NG works.

 Copyrigth....: (C) 2007 Yann E. MORIN <yann.morin.1998@anciens.enib.fr>

 License......: Creative Commons Attribution Share Alike (CC-by-sa), v2.5

 ________________

                /

 Introduction  /

 _____________/

 crosstool-NG aims at building toolchains. Toolchains are an essential component

 in a software development project. It will compile, assemble and link the code

 that is being developped. Some pieces of the toolchain will eventually end up

 in the resulting binary/ies: static libraries are but an example.

 So, a toolchain is a very sensitive piece of software, as any bug in one of the

 components, or a poorly configured component, can lead to execution problems,

 ranging from poor performance, to applications ending unexpectedly, to

 mis-behaving software (which more than often is hard to detect), to hardware

 damage, or even to human risks (which is more than regretable).

 Toolchains are made of different piece of software, each being quite complex

 and requiring specially crafted options to build and work seamlessly. This

 is usually not that easy, even in the not-so-trivial case of native toolchains.

 The work reaches a higher degree of complexity when it comes to cross-

 compilation, where it can become quite a nightmare...

 Some cross-toolchains exist on the internet, and can be used for general

 development, but they have a number of limitations:

   - they can be general purpose, in that they are configured for the majority:

     no optimisation for your specific target,

   - they can be prepared for a specific target and thus are not easy to use,

     nor optimised for, or even supporting your target,

   - they often are using ageing components (compiler, C library, etc...) not

     supporting special features of your shiny new processor;

 On the other side, these toolchain offer some advantages:

   - they are ready to use and quite easy to install and setup,

   - they are proven if used by a wide community.

 But once you want to get all the juice out of your specific hardware, you will

 want to build your own toolchain. This is where crosstool-NG comes into play.

 There are also a number of tools that builds toolchains for specific needs,

 which is not really scalable. Examples are:

   - buildroot (buildroot.uclibc.org) whose main puprpose is to build root file

     systems, hence the name. But once you have your toolchain with buildroot,

     part of it is installed in the root-to-be, so if you want to build a whole

     new root, you either have to save the existing one as a template and

     restore it later, or restart again from scratch. This is not convenient,

   - ptxdist (www.pengutronix.de/software/ptxdist), whose purpose is very

     similar to buildroot,

   - other projects (openembeded.org for example), which is again used to

     build root file systems.

 crosstool-NG is really targetted at building toolchains, and only toolchains.

 It is then up to you to use it the way you want.

 ___________

           /

 History  /

 ________/

 crosstool was first 'conceived' by Dan Kegel, which offered it to the community,

 as a set of scripts, a repository of patches, and some pre-configured, general

 purpose setup files to be used to configure crosstool. This is available at

 http://www.kegel.com/crosstool, and the subversion repository is hosted on

 google at http://code.google.com/p/crosstool/.

 At the time of writing, crosstool only supports building with one C library,

 namely glibc, and one C compiler, gcc; it is cripled with historical support

 for legacy components, and is some kind of a mess to upgrade. Also, submited

 patches take a loooong time before they are integrated mainline.

 I once managed to add support for uClibc-based toolchains, but it did not make

 into mainline, mostly because I don't have time to port the patch forward to

 the new versions, due in part to the big effort it was taking.

 So I decided to clean up crosstool in the state it was, re-order the things

 in place, and add appropriate support for what I needed, that is uClibc

 support. That was a disaster, as inclusion into mainline is slow as hell,

 and the changes were so numerous.

 The only option left to me was rewrite crosstool from scratch. I decided to go

 this way, and name the new implementation crosstool-NG, standing for crosstool

 Next Generation, as many other comunity projects do, and as a wink at the TV

 series "Star Trek: The Next Generation". ;-)

 ____________________________

                            /

 Configuring crosstool-NG  /

 _________________________/

 crosstool-NG is configured the same way you configure your Linux kernel: by

 using a curses-based menu. It is assumed you now how to handle this.

 To enter the menu, type:

   ct-ng menuconfig

 Almost every config item has a help entry. Read them carefully.

 String and number options can refer to environment variables. In such a case,

 you must use the shell syntax: ${VAR}. You shall neither single- nor double-

 quote the string options.

 There are three environment variables that are computed by crosstool-NG, and

 that you can use:

 CT_TARGET:

   It represents the target triplet you are building for. You can use it for

   example in the installation/prefix directory, such as:

     /opt/x-tools/${CT_TARGET}

 CT_TOP_DIR:

   The top directory where crosstool-NG is running. You shouldn't need it in

   most cases. There is one case where you may need it: if you have local

   patches and you store them in your running directory, you can refer to them

   by using CT_TOP_DIR, such as:

     ${CT_TOP_DIR}/patches.myproject

 CT_VERSION:

   The version of crosstool-NG you are using. Not much use for you, but it's

   there if you need it.

 Interesting config options |

 ---------------------------*

 CT_LOCAL_TARBALLS_DIR:

   If you already have sone tarballs in a direcotry, enter it here. That will

   speed up the retrieving phase, where crosstool-NG would otherwise download

   those tarballs.

 CT_PREFIX_DIR:

   This is where the toolchain will be installed in (and for now, where it

   will run from).

 CT_TARGET_VENDOR:

   An identifier for your toolchain, will take place in the vendor part of the

   target triplet. It shall *not* contain spaces or dashes. Usually, keep it

   to a one-word string, or use underscores to separate words if you need.

   Avoid dots, commas, and special characters.

 CT_TARGET_ALIAS:

   An alias for the toolchian. It will be used as a prefix to the toolchain

   tools. For example, you will have ${CT_TARGET_ALIAS}-gcc

 Also, if you think you don't see enough versions, you can try to enable one of

 those:

 CT_OBSOLETE:

   Show obsolete versions or tools. Most of the time, you don't want to base

   your toolchain on too old a version (of gcc, for example). But at times, it

   can come handy to use such an old version for regression tests. Those old

   versions are hidden behind CT_BSOLETE.

 CT_EXPERIMENTAL:

   Show experimental versions or tools. Again, you might not want to base your

   toolchain on too recent tools (eg. gcc) for production. But if you need a

   feature present only in a recent version, or a new tool, you can find them

   hidden behind CT_EXPERIMENTAL.

 CT_BROKEN:

   Show broken versions or tools. Some usefull tools are currently broken: they

   won't compile, run, or worse, cause defects when running. But if you are

   brave enough, you can try and debug them. They are hidden behind CT_BROKEN,

   which itself is hiddent behind EXPERIMENTAL.

 Re-building an existing toolchain |

 ----------------------------------+

 If you have an existing toolchain, you can re-use the options used to build it

 to create a new toolchain. That needs a very little bit of effort on your side

 but is quite easy. The options to build a toolchain are saved in the build log

 file that is saved within the toolchain. crosstool-NG can extract those options

 to recreate a new configuration:

   ct-ng extractconfig </path/to/your/build.log

 will extract those options, prompt you for the new ones, which you can later

 edit with menuconfig.

 Of course, if your build log was compressed, you'd have to use something like:

   bzcat /path/to/your/build.log.bz2 |ct-ng extractconfig

 ________________________

                        /

 Running crosstool-NG  /

 _____________________/

 crosstool-NG is configured by a configurator presenting a menu-stuctured set of

 options. These options let you specify the way you want your toolchain built,

 where you want it installed, what architecture and specific processor it

 will support, the version of the components you want to use, etc... The

 value for those options are then stored in a configuration file.

 To build the toolchain, simply type:

   ct-ng build

 This will use the above configuration to retrieve, extract and patch the

 components, build, install and eventually test your newly built toolchain.

 You are then free to add the toolchain /bin directory in your PATH to use

 it at will.

 In any case, you can get some terse help. Just type:

   ct-ng help

 or:

   man 1 ct-ng

 Stoping and restarting a build |

 -------------------------------*

 If you want to stop the build after a step you are debugging, you can pass the

 variable STOP to make:

   ct-ng STOP=some_step

 Conversely, if you want to restart a build at a specific step you are

 debugging, you can pass the RESTART variable to make:

   ct-ng RESTART=some_step

 Alternatively, you can call make with the name of a step to just do that step:

   ct-ng libc_headers

 is equivalent to:

   ct-ng RESTART=libs_headers STOP=libc_headers

 The shortcuts -step_name and step_name- allow to respectively stop or restart

 at that step. Thus:

   ct-ng -libc_headers        and:    ct-ng libc_headers-

 are equivalent to:

   ct-ng STOP=libc_headers    and:    ct-ng RESTART=libc_headers

 To obtain the list of acceptable steps, please call:

   ct-ng liststeps

 Note that in order to restart a build, you'll have to say 'Y' to the config

 option CT_DEBUG_CT_SAVE_STEPS, and that the previous build effectively went

 that far.

 Testing all toolchains at once |

 -------------------------------*

 You can test-build all samples; simply call:

   ct-ng regtest

 _______________________

                       /

 Using the toolchain  /

 ____________________/

 Using the toolchain is as simple as adding the toolchain's bin directory in

 your PATH, such as:

   export PATH="${PATH}:/your/toolchain/path/bin"

 and then using the target triplet to tell the build systems to use your

 toolchain:

   ./configure --target=your-target-triplet

   make CC=your-target-triplet-gcc

   make CROSS_COMPILE=your-target-triplet-

   and so on...

 When your root directory is ready, it is still missing some important bits: the

 toolchain's libraries. To populate your root directory with those libs, just

 run:

   your-target-triplet-populate -s /your/root -d /your/root-populated

 This will copy /your/root into /your/root-populated, and put the needed and only

 the needed libraries there. Thus you don't polute /your/root with any cruft that

 would no longer be needed should you have to remove stuff. /your/root always

 contains only those things you install in it.

 You can then use /your/root-populated to build up your file system image, a

 tarball, or to NFS-mount it from your target, or whatever you need.

 ___________________

                   /

 Toolchain types  /

 ________________/

 There are four kinds of toolchains you could encounter.

 First off, you must understand the following: when it comes to compilers there

 are up to four machines involved:

   1) the machine configuring the toolchain components: the config machine

   2) the machine building the toolchain components:    the build machine

   3) the machine running the toolchain:                the host machine

   4) the machine the toolchain is generating code for: the target machine

 We can most of the time assume that the config machine and the build machine

 are the same. Most of the time, this will be true. The only time it isn't

 is if you're using distributed compilation (such as distcc). Let's forget

 this for the sake of simplicity.

 So we're left with three machines:

  - build

  - host

  - target

 Any toolchain will involve those three machines. You can be as pretty sure of

 this as "2 and 2 are 4". Here is how they come into play:

 1) build == host == target

     This is a plain native toolchain, targetting the exact same machine as the

     one it is built on, and running again on this exact same machine. You have

     to build such a toolchain when you want to use an updated component, such

     as a newer gcc for example.

     crosstool-NG calls it "native".

 2) build == host != target

     This is a classic cross-toolchain, which is expected to be run on the same

     machine it is compiled on, and generate code to run on a second machine,

     the target.

     crosstool-NG calls it "cross".

 3) build != host == target

     Such a toolchain is also a native toolchain, as it targets the same machine

     as it runs on. But it is build on another machine. You want such a

     toolchain when porting to a new architecture, or if the build machine is

     much faster than the host machine.

     crosstool-NG calls it "cross-native".

 4) build != host != target

     This one is called a canadian-toolchain (*), and is tricky. The three

     machines in play are different. You might want such a toolchain if you

     have a fast build machine, but the users will use it on another machine,

     and will produce code to run on a third machine.

     crosstool-NG calls it "canadian".

 crosstool-NG can build all these kinds of toolchains (or is aiming at it,

 anyway!)

 (*) The term Canadian Cross came about because at the time that these issues

     were all being hashed out, Canada had three national political parties.

     http://en.wikipedia.org/wiki/Cross_compiler

 _____________

             /

 Internals  /

 __________/

 Internally, crosstool-NG is script-based. To ease usage, the frontend is

 Makefile-based.

 Makefile front-end |

 -------------------*

 The entry point to crosstool-NG is the Makefile script "ct-ng". Calling this

 script with an action will act exactly as if the Makefile was in the current

 working directory and make was called with the action as rule. Thus:

   ct-ng menuconfig

 is equivalent to having the Makefile in CWD, and calling:

   make menuconfig

 Having ct-ng as it is avoids copying the Makefile everywhere, and acts as a

 traditional command.

 ct-ng loads sub- Makefiles from the library directory $(CT_LIB_DIR), as set up

 at configuration time with ./configure.

 ct-ng also search for config files, sub-tools, samples, scripts and patches in

 that library directory.

 Kconfig parser |

 ---------------*

 The kconfig language is a hacked version, vampirised from the toybox project

 by Rob LANDLEY (http://www.landley.net/code/toybox/), itself coming from the

 Linux kernel (http://www.linux.org/ http://www.kernel.org/), and (heavily)

 adapted to my needs.

 The kconfig parsers (conf and mconf) are not installed pre-built, but as

 source files. Thus you can have the directory where crosstool-NG is installed,

 exported (via NFS or whatever) and have clients with different architectures

 use the same crosstool-NG installation, and most notably, the same set of

 patches.

 Build scripts |

 --------------*

 To Be Written later...