docs/overview.txt
author "Yann E. MORIN" <yann.morin.1998@anciens.enib.fr>
Tue Jul 17 21:49:43 2007 +0000 (2007-07-17)
changeset 277 4c32aa5569b2
parent 276 835178a32fc1
child 294 7947c33280e8
permissions -rw-r--r--
Re-order and rework a bit docs/overview.txt. Plus a typo fix.
     1 File.........: overview.txt
     2 Content......: Overview of how crosstool-NG works.
     3 Copyrigth....: (C) 2007 Yann E. MORIN <yann.morin.1998@anciens.enib.fr>
     4 License......: Creative Commons Attribution Share Alike (CC-by-sa), v2.5
     5 
     6 ________________
     7                /
     8 Introduction  /
     9 _____________/
    10 
    11 crosstool-NG aims at building toolchains. Toolchains are an essential component
    12 in a software development project. It will compile, assemble and link the code
    13 that is being developped. Some pieces of the toolchain will eventually end up
    14 in the resulting binary/ies: static libraries are but an example.
    15 
    16 So, a toolchain is a very sensitive piece of software, as any bug in one of the
    17 components, or a poorly configured component, can lead to execution problems,
    18 ranging from poor performance, to applications ending unexpectedly, to
    19 mis-behaving software (which more than often is hard to detect), to hardware
    20 damage, or even to human risks (which is more than regretable).
    21 
    22 Toolchains are made of different piece of software, each being quite complex
    23 and requiring specially crafted options to build and work seamlessly. This
    24 is usually not that easy, even in the not-so-trivial case of native toolchains.
    25 The work reaches a higher degree of complexity when it comes to cross-
    26 compilation, where it can become quite a nightmare...
    27 
    28 Some cross-toolchains exist on the internet, and can be used for general
    29 development, but they have a number of limitations:
    30   - they can be general purpose, in that they are configured for the majority:
    31     no optimisation for your specific target,
    32   - they can be prepared for a specific target and thus are not easy to use,
    33     nor optimised for, or even supporting your target,
    34   - they often are using ageing components (compiler, C library, etc...) not
    35     supporting special features of your shiny new processor;
    36 On the other side, these toolchain offer some advantages:
    37   - they are ready to use and quite easy to install and setup,
    38   - they are proven if used by a wide community.
    39 
    40 But once you want to get all the juice out of your specific hardware, you will
    41 want to build your own toolchain. This is where crosstool-NG comes into play.
    42 
    43 There are also a number of tools that builds toolchains for specific needs,
    44 which is not really scalable. Examples are:
    45   - buildroot (buildroot.uclibc.org) whose main puprpose is to build root file
    46     systems, hence the name. But once you have your toolchain with buildroot,
    47     part of it is installed in the root-to-be, so if you want to build a whole
    48     new root, you either have to save the existing one as a template and
    49     restore it later, or restart again from scratch. This is not convenient,
    50   - ptxdist (www.pengutronix.de/software/ptxdist), whose purpose is very
    51     similar to buildroot,
    52   - other projects (openembeded.org for example), which is again used to
    53     build root file systems.
    54 
    55 crosstool-NG is really targetted at building toolchains, and only toolchains.
    56 It is then up to you to use it the way you want.
    57 
    58 ___________
    59           /
    60 History  /
    61 ________/
    62 
    63 crosstool was first 'conceived' by Dan Kegel, which offered it to the community,
    64 as a set of scripts, a repository of patches, and some pre-configured, general
    65 purpose setup files to be used to configure crosstool. This is available at
    66 http://www.kegel.com/crosstool, and the subversion repository is hosted on
    67 google at http://code.google.com/p/crosstool/.
    68 
    69 At the time of writing, crosstool only supports building with one C library,
    70 namely glibc, and one C compiler, gcc; it is cripled with historical support
    71 for legacy components, and is some kind of a mess to upgrade. Also, submited
    72 patches take a loooong time before they are integrated mainline.
    73 
    74 I once managed to add support for uClibc-based toolchains, but it did not make
    75 into mainline, mostly because I don't have time to port the patch forward to
    76 the new versions, due in part to the big effort it was taking.
    77 
    78 So I decided to clean up crosstool in the state it was, re-order the things
    79 in place, and add appropriate support for what I needed, that is uClibc
    80 support. That was a disaster, as inclusion into mainline is slow as hell,
    81 and the changes were so numerous.
    82 
    83 The only option left to me was rewrite crosstool from scratch. I decided to go
    84 this way, and name the new implementation crosstool-NG, standing for crosstool
    85 Next Generation, as many other comunity projects do, and as a wink at the TV
    86 series "Star Trek: The Next Generation". ;-)
    87 
    88 ____________________________
    89                            /
    90 Configuring crosstool-NG  /
    91 _________________________/
    92 
    93 crosstool-NG is configured by a configurator presenting a menu-stuctured set of
    94 options. These options let you specify the way you want your toolchain built,
    95 where you want it installed, what architecture and specific processor it
    96 will support, the version of the components you want to use, etc... The
    97 value for those options are then stored in a configuration file.
    98 
    99 The configurator works the same way you configure your Linux kernel.It is
   100 assumed you now how to handle this.
   101 
   102 To enter the menu, type:
   103   ct-ng menuconfig
   104 
   105 Almost every config item has a help entry. Read them carefully.
   106 
   107 String and number options can refer to environment variables. In such a case,
   108 you must use the shell syntax: ${VAR}. You shall neither single- nor double-
   109 quote the string options.
   110 
   111 There are three environment variables that are computed by crosstool-NG, and
   112 that you can use:
   113 
   114 CT_TARGET:
   115   It represents the target triplet you are building for. You can use it for
   116   example in the installation/prefix directory, such as:
   117     /opt/x-tools/${CT_TARGET}
   118 
   119 CT_TOP_DIR:
   120   The top directory where crosstool-NG is running. You shouldn't need it in
   121   most cases. There is one case where you may need it: if you have local
   122   patches and you store them in your running directory, you can refer to them
   123   by using CT_TOP_DIR, such as:
   124     ${CT_TOP_DIR}/patches.myproject
   125 
   126 CT_VERSION:
   127   The version of crosstool-NG you are using. Not much use for you, but it's
   128   there if you need it.
   129 
   130 
   131 Interesting config options |
   132 ---------------------------*
   133 
   134 CT_LOCAL_TARBALLS_DIR:
   135   If you already have some tarballs in a direcotry, enter it here. That will
   136   speed up the retrieving phase, where crosstool-NG would otherwise download
   137   those tarballs.
   138 
   139 CT_PREFIX_DIR:
   140   This is where the toolchain will be installed in (and for now, where it
   141   will run from). Common use it to add the target triplet in the directory
   142   path, such as (see above):
   143     /opt/x-tools/${CT_TARGET}
   144 
   145 CT_TARGET_VENDOR:
   146   An identifier for your toolchain, will take place in the vendor part of the
   147   target triplet. It shall *not* contain spaces or dashes. Usually, keep it
   148   to a one-word string, or use underscores to separate words if you need.
   149   Avoid dots, commas, and special characters.
   150 
   151 CT_TARGET_ALIAS:
   152   An alias for the toolchian. It will be used as a prefix to the toolchain
   153   tools. For example, you will have ${CT_TARGET_ALIAS}-gcc
   154 
   155 Also, if you think you don't see enough versions, you can try to enable one of
   156 those:
   157 
   158 CT_OBSOLETE:
   159   Show obsolete versions or tools. Most of the time, you don't want to base
   160   your toolchain on too old a version (of gcc, for example). But at times, it
   161   can come handy to use such an old version for regression tests. Those old
   162   versions are hidden behind CT_BSOLETE.
   163 
   164 CT_EXPERIMENTAL:
   165   Show experimental versions or tools. Again, you might not want to base your
   166   toolchain on too recent tools (eg. gcc) for production. But if you need a
   167   feature present only in a recent version, or a new tool, you can find them
   168   hidden behind CT_EXPERIMENTAL.
   169 
   170 CT_BROKEN:
   171   Show broken versions or tools. Some usefull tools are currently broken: they
   172   won't compile, run, or worse, cause defects when running. But if you are
   173   brave enough, you can try and debug them. They are hidden behind CT_BROKEN,
   174   which itself is hiddent behind EXPERIMENTAL.
   175 
   176 Re-building an existing toolchain |
   177 ----------------------------------+
   178 
   179 If you have an existing toolchain, you can re-use the options used to build it
   180 to create a new toolchain. That needs a very little bit of effort on your side
   181 but is quite easy. The options to build a toolchain are saved in the build log
   182 file that is saved within the toolchain. crosstool-NG can extract those options
   183 to recreate a new configuration:
   184   ct-ng extractconfig </path/to/your/build.log
   185 
   186 will extract those options, prompt you for the new ones, which you can later
   187 edit with menuconfig.
   188 
   189 Of course, if your build log was compressed, you'd have to use something like:
   190   bzcat /path/to/your/build.log.bz2 |ct-ng extractconfig
   191 
   192 ________________________
   193                        /
   194 Running crosstool-NG  /
   195 _____________________/
   196 
   197 To build the toolchain, simply type:
   198   ct-ng build
   199 
   200 This will use the above configuration to retrieve, extract and patch the
   201 components, build, install and eventually test your newly built toolchain.
   202 
   203 You are then free to add the toolchain /bin directory in your PATH to use
   204 it at will.
   205 
   206 In any case, you can get some terse help. Just type:
   207   ct-ng help
   208 or:
   209   man 1 ct-ng
   210 
   211 
   212 Stoping and restarting a build |
   213 -------------------------------*
   214 
   215 If you want to stop the build after a step you are debugging, you can pass the
   216 variable STOP to make:
   217   ct-ng STOP=some_step
   218 
   219 Conversely, if you want to restart a build at a specific step you are
   220 debugging, you can pass the RESTART variable to make:
   221   ct-ng RESTART=some_step
   222 
   223 Alternatively, you can call make with the name of a step to just do that step:
   224   ct-ng libc_headers
   225 is equivalent to:
   226   ct-ng RESTART=libs_headers STOP=libc_headers
   227 
   228 The shortcuts -step_name and step_name- allow to respectively stop or restart
   229 at that step. Thus:
   230   ct-ng -libc_headers        and:    ct-ng libc_headers-
   231 are equivalent to:
   232   ct-ng STOP=libc_headers    and:    ct-ng RESTART=libc_headers
   233 
   234 To obtain the list of acceptable steps, please call:
   235   ct-ng liststeps
   236 
   237 Note that in order to restart a build, you'll have to say 'Y' to the config
   238 option CT_DEBUG_CT_SAVE_STEPS, and that the previous build effectively went
   239 that far.
   240 
   241 
   242 Testing all toolchains at once |
   243 -------------------------------*
   244 
   245 You can test-build all samples; simply call:
   246   ct-ng regtest
   247 
   248 _______________________
   249                       /
   250 Using the toolchain  /
   251 ____________________/
   252 
   253 Using the toolchain is as simple as adding the toolchain's bin directory in
   254 your PATH, such as:
   255   export PATH="${PATH}:/your/toolchain/path/bin"
   256 
   257 and then using the target triplet to tell the build systems to use your
   258 toolchain:
   259   ./configure --target=your-target-triplet
   260   make CC=your-target-triplet-gcc
   261   make CROSS_COMPILE=your-target-triplet-
   262   and so on...
   263 
   264 When your root directory is ready, it is still missing some important bits: the
   265 toolchain's libraries. To populate your root directory with those libs, just
   266 run:
   267   your-target-triplet-populate -s /your/root -d /your/root-populated
   268 
   269 This will copy /your/root into /your/root-populated, and put the needed and only
   270 the needed libraries there. Thus you don't polute /your/root with any cruft that
   271 would no longer be needed should you have to remove stuff. /your/root always
   272 contains only those things you install in it.
   273 
   274 You can then use /your/root-populated to build up your file system image, a
   275 tarball, or to NFS-mount it from your target, or whatever you need.
   276 
   277 ___________________
   278                   /
   279 Toolchain types  /
   280 ________________/
   281 
   282 There are four kinds of toolchains you could encounter.
   283 
   284 First off, you must understand the following: when it comes to compilers there
   285 are up to four machines involved:
   286   1) the machine configuring the toolchain components: the config machine
   287   2) the machine building the toolchain components:    the build machine
   288   3) the machine running the toolchain:                the host machine
   289   4) the machine the toolchain is generating code for: the target machine
   290 
   291 We can most of the time assume that the config machine and the build machine
   292 are the same. Most of the time, this will be true. The only time it isn't
   293 is if you're using distributed compilation (such as distcc). Let's forget
   294 this for the sake of simplicity.
   295 
   296 So we're left with three machines:
   297  - build
   298  - host
   299  - target
   300 
   301 Any toolchain will involve those three machines. You can be as pretty sure of
   302 this as "2 and 2 are 4". Here is how they come into play:
   303 
   304 1) build == host == target
   305     This is a plain native toolchain, targetting the exact same machine as the
   306     one it is built on, and running again on this exact same machine. You have
   307     to build such a toolchain when you want to use an updated component, such
   308     as a newer gcc for example.
   309     crosstool-NG calls it "native".
   310 
   311 2) build == host != target
   312     This is a classic cross-toolchain, which is expected to be run on the same
   313     machine it is compiled on, and generate code to run on a second machine,
   314     the target.
   315     crosstool-NG calls it "cross".
   316 
   317 3) build != host == target
   318     Such a toolchain is also a native toolchain, as it targets the same machine
   319     as it runs on. But it is build on another machine. You want such a
   320     toolchain when porting to a new architecture, or if the build machine is
   321     much faster than the host machine.
   322     crosstool-NG calls it "cross-native".
   323 
   324 4) build != host != target
   325     This one is called a canadian-toolchain (*), and is tricky. The three
   326     machines in play are different. You might want such a toolchain if you
   327     have a fast build machine, but the users will use it on another machine,
   328     and will produce code to run on a third machine.
   329     crosstool-NG calls it "canadian".
   330 
   331 crosstool-NG can build all these kinds of toolchains (or is aiming at it,
   332 anyway!)
   333 
   334 (*) The term Canadian Cross came about because at the time that these issues
   335     were all being hashed out, Canada had three national political parties.
   336     http://en.wikipedia.org/wiki/Cross_compiler
   337 
   338 _____________
   339             /
   340 Internals  /
   341 __________/
   342 
   343 Internally, crosstool-NG is script-based. To ease usage, the frontend is
   344 Makefile-based.
   345 
   346 Makefile front-end |
   347 -------------------*
   348 
   349 The entry point to crosstool-NG is the Makefile script "ct-ng". Calling this
   350 script with an action will act exactly as if the Makefile was in the current
   351 working directory and make was called with the action as rule. Thus:
   352   ct-ng menuconfig
   353 is equivalent to having the Makefile in CWD, and calling:
   354   make menuconfig
   355 
   356 Having ct-ng as it is avoids copying the Makefile everywhere, and acts as a
   357 traditional command.
   358 
   359 ct-ng loads sub- Makefiles from the library directory $(CT_LIB_DIR), as set up
   360 at configuration time with ./configure.
   361 
   362 ct-ng also search for config files, sub-tools, samples, scripts and patches in
   363 that library directory.
   364 
   365 Kconfig parser |
   366 ---------------*
   367 
   368 The kconfig language is a hacked version, vampirised from the toybox project
   369 by Rob LANDLEY (http://www.landley.net/code/toybox/), itself coming from the
   370 Linux kernel (http://www.linux.org/ http://www.kernel.org/), and (heavily)
   371 adapted to my needs.
   372 
   373 The kconfig parsers (conf and mconf) are not installed pre-built, but as
   374 source files. Thus you can have the directory where crosstool-NG is installed,
   375 exported (via NFS or whatever) and have clients with different architectures
   376 use the same crosstool-NG installation, and most notably, the same set of
   377 patches.
   378 
   379 Build scripts |
   380 --------------*
   381 
   382 To Be Written later...