docs/overview.txt
author Johannes Stezenbach <js@sig21.net>
Thu Jul 29 19:30:37 2010 +0200 (2010-07-29)
branch1.7
changeset 2047 ace1d90c9b15
parent 1843 266166448ffd
child 1963 ea472c30c356
permissions -rw-r--r--
scripts: remove . from $PATH

Add CT_SanitizePath function which removes entries referring to ., /tmp
and non-existing directories from $PATH, and call it early in the
build script.

If . is in PATH, gcc-4.4.4 build breaks:

[ALL ] checking what assembler to use...
/tmp/build/targets/arm-unknown-linux-uclibcgnueabi/build/gcc-core-static/arm-unknown-linux-uclibcgnueabi/bin/as
...
[ALL ] config.status: creating as

i.e. "as" is supposed to be the arm-unknown-linux-uclibcgnueabi cross assembler,
but config.status creates a local "as" script which is calling the
host assembler.

Signed-off-by: Johannes Stezenbach <js@sig21.net>
[Yann E. MORIN: style fixes + explanations]
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@anciens.enib.fr>
(transplanted from 20dd8cef1c8adff0aa3e78ae6d7acfbc45ed5a83)
     1 File.........: overview.txt
     2 Content......: Overview of how crosstool-NG works.
     3 Copyrigth....: (C) 2007 Yann E. MORIN <yann.morin.1998@anciens.enib.fr>
     4 License......: Creative Commons Attribution Share Alike (CC-by-sa), v2.5
     5 
     6 ____________________
     7                    /
     8 Table Of Content  /
     9 _________________/
    10 
    11 
    12 Introduction
    13 History
    14 Referring to crosstool-NG
    15 Installing crosstool-NG
    16   Install method
    17   The hacker's way
    18   Preparing for packaging
    19   Shell completion
    20   Contributed code
    21 Configuring crosstool-NG
    22   Interesting config options
    23   Re-building an existing toolchain
    24   Using as a backend for a build-system
    25 Running crosstool-NG
    26   Stopping and restarting a build
    27   Testing all toolchains at once
    28   Overriding the number of // jobs
    29   Note on // jobs
    30   Tools wrapper
    31 Using the toolchain
    32   The 'populate' script
    33 Toolchain types
    34   Seemingly-native toolchains
    35 Contributing
    36   Sending a bug report
    37   Sending patches
    38 Internals
    39   Makefile front-end
    40   Kconfig parser
    41   Architecture-specific
    42   Adding a new version of a component
    43   Build scripts
    44 
    45 
    46 ________________
    47                /
    48 Introduction  /
    49 _____________/
    50 
    51 crosstool-NG aims at building toolchains. Toolchains are an essential component
    52 in a software development project. It will compile, assemble and link the code
    53 that is being developed. Some pieces of the toolchain will eventually end up
    54 in the resulting binary/ies: static libraries are but an example.
    55 
    56 So, a toolchain is a very sensitive piece of software, as any bug in one of the
    57 components, or a poorly configured component, can lead to execution problems,
    58 ranging from poor performance, to applications ending unexpectedly, to
    59 mis-behaving software (which more than often is hard to detect), to hardware
    60 damage, or even to human risks (which is more than regrettable).
    61 
    62 Toolchains are made of different piece of software, each being quite complex
    63 and requiring specially crafted options to build and work seamlessly. This
    64 is usually not that easy, even in the not-so-trivial case of native toolchains.
    65 The work reaches a higher degree of complexity when it comes to cross-
    66 compilation, where it can become quite a nightmare...
    67 
    68 Some cross-toolchains exist on the internet, and can be used for general
    69 development, but they have a number of limitations:
    70   - they can be general purpose, in that they are configured for the majority:
    71     no optimisation for your specific target,
    72   - they can be prepared for a specific target and thus are not easy to use,
    73     nor optimised for, or even supporting your target,
    74   - they often are using aging components (compiler, C library, etc...) not
    75     supporting special features of your shiny new processor;
    76 On the other side, these toolchain offer some advantages:
    77   - they are ready to use and quite easy to install and setup,
    78   - they are proven if used by a wide community.
    79 
    80 But once you want to get all the juice out of your specific hardware, you will
    81 want to build your own toolchain. This is where crosstool-NG comes into play.
    82 
    83 There are also a number of tools that build toolchains for specific needs,
    84 which are not really scalable. Examples are:
    85   - buildroot (buildroot.uclibc.org) whose main purpose is to build root file
    86     systems, hence the name. But once you have your toolchain with buildroot,
    87     part of it is installed in the root-to-be, so if you want to build a whole
    88     new root, you either have to save the existing one as a template and
    89     restore it later, or restart again from scratch. This is not convenient,
    90   - ptxdist (www.pengutronix.de/software/ptxdist), whose purpose is very
    91     similar to buildroot,
    92   - other projects (openembedded.org for example), which are again used to
    93     build root file systems.
    94 
    95 crosstool-NG is really targeted at building toolchains, and only toolchains.
    96 It is then up to you to use it the way you want.
    97 
    98 
    99 ___________
   100           /
   101 History  /
   102 ________/
   103 
   104 crosstool was first 'conceived' by Dan Kegel, who offered it to the community
   105 as a set of scripts, a repository of patches, and some pre-configured, general
   106 purpose setup files to be used to configure crosstool. This is available at
   107 http://www.kegel.com/crosstool, and the subversion repository is hosted on
   108 google at http://code.google.com/p/crosstool/.
   109 
   110 I once managed to add support for uClibc-based toolchains, but it did not make
   111 into mainline, mostly because I didn't have time to port the patch forward to
   112 the new versions, due in part to the big effort it was taking.
   113 
   114 So I decided to clean up crosstool in the state it was, re-order the things
   115 in place, add appropriate support for what I needed, that is uClibc support
   116 and a menu-driven configuration, named the new implementation crosstool-NG,
   117 (standing for crosstool Next Generation, as many other comunity projects do,
   118 and as a wink at the TV series "Star Trek: The Next Generation" ;-) ) and
   119 made it available to the community, in case it was of interest to any one.
   120 
   121 
   122 _____________________________
   123                             /
   124 Referring to crosstool-NG  /
   125 __________________________/
   126 
   127 
   128 The long name of the project is crosstool-NG:
   129   * no leading uppercase (except as first word in a sentence)
   130   * crosstool and NG separated with a hyphen (dash)
   131   * NG in uppercase
   132 
   133 Crosstool-NG can also be referred to by its short name CT-NG:
   134   * all in uppercase
   135   * CT and NG separated with a hyphen (dash)
   136 
   137 The long name is preferred over the short name, except in mail subjects, where
   138 the short name is a better fit.
   139 
   140 When referring to a specific version of crosstool-NG, append the version number
   141 either as:
   142   * crosstool-NG X.Y.Z
   143     - the long name, a space, and the version string
   144   * crosstool-ng-X.Y.Z
   145     - the long name in lowercase, a hyphen (dash), and the version string
   146     - this is used to name the release tarballs
   147   * crosstool-ng-X.Y.Z+hg_id
   148     - the long name in lowercase, a hyphen, the version string, and the Hg id
   149       (as returned by: ct-ng version)
   150     - this is used to differentiate between releases and snapshots
   151 
   152 The frontend to crosstool-NG is the command ct-ng:
   153   * all in lowercase
   154   * ct and ng separated by a hyphen (dash)
   155 
   156 
   157 ___________________________
   158                           /
   159 Installing crosstool-NG  /
   160 ________________________/
   161 
   162 There are two ways you can use crosstool-NG:
   163  - build and install it, then get rid of the sources like you'd do for most
   164    programs,
   165  - or only build it and run from the source directory.
   166 
   167 The former should be used if you got crosstool-NG from a packaged tarball, see
   168 "Install method", below, while the latter is most useful for developpers that
   169 use a clone of the repository, and want to submit patches, see "The Hacker's
   170 way", below.
   171 
   172 Install method |
   173 ---------------+
   174 
   175 If you go for the install, then you just follow the classical, but yet easy
   176 ./configure way:
   177   ./configure --prefix=/some/place
   178   make
   179   make install
   180   export PATH="${PATH}:/some/place/bin"
   181 
   182 You can then get rid of crosstool-NG source. Next create a directory to serve
   183 as a working place, cd in there and run:
   184   ct-ng help
   185 
   186 See below for complete usage.
   187 
   188 The Hacker's way |
   189 -----------------+
   190 
   191 If you go the hacker's way, then the usage is a bit different, although very
   192 simple:
   193   ./configure --local
   194   make
   195 
   196 Now, *do not* remove crosstool-NG sources. They are needed to run crosstool-NG!
   197 Stay in the directory holding the sources, and run:
   198   ./ct-ng help
   199 
   200 See below for complete usage.
   201 
   202 Now, provided you used a clone of the repository, you can send me your changes.
   203 See the section titled CONTRIBUTING, below, for how to submit changees.
   204 
   205 Preparing for packaging |
   206 ------------------------+
   207 
   208 If you plan on packaging crosstool-NG, you surely don't want to install it
   209 in your root file system. The install procedure of crosstool-NG honors the
   210 DESTDIR variable:
   211 
   212   ./configure --prefix=/usr
   213   make
   214   make DESTDIR=/packaging/place install
   215 
   216 Shell completion |
   217 -----------------+
   218 
   219 crosstool-NG comes with a shell script fragment that defines bash-compatible
   220 completion. That shell fragment is currently not installed automatically, but
   221 this is planned.
   222 
   223 To install the shell script fragment, you have two options:
   224  - install system-wide, most probably by copying ct-ng.comp into
   225    /etc/bash_completion.d/
   226  - install for a single user, by copying ct-ng.comp into ${HOME}/ and
   227    sourcing this file from your ${HOME}/.bashrc
   228 
   229 Contributed code |
   230 -----------------+
   231 
   232 Some people contibuted code that couldn't get merged for various reasons. This
   233 code is available as lzma-compressed patches, in the contrib/ sub-directory.
   234 These patches are to be applied to the source of crosstool-NG, prior to
   235 installing, using something like the following:
   236   lzcat contrib/foobar.patch.lzma |patch -p1
   237 
   238 There is no guarantee that a particuliar contribution applies to the current
   239 version of crosstool-ng, or that it will work at all. Use contributions at
   240 your own risk.
   241 
   242 
   243 ____________________________
   244                            /
   245 Configuring crosstool-NG  /
   246 _________________________/
   247 
   248 crosstool-NG is configured with a configurator presenting a menu-stuctured set
   249 of options. These options let you specify the way you want your toolchain
   250 built, where you want it installed, what architecture and specific processor it
   251 will support, the version of the components you want to use, etc... The
   252 value for those options are then stored in a configuration file.
   253 
   254 The configurator works the same way you configure your Linux kernel. It is
   255 assumed you now how to handle this.
   256 
   257 To enter the menu, type:
   258   ct-ng menuconfig
   259 
   260 Almost every config item has a help entry. Read them carefully.
   261 
   262 String and number options can refer to environment variables. In such a case,
   263 you must use the shell syntax: ${VAR}. You shall neither single- nor double-
   264 quote the string/number options.
   265 
   266 There are three environment variables that are computed by crosstool-NG, and
   267 that you can use:
   268 
   269 CT_TARGET:
   270   It represents the target tuple you are building for. You can use it for
   271   example in the installation/prefix directory, such as:
   272     /opt/x-tools/${CT_TARGET}
   273 
   274 CT_TOP_DIR:
   275   The top directory where crosstool-NG is running. You shouldn't need it in
   276   most cases. There is one case where you may need it: if you have local
   277   patches and you store them in your running directory, you can refer to them
   278   by using CT_TOP_DIR, such as:
   279     ${CT_TOP_DIR}/patches.myproject
   280 
   281 CT_VERSION:
   282   The version of crosstool-NG you are using. Not much use for you, but it's
   283   there if you need it.
   284 
   285 Interesting config options |
   286 ---------------------------+
   287 
   288 CT_LOCAL_TARBALLS_DIR:
   289   If you already have some tarballs in a direcotry, enter it here. That will
   290   speed up the retrieving phase, where crosstool-NG would otherwise download
   291   those tarballs.
   292 
   293 CT_PREFIX_DIR:
   294   This is where the toolchain will be installed in (and for now, where it
   295   will run from). Common use is to add the target tuple in the directory
   296   path, such as (see above):
   297     /opt/x-tools/${CT_TARGET}
   298 
   299 CT_TARGET_VENDOR:
   300   An identifier for your toolchain, will take place in the vendor part of the
   301   target tuple. It shall *not* contain spaces or dashes. Usually, keep it
   302   to a one-word string, or use underscores to separate words if you need.
   303   Avoid dots, commas, and special characters.
   304 
   305 CT_TARGET_ALIAS:
   306   An alias for the toolchian. It will be used as a prefix to the toolchain
   307   tools. For example, you will have ${CT_TARGET_ALIAS}-gcc
   308 
   309 Also, if you think you don't see enough versions, you can try to enable one of
   310 those:
   311 
   312 CT_OBSOLETE:
   313   Show obsolete versions or tools. Most of the time, you don't want to base
   314   your toolchain on too old a version (of gcc, for example). But at times, it
   315   can come handy to use such an old version for regression tests. Those old
   316   versions are hidden behind CT_OBSOLETE. Those versions (or features) are so
   317   marked because maintaining support for those in crosstool-NG would be too
   318   costly, time-wise, and time is dear.
   319 
   320 CT_EXPERIMENTAL:
   321   Show experimental versions or tools. Again, you might not want to base your
   322   toolchain on too recent tools (eg. gcc) for production. But if you need a
   323   feature present only in a recent version, or a new tool, you can find them
   324   hidden behind CT_EXPERIMENTAL. Those versions (or features) did not (yet)
   325   receive thorough testing in crosstool-NG, and/or are not mature enough to
   326   be blindly trusted.
   327 
   328 Re-building an existing toolchain |
   329 ----------------------------------+
   330 
   331 If you have an existing toolchain, you can re-use the options used to build it
   332 to create a new toolchain. That needs a very little bit of effort on your side
   333 but is quite easy. The options to build a toolchain are saved with the
   334 toolchain, and you can retrieve this configuration by running:
   335   ${CT_TARGET}-ct-ng.config
   336 
   337 An alternate method is to extract the configuration from a build.log file.
   338 This will be necessary if your toolchain was build with crosstool-NG prior
   339 to 1.4.0, but can be used with build.log files from any version:
   340   ct-ng extractconfig <build.log >.config
   341 
   342 Or, if your build.log file is compressed (most probably!):
   343   bzcat build.log.bz2 |ct-ng extractconfig >.config
   344 
   345 The above commands will dump the configuration to stdout, so to rebuild a
   346 toolchain with this configuration, just redirect the output to the
   347 .config file:
   348   ${CT_TARGET}-ct-ng.config >.config
   349   ct-ng oldconfig
   350 
   351 Then, you can review and change the configuration by running:
   352   ct-ng menuconfig
   353 
   354 Using as a backend for a build-system |
   355 --------------------------------------+
   356 
   357 Crosstool-NG can be used as a backend for an automated build-system. In this
   358 case, some components that are expected to run on the target (eg. the native
   359 gdb, ltrace, DUMA...) are not available in the menuconfig, and they are not
   360 build either, as it is considered the responsibility of the build-system to
   361 build its own versions of those tools.
   362 
   363 If you want to use crosstool-NG as a backend to generate your toolchains for
   364 your build-system, you have to set and export this environment variable:
   365   CT_IS_A_BACKEND=y
   366 
   367 (case is not sensitive, you can say Y).
   368 
   369 
   370 ________________________
   371                        /
   372 Running crosstool-NG  /
   373 _____________________/
   374 
   375 To build the toolchain, simply type:
   376   ct-ng build
   377 
   378 This will use the above configuration to retrieve, extract and patch the
   379 components, build, install and eventually test your newly built toolchain.
   380 
   381 You are then free to add the toolchain /bin directory in your PATH to use
   382 it at will.
   383 
   384 In any case, you can get some terse help. Just type:
   385   ct-ng help
   386 or:
   387   man 1 ct-ng
   388 
   389 Stopping and restarting a build |
   390 --------------------------------+
   391 
   392 If you want to stop the build after a step you are debugging, you can pass the
   393 variable STOP to make:
   394   ct-ng build STOP=some_step
   395 
   396 Conversely, if you want to restart a build at a specific step you are
   397 debugging, you can pass the RESTART variable to make:
   398   ct-ng build RESTART=some_step
   399 
   400 Alternatively, you can call make with the name of a step to just do that step:
   401   ct-ng libc_headers
   402 is equivalent to:
   403   ct-ng build RESTART=libc_headers STOP=libc_headers
   404 
   405 The shortcuts +step_name and step_name+ allow to respectively stop or restart
   406 at that step. Thus:
   407   ct-ng +libc_headers              and:    ct-ng libc_headers+
   408 are equivalent to:
   409   ct-ng build STOP=libc_headers    and:    ct-ng build RESTART=libc_headers
   410 
   411 To obtain the list of acceptable steps, please call:
   412   ct-ng list-steps
   413 
   414 Note that in order to restart a build, you'll have to say 'Y' to the config
   415 option CT_DEBUG_CT_SAVE_STEPS, and that the previous build effectively went
   416 that far.
   417 
   418 Building all toolchains at once |
   419 --------------------------------+
   420 
   421 You can build all samples; simply call:
   422   ct-ng build-all
   423 
   424 Overriding the number of // jobs |
   425 ---------------------------------+
   426 
   427 If you want to override the number of jobs to run in // (the -j option to
   428 make), you can either re-enter the menuconfig, or simply add it on the command
   429 line, as such:
   430   ct-ng build.4
   431 
   432 which tells crosstool-NG to override the number of // jobs to 4.
   433 
   434 You can see the actions that support overriding the number of // jobs in
   435 the help menu. Those are the ones with [.#] after them (eg. build[.#] or
   436 build-all[.#], and so on...).
   437 
   438 Note on // jobs |
   439 ----------------+
   440 
   441 The crosstool-NG script 'ct-ng' is a Makefile-script. It does *not* execute
   442 in parallel (there is not much to gain). When speaking of // jobs, we are
   443 refering to the number of // jobs when making the *components*. That is, we
   444 speak of the number of // jobs used to build gcc, glibc, and so on...
   445 
   446 Tools wrapper |
   447 --------------+
   448 
   449 Starting with gcc-4.3 come two new dependencies: GMP and MPFR. With gcc-4.4,
   450 come three new ones: PPL, CLooG/ppl and MPC. With gcc-4.5 again comes a new
   451 dependency on libelf. These are libraries that enable advanced features to
   452 gcc. Additionally, some of those libraries can be used by binutils and gdb.
   453 Unfortunately, not all systems on which crosstool-NG runs have all of those
   454 libraries. And for those that do, the versions of those libraries may be
   455 older than the version required by gcc (and binutils and gdb). To date,
   456 Debian stable (aka Lenny) is lagging behind on some, and is missing the
   457 others.
   458 
   459 This is why crosstool-NG builds its own set of libraries as part of the
   460 toolchain.
   461 
   462 The companion libraries can be built either as static libraries, or as shared
   463 libraries. The default is to build static libraries, and is the safe way.
   464 If you decide to use static companion libraries, then you can stop reading
   465 this section.
   466 
   467 But if you prefer to have shared libraries, then read on...
   468 
   469 Building shared companion libraries poses no problem at build time, as
   470 crosstool-NG correctly points gcc (and binutils and gdb) to the correct
   471 place where our own version of the libraries are installed. But it poses
   472 a problem when gcc et al. are run: the place where the libraries are is most
   473 probably not known to the host dynamic linker. Still worse, if the host system
   474 has its own versions, then ld.so would load the wrong libraries!
   475 
   476 So we have to force the dynamic linker to load the correct version. We do this
   477 by using the LD_LIBRARY_PATH variable, that informs the dynamic linker where
   478 to look for shared libraries prior to searching its standard places. But we
   479 can't impose that burden on all the system (because it'd be a nightmare to
   480 configure, and because two toolchains on the same system may use different
   481 versions of the libraries); so we have to do it on a per-toolchain basis.
   482 
   483 So we rename all binaries of the toolchain (by adding a dot '.' as their first
   484 character), and add a small program, the so-called "tools wrapper", that
   485 correctly sets LD_LIBRARY_PATH prior to running the real tool.
   486 
   487 First, the wrapper was written as a POSIX-compliant shell script. That shell
   488 script is very simple, if not trivial, and works great. The only drawback is
   489 that it does not work on host systems that lack a shell, for example the
   490 MingW32 environment. To solve the issue, the wrapper has been re-written in C,
   491 and compiled at build time. This C wrapper is much more complex than the shell
   492 script, and although it sems to be working, it's been only lightly tested.
   493 Some of the expected short-comings with this C wrapper are;
   494  - multi-byte file names may not be handled correctly
   495  - it's really big for what it does
   496 
   497 So, the default wrapper installed with your toolchain is the shell script.
   498 If you know that your system is missing a shell, then you shall use the C
   499 wrapper (and report back whether it works, or does not work, for you).
   500 
   501 A final word on the subject: do not build shared libraries. Build them
   502 static, and you'll be safe.
   503 
   504 
   505 _______________________
   506                       /
   507 Using the toolchain  /
   508 ____________________/
   509 
   510 Using the toolchain is as simple as adding the toolchain's bin directory in
   511 your PATH, such as:
   512   export PATH="${PATH}:/your/toolchain/path/bin"
   513 
   514 and then using the target tuple to tell the build systems to use your
   515 toolchain:
   516   ./configure --target=your-target-tuple
   517 or
   518   make CC=your-target-tuple-gcc
   519 or
   520   make CROSS_COMPILE=your-target-tuple-
   521 and so on...
   522 
   523 It is strongly advised not to use the toolchain sys-root directory as an
   524 install directory for your programs/packages. If you do so, you will not be
   525 able to use your toolchain for another project. It is even strongly advised
   526 that your toolchain is chmod-ed to read-only once successfully build, so that
   527 you don't go polluting your toolchain with your programs/packages' files.
   528 
   529 Thus, when you build a program/package, install it in a separate directory,
   530 eg. /your/root. This directory is the /image/ of what would be in the root file
   531 system of your target, and will contain all that your programs/packages have
   532 installed.
   533 
   534 The 'populate' script |
   535 ----------------------+
   536 
   537 When your root directory is ready, it is still missing some important bits: the
   538 toolchain's libraries. To populate your root directory with those libs, just
   539 run:
   540   your-target-tuple-populate -s /your/root -d /your/root-populated
   541 
   542 This will copy /your/root into /your/root-populated, and put the needed and only
   543 the needed libraries there. Thus you don't polute /your/root with any cruft that
   544 would no longer be needed should you have to remove stuff. /your/root always
   545 contains only those things you install in it.
   546 
   547 You can then use /your/root-populated to build up your file system image, a
   548 tarball, or to NFS-mount it from your target, or whatever you need.
   549 
   550 The populate script accepts the following options:
   551 
   552  -s src_dir
   553     Use 'src_dir' as the un-populated root directory.
   554 
   555  -d dst_dir
   556     Put the populated root directory in 'dst_dir'.
   557 
   558  -l lib1 [...]
   559     Always add specified libraries.
   560 
   561  -L file
   562     Always add libraries listed in 'file'.
   563 
   564  -f
   565     Remove 'dst_dir' if it previously existed; continue even if any library
   566     specified with -l or -L is missing.
   567 
   568  -v
   569     Be verbose, and tell what's going on (you can see exactly where libs are
   570     coming from).
   571 
   572  -h
   573     Print the help.
   574 
   575 See 'your-target-tuple-populate -h' for more information on the options.
   576 
   577 Here is how populate works:
   578 
   579   1) performs some sanity checks:
   580      - src_dir and dst_dir are specified
   581      - src_dir exists
   582      - unless forced, dst_dir does not exist
   583      - src_dir != dst_dir
   584 
   585   2) copy src_dir to dst_dir
   586 
   587   3) add forced libraries to dst_dir
   588      - build the list from -l and -L options
   589      - get forced libraries from the sysroot (see below for heuristics)
   590        - abort on the first missing library, unless -f is specified
   591 
   592   4) add all missing libraries to dst_dir
   593      - scan dst_dir for every ELF files that are 'executable' or
   594        'shared object'
   595      - list the "NEEDED Shared library" fields
   596        - check if the library is already in dst_dir/lib or dst_dir/usr/lib
   597        - if not, get the library from the sysroot
   598          - if it's in sysroot/lib, copy it to dst_dir/lib
   599          - if it's in sysroot/usr/lib, copy it to dst_dir/usr/lib
   600          - in both cases, use the SONAME of the library to create the file
   601            in dst_dir
   602          - if it was not found in the sysroot, this is an error.
   603 
   604 
   605 ___________________
   606                   /
   607 Toolchain types  /
   608 ________________/
   609 
   610 There are four kinds of toolchains you could encounter.
   611 
   612 First off, you must understand the following: when it comes to compilers there
   613 are up to four machines involved:
   614   1) the machine configuring the toolchain components: the config machine
   615   2) the machine building the toolchain components:    the build machine
   616   3) the machine running the toolchain:                the host machine
   617   4) the machine the toolchain is generating code for: the target machine
   618 
   619 We can most of the time assume that the config machine and the build machine
   620 are the same. Most of the time, this will be true. The only time it isn't
   621 is if you're using distributed compilation (such as distcc). Let's forget
   622 this for the sake of simplicity.
   623 
   624 So we're left with three machines:
   625  - build
   626  - host
   627  - target
   628 
   629 Any toolchain will involve those three machines. You can be as pretty sure of
   630 this as "2 and 2 are 4". Here is how they come into play:
   631 
   632 1) build == host == target
   633     This is a plain native toolchain, targetting the exact same machine as the
   634     one it is built on, and running again on this exact same machine. You have
   635     to build such a toolchain when you want to use an updated component, such
   636     as a newer gcc for example.
   637     crosstool-NG calls it "native".
   638 
   639 2) build == host != target
   640     This is a classic cross-toolchain, which is expected to be run on the same
   641     machine it is compiled on, and generate code to run on a second machine,
   642     the target.
   643     crosstool-NG calls it "cross".
   644 
   645 3) build != host == target
   646     Such a toolchain is also a native toolchain, as it targets the same machine
   647     as it runs on. But it is build on another machine. You want such a
   648     toolchain when porting to a new architecture, or if the build machine is
   649     much faster than the host machine.
   650     crosstool-NG calls it "cross-native".
   651 
   652 4) build != host != target
   653     This one is called a canadian-toolchain (*), and is tricky. The three
   654     machines in play are different. You might want such a toolchain if you
   655     have a fast build machine, but the users will use it on another machine,
   656     and will produce code to run on a third machine.
   657     crosstool-NG calls it "canadian".
   658 
   659 crosstool-NG can build all these kinds of toolchains (or is aiming at it,
   660 anyway!)
   661 
   662 (*) The term Canadian Cross came about because at the time that these issues
   663     were all being hashed out, Canada had three national political parties.
   664     http://en.wikipedia.org/wiki/Cross_compiler
   665 
   666 
   667 ________________
   668                /
   669 Contributing  /
   670 _____________/
   671 
   672 Sending a bug report |
   673 ---------------------+
   674 
   675 If you need to send a bug report, please send a mail with subject
   676 prefixed with "[CT_NG]" with to following destinations:
   677     TO: yann.morin.1998 (at) anciens.enib.fr
   678     CC: crossgcc (at) sourceware.org
   679 
   680 Sending patches |
   681 ----------------+
   682 
   683 If you want to enhance crosstool-NG, there's a to-do list in the TODO file.
   684 
   685 Patches should come with the appropriate SoB line. A SoB line is typically
   686 something like:
   687    Signed-off-by: John DOE <john.doe@somewhere.net>
   688 
   689 The SoB line is clearly described in Documentation/SubmittingPatches , section
   690 12, of your favourite Linux kernel source tree.
   691 
   692 Then you'll need to correctly configure Mercurial. There are two extensions
   693 that you may find usefull:
   694   - mq        : http://mercurial.selenic.com/wiki/MqExtension
   695   - patchbomb : http://mercurial.selenic.com/wiki/PatchbombExtension
   696 
   697 Commit messages should look like (without leading pipes):
   698  |component: short, one-line description
   699  |
   700  |optional longer description
   701  |on multiple lines if needed
   702 
   703 Here is an example commit message (see revision a53a5e1d61db):
   704  |comp-libs/cloog: fix building
   705  |
   706  |For CLooG/PPL 0.15.3, the directory name was simply cloog-ppl.
   707  |For any later versions, the directory name does have the version, such as
   708  |cloog-ppl-0.15.4.
   709 
   710 Here's a typical hacking session:
   711   hg clone http://ymorin.is-a-geek.org/hg/crosstool-ng crosstool-ng
   712   cd crosstool-ng
   713   hg qinit
   714   hg qnew -D -U -e my_first_patch
   715   *edit patch description*
   716   *hack* *hack* *check* *fails* *hack* *hack* *check* *works*
   717   hg qref -D -e
   718   *edit patch description, serving as commit message*
   719   hg qnew -D -U -e my_second_patch
   720   *edit patch description*
   721   *hack* *hack* *check* *fails* *hack* *hack* *check* *works*
   722   hg qref -D -e
   723   *edit patch description, serving as commit message*
   724   hg email --outgoing --intro   \
   725            --from '"Your Full NAME" <your.email (at) your.domain>'   \
   726            --to '"Yann E. MORIN" <yann.morin.1998 (at) anciens.enib.fr>'    \
   727            --cc 'crossgcc (at) sourceware.org'
   728   *edit introductory message*
   729   *wait for feedback*
   730   *re-send if no answer for a few days*
   731 
   732 Note: replace '(at)' above with a plain '@'.
   733 
   734 
   735 _____________
   736             /
   737 Internals  /
   738 __________/
   739 
   740 Internally, crosstool-NG is script-based. To ease usage, the frontend is
   741 Makefile-based.
   742 
   743 Makefile front-end |
   744 -------------------+
   745 
   746 The entry point to crosstool-NG is the Makefile script "ct-ng". Calling this
   747 script with an action will act exactly as if the Makefile was in the current
   748 working directory and make was called with the action as rule. Thus:
   749   ct-ng menuconfig
   750 
   751 is equivalent to having the Makefile in CWD, and calling:
   752   make menuconfig
   753 
   754 Having ct-ng as it is avoids copying the Makefile everywhere, and acts as a
   755 traditional command.
   756 
   757 ct-ng loads sub- Makefiles from the library directory $(CT_LIB_DIR), as set up
   758 at configuration time with ./configure.
   759 
   760 ct-ng also searches for config files, sub-tools, samples, scripts and patches in
   761 that library directory.
   762 
   763 Because of a stupid make behavior/bug I was unable to track down, implicit make
   764 rules are disabled: installing with --local would triger those rules, and mconf
   765 was unbuildable.
   766 
   767 Kconfig parser |
   768 ---------------+
   769 
   770 The kconfig language is a hacked version, vampirised from the Linux kernel
   771 (http://www.kernel.org/), and (heavily) adapted to my needs.
   772 
   773 The list of the most notable changes (at least the ones I remember) follows:
   774 - the CONFIG_ prefix has been replaced with CT_
   775 - a leading | in prompts is skipped, and subsequent leading spaces are not
   776   trimmed; otherwise leading spaces are silently trimmed
   777 - removed the warning about undefined environment variable
   778 
   779 The kconfig parsers (conf and mconf) are not installed pre-built, but as
   780 source files. Thus you can have the directory where crosstool-NG is installed,
   781 exported (via NFS or whatever) and have clients with different architectures
   782 use the same crosstool-NG installation, and most notably, the same set of
   783 patches.
   784 
   785 Architecture-specific |
   786 ----------------------+
   787 
   788 Note: this chapter is not really well written, and might thus be a little bit
   789 complex to understand. To get a better grasp of what an architecture is, the
   790 reader is kindly encouraged to look at the "arch/" sub-directory, and to the
   791 existing architectures to see how things are laid out.
   792 
   793 An architecture is defined by:
   794 
   795  - a human-readable name, in lower case letters, with numbers as appropriate.
   796    The underscore is allowed; space and special characters are not.
   797      Eg.: arm, x86_64
   798  - a file in "config/arch/", named after the architecture's name, and suffixed
   799    with ".in".
   800      Eg.: config/arch/arm.in
   801  - a file in "scripts/build/arch/", named after the architecture's name, and
   802    suffixed with ".sh".
   803      Eg.: scripts/build/arch/arm.sh
   804 
   805 The architecture's ".in" file API:
   806  > the config option "ARCH_%arch%" (where %arch% is to be replaced with the
   807    actual architecture name).
   808    That config option must have *neither* a type, *nor* a prompt! Also, it can
   809    *not* depend on any other config option (EXPERIMENTAL is managed as above).
   810      Eg.:
   811        config ARCH_arm
   812    + mandatory:
   813        defines a (terse) help entry for this architecture:
   814        Eg.:
   815          config ARCH_arm
   816            help
   817              The ARM architecture.
   818    + optional:
   819        selects adequate associated config options.
   820        Note: 64-bit architectures *shall* select ARCH_64
   821        Eg.:
   822          config ARCH_arm
   823            select ARCH_SUPPORTS_BOTH_ENDIAN
   824            select ARCH_DEFAULT_LE
   825            help
   826              The ARM architecture.
   827        Eg.:
   828          config ARCH_x86_64
   829             select ARCH_64
   830             help
   831               The x86_64 architecture.
   832 
   833  > other target-specific options, at your discretion. Note however that to
   834    avoid name-clashing, such options shall be prefixed with "ARCH_%arch%",
   835    where %arch% is again replaced by the actual architecture name.
   836    (Note: due to historical reasons, and lack of time to clean up the code,
   837     I may have left some config options that do not completely conform to
   838     this, as the architecture name was written all upper case. However, the
   839     prefix is unique among architectures, and does not cause harm).
   840 
   841 The architecture's ".sh" file API:
   842  > the function "CT_DoArchTupleValues"
   843    + parameters: none
   844    + environment:
   845      - all variables from the ".config" file,
   846      - the two variables "target_endian_eb" and "target_endian_el" which are
   847        the endianness suffixes
   848    + return value: 0 upon success, !0 upon failure
   849    + provides:
   850      - mandatory
   851      - the environment variable CT_TARGET_ARCH
   852      - contains:
   853        the architecture part of the target tuple.
   854        Eg.: "armeb" for big endian ARM
   855             "i386" for an i386
   856    + provides:
   857      - optional
   858      - the environment variable CT_TARGET_SYS
   859      - contains:
   860        the sytem part of the target tuple.
   861        Eg.: "gnu" for glibc on most architectures
   862             "gnueabi" for glibc on an ARM EABI
   863      - defaults to:
   864        - for glibc-based toolchain: "gnu"
   865        - for uClibc-based toolchain: "uclibc"
   866    + provides:
   867      - optional
   868      - the environment variables to configure the cross-gcc (defaults)
   869        - CT_ARCH_WITH_ARCH    : the gcc ./configure switch to select architecture level         ( "--with-arch=${CT_ARCH_ARCH}"   )
   870        - CT_ARCH_WITH_ABI     : the gcc ./configure switch to select ABI level                  ( "--with-abi=${CT_ARCH_ABI}"     )
   871        - CT_ARCH_WITH_CPU     : the gcc ./configure switch to select CPU instruction set        ( "--with-cpu=${CT_ARCH_CPU}"     )
   872        - CT_ARCH_WITH_TUNE    : the gcc ./configure switch to select scheduling                 ( "--with-tune=${CT_ARCH_TUNE}"   )
   873        - CT_ARCH_WITH_FPU     : the gcc ./configure switch to select FPU type                   ( "--with-fpu=${CT_ARCH_FPU}"     )
   874        - CT_ARCH_WITH_FLOAT   : the gcc ./configure switch to select floating point arithmetics ( "--with-float=soft" or /empty/  )
   875    + provides:
   876      - optional
   877      - the environment variables to pass to the cross-gcc to build target binaries (defaults)
   878        - CT_ARCH_ARCH_CFLAG   : the gcc switch to select architecture level                     ( "-march=${CT_ARCH_ARCH}"            )
   879        - CT_ARCH_ABI_CFLAG    : the gcc switch to select ABI level                              ( "-mabi=${CT_ARCH_ABI}"              )
   880        - CT_ARCH_CPU_CFLAG    : the gcc switch to select CPU instruction set                    ( "-mcpu=${CT_ARCH_CPU}"              )
   881        - CT_ARCH_TUNE_CFLAG   : the gcc switch to select scheduling                             ( "-mtune=${CT_ARCH_TUNE}"            )
   882        - CT_ARCH_FPU_CFLAG    : the gcc switch to select FPU type                               ( "-mfpu=${CT_ARCH_FPU}"              )
   883        - CT_ARCH_FLOAT_CFLAG  : the gcc switch to choose floating point arithmetics             ( "-msoft-float" or /empty/           )
   884        - CT_ARCH_ENDIAN_CFLAG : the gcc switch to choose big or little endian                   ( "-mbig-endian" or "-mlittle-endian" )
   885      - default to:
   886        see above.
   887    + provides:
   888      - optional
   889      - the environement variables to configure the core and final compiler, specific to this architecture:
   890        - CT_ARCH_CC_CORE_EXTRA_CONFIG   : additional, architecture specific core gcc ./configure flags
   891        - CT_ARCH_CC_EXTRA_CONFIG        : additional, architecture specific final gcc ./configure flags
   892      - default to:
   893        - all empty
   894    + provides:
   895      - optional
   896      - the architecture-specific CFLAGS and LDFLAGS:
   897        - CT_ARCH_TARGET_CLFAGS
   898        - CT_ARCH_TARGET_LDFLAGS
   899      - default to:
   900        - all empty
   901 
   902 You can have a look at "config/arch/arm.in" and "scripts/build/arch/arm.sh" for
   903 a quite complete example of what an actual architecture description looks like.
   904 
   905 Kernel specific |
   906 ----------------+
   907 
   908 A kernel is defined by:
   909 
   910  - a human-readable name, in lower case letters, with numbers as appropriate.
   911    The underscore is allowed; space and special characters are not (although
   912    they are internally replaced with underscores.
   913      Eg.: linux, bare-metal
   914  - a file in "config/kernel/", named after the kernel name, and suffixed with
   915    ".in".
   916      Eg.: config/kernel/linux.in, config/kernel/bare-metal.in
   917  - a file in "scripts/build/kernel/", named after the kernel name, and suffixed
   918    with ".sh".
   919      Eg.: scripts/build/kernel/linux.sh, scripts/build/kernel/bare-metal.sh
   920 
   921 The kernel's ".in" file must contain:
   922  > an optional lines containing exactly "# EXPERIMENTAL", starting on the
   923    first column, and without any following space or other character.
   924    If this line is present, then this kernel is considered EXPERIMENTAL,
   925    and correct dependency on EXPERIMENTAL will be set.
   926 
   927  > the config option "KERNEL_%kernel_name%" (where %kernel_name% is to be
   928    replaced with the actual kernel name, with all special characters and
   929    spaces replaced by underscores).
   930    That config option must have *neither* a type, *nor* a prompt! Also, it can
   931    *not* depends on EXPERIMENTAL.
   932      Eg.: KERNEL_linux, KERNEL_bare_metal
   933    + mandatory:
   934        defines a (terse) help entry for this kernel.
   935        Eg.:
   936          config KERNEL_bare_metal
   937            help
   938              Build a compiler for use without any kernel.
   939    + optional:
   940        selects adequate associated config options.
   941        Eg.:
   942          config KERNEL_bare_metal
   943            select BARE_METAL
   944            help
   945              Build a compiler for use without any kernel.
   946 
   947  > other kernel specific options, at your discretion. Note however that, to
   948    avoid name-clashing, such options should be prefixed with
   949    "KERNEL_%kernel_name%", where %kernel_name% is again tp be replaced with
   950    the actual kernel name.
   951    (Note: due to historical reasons, and lack of time to clean up the code,
   952     I may have left some config options that do not completely conform to
   953     this, as the kernel name was written all upper case. However, the prefix
   954     is unique among kernels, and does not cause harm).
   955 
   956 The kernel's ".sh" file API:
   957  > is a bash script fragment
   958 
   959  > defines the function CT_DoKernelTupleValues
   960    + see the architecture's CT_DoArchTupleValues, except for:
   961    + set the environment variable CT_TARGET_KERNEL, the kernel part of the
   962      target tuple
   963    + return value: ignored
   964 
   965  > defines the function "do_kernel_get":
   966    + parameters: none
   967    + environment:
   968       - all variables from the ".config" file.
   969    + return value: 0 for success, !0 for failure.
   970    + behavior: download the kernel's sources, and store the tarball into
   971      "${CT_TARBALLS_DIR}". To this end, a functions is available, that
   972      abstracts downloading tarballs:
   973      - CT_DoGet <tarball_base_name> <URL1 [URL...]>
   974        Eg.: CT_DoGet linux-2.6.26.5 ftp://ftp.kernel.org/pub/linux/kernel/v2.6
   975      Note: retrieving sources from svn, cvs, git and the likes is not supported
   976      by CT_DoGet. You'll have to do this by hand, as it is done for eglibc in
   977      "scripts/build/libc/eglibc.sh"
   978 
   979  > defines the function "do_kernel_extract":
   980    + parameters: none
   981    + environment:
   982       - all variables from the ".config" file,
   983    + return value: 0 for success, !0 for failure.
   984    + behavior: extract the kernel's tarball into "${CT_SRC_DIR}", and apply
   985      required patches. To this end, a function is available, that abstracts
   986      extracting tarballs:
   987      - CT_ExtractAndPatch <tarball_base_name>
   988        Eg.: CT_ExtractAndPatch linux-2.6.26.5
   989 
   990  > defines the function "do_kernel_headers":
   991    + parameters: none
   992    + environment:
   993       - all variables from the ".config" file,
   994    + return value: 0 for success, !0 for failure.
   995    + behavior: install the kernel headers (if any) in "${CT_SYSROOT_DIR}/usr/include"
   996 
   997  > defines any kernel-specific helper functions
   998    These functions, if any, must be prefixed with "do_kernel_%CT_KERNEL%_",
   999    where '%CT_KERNEL%' is to be replaced with the actual kernel name, to avoid
  1000    any name-clashing.
  1001 
  1002 You can have a look at "config/kernel/linux.in" and "scripts/build/kernel/linux.sh"
  1003 as an example of what a complex kernel description looks like.
  1004 
  1005 Adding a new version of a component |
  1006 ------------------------------------+
  1007 
  1008 When a new component, such as the Linux kernel, gcc or any other is released,
  1009 adding the new version to crosstool-NG is quite easy. There is a script that
  1010 will do all that for you:
  1011   scripts/addToolVersion.sh
  1012 
  1013 Run it with no option to get some help.
  1014 
  1015 Build scripts |
  1016 --------------+
  1017 
  1018 To Be Written later...