docs/overview.txt
author Richard Strand <richard.strand@icomera.com>
Fri Jan 08 18:48:21 2010 +0000 (2010-01-08)
changeset 1714 5d693a13c84a
parent 1643 19205498184e
child 1745 41c2a6716511
permissions -rw-r--r--
documentation: Remove refs to seemingly native toolchains issue

Remove references to the seemingly native toolchains do not build
issue as it seems to be fixed.

Signed-off-by: Richard Strand <richard.strand@icomera.com>
     1 File.........: overview.txt
     2 Content......: Overview of how crosstool-NG works.
     3 Copyrigth....: (C) 2007 Yann E. MORIN <yann.morin.1998@anciens.enib.fr>
     4 License......: Creative Commons Attribution Share Alike (CC-by-sa), v2.5
     5 
     6 ____________________
     7                    /
     8 Table Of Content  /
     9 _________________/
    10 
    11 
    12 Introduction
    13 History
    14 Referring to crosstool-NG
    15 Installing crosstool-NG
    16   Install method
    17   The hacker's way
    18   Preparing for packaging
    19   Shell completion
    20   Contributed code
    21 Configuring crosstool-NG
    22   Interesting config options
    23   Re-building an existing toolchain
    24 Running crosstool-NG
    25   Stopping and restarting a build
    26   Testing all toolchains at once
    27   Overriding the number of // jobs
    28   Note on // jobs
    29   Tools wrapper
    30 Using the toolchain
    31   The 'populate' script
    32 Toolchain types
    33   Seemingly-native toolchains
    34 Contributing
    35   Sending a bug report
    36   Sending patches
    37 Internals
    38   Makefile front-end
    39   Kconfig parser
    40   Architecture-specific
    41   Adding a new version of a component
    42   Build scripts
    43 
    44 
    45 ________________
    46                /
    47 Introduction  /
    48 _____________/
    49 
    50 crosstool-NG aims at building toolchains. Toolchains are an essential component
    51 in a software development project. It will compile, assemble and link the code
    52 that is being developed. Some pieces of the toolchain will eventually end up
    53 in the resulting binary/ies: static libraries are but an example.
    54 
    55 So, a toolchain is a very sensitive piece of software, as any bug in one of the
    56 components, or a poorly configured component, can lead to execution problems,
    57 ranging from poor performance, to applications ending unexpectedly, to
    58 mis-behaving software (which more than often is hard to detect), to hardware
    59 damage, or even to human risks (which is more than regrettable).
    60 
    61 Toolchains are made of different piece of software, each being quite complex
    62 and requiring specially crafted options to build and work seamlessly. This
    63 is usually not that easy, even in the not-so-trivial case of native toolchains.
    64 The work reaches a higher degree of complexity when it comes to cross-
    65 compilation, where it can become quite a nightmare...
    66 
    67 Some cross-toolchains exist on the internet, and can be used for general
    68 development, but they have a number of limitations:
    69   - they can be general purpose, in that they are configured for the majority:
    70     no optimisation for your specific target,
    71   - they can be prepared for a specific target and thus are not easy to use,
    72     nor optimised for, or even supporting your target,
    73   - they often are using aging components (compiler, C library, etc...) not
    74     supporting special features of your shiny new processor;
    75 On the other side, these toolchain offer some advantages:
    76   - they are ready to use and quite easy to install and setup,
    77   - they are proven if used by a wide community.
    78 
    79 But once you want to get all the juice out of your specific hardware, you will
    80 want to build your own toolchain. This is where crosstool-NG comes into play.
    81 
    82 There are also a number of tools that build toolchains for specific needs,
    83 which are not really scalable. Examples are:
    84   - buildroot (buildroot.uclibc.org) whose main purpose is to build root file
    85     systems, hence the name. But once you have your toolchain with buildroot,
    86     part of it is installed in the root-to-be, so if you want to build a whole
    87     new root, you either have to save the existing one as a template and
    88     restore it later, or restart again from scratch. This is not convenient,
    89   - ptxdist (www.pengutronix.de/software/ptxdist), whose purpose is very
    90     similar to buildroot,
    91   - other projects (openembedded.org for example), which are again used to
    92     build root file systems.
    93 
    94 crosstool-NG is really targeted at building toolchains, and only toolchains.
    95 It is then up to you to use it the way you want.
    96 
    97 
    98 ___________
    99           /
   100 History  /
   101 ________/
   102 
   103 crosstool was first 'conceived' by Dan Kegel, who offered it to the community
   104 as a set of scripts, a repository of patches, and some pre-configured, general
   105 purpose setup files to be used to configure crosstool. This is available at
   106 http://www.kegel.com/crosstool, and the subversion repository is hosted on
   107 google at http://code.google.com/p/crosstool/.
   108 
   109 I once managed to add support for uClibc-based toolchains, but it did not make
   110 into mainline, mostly because I didn't have time to port the patch forward to
   111 the new versions, due in part to the big effort it was taking.
   112 
   113 So I decided to clean up crosstool in the state it was, re-order the things
   114 in place, add appropriate support for what I needed, that is uClibc support
   115 and a menu-driven configuration, named the new implementation crosstool-NG,
   116 (standing for crosstool Next Generation, as many other comunity projects do,
   117 and as a wink at the TV series "Star Trek: The Next Generation" ;-) ) and
   118 made it available to the community, in case it was of interest to any one.
   119 
   120 
   121 _____________________________
   122                             /
   123 Referring to crosstool-NG  /
   124 __________________________/
   125 
   126 
   127 The long name of the project is crosstool-NG:
   128   * no leading uppercase (except as first word in a sentence)
   129   * crosstool and NG separated with a hyphen (dash)
   130   * NG in uppercase
   131 
   132 Crosstool-NG can also be referred to by its short name CT-NG:
   133   * all in uppercase
   134   * CT and NG separated with a hyphen (dash)
   135 
   136 The long name is preferred over the short name, except in mail subjects, where
   137 the short name is a better fit.
   138 
   139 When referring to a specific version of crosstool-NG, append the version number
   140 either as:
   141   * crosstool-NG X.Y.Z
   142     - the long name, a space, and the version string
   143   * crosstool-ng-X.Y.Z
   144     - the long name in lowercase, a hyphen (dash), and the version string
   145     - this is used to name the release tarballs
   146   * crosstool-ng-X.Y.Z+hg_id
   147     - the long name in lowercase, a hyphen, the version string, and the Hg id
   148       (as returned by: ct-ng version)
   149     - this is used to differentiate between releases and snapshots
   150 
   151 The frontend to crosstool-NG is the command ct-ng:
   152   * all in lowercase
   153   * ct and ng separated by a hyphen (dash)
   154 
   155 
   156 ___________________________
   157                           /
   158 Installing crosstool-NG  /
   159 ________________________/
   160 
   161 There are two ways you can use crosstool-NG:
   162  - build and install it, then get rid of the sources like you'd do for most
   163    programs,
   164  - or only build it and run from the source directory.
   165 
   166 The former should be used if you got crosstool-NG from a packaged tarball, see
   167 "Install method", below, while the latter is most useful for developpers that
   168 use a clone of the repository, and want to submit patches, see "The Hacker's
   169 way", below.
   170 
   171 Install method |
   172 ---------------+
   173 
   174 If you go for the install, then you just follow the classical, but yet easy
   175 ./configure way:
   176   ./configure --prefix=/some/place
   177   make
   178   make install
   179   export PATH="${PATH}:/some/place/bin"
   180 
   181 You can then get rid of crosstool-NG source. Next create a directory to serve
   182 as a working place, cd in there and run:
   183   ct-ng help
   184 
   185 See below for complete usage.
   186 
   187 The Hacker's way |
   188 -----------------+
   189 
   190 If you go the hacker's way, then the usage is a bit different, although very
   191 simple:
   192   ./configure --local
   193   make
   194 
   195 Now, *do not* remove crosstool-NG sources. They are needed to run crosstool-NG!
   196 Stay in the directory holding the sources, and run:
   197   ./ct-ng help
   198 
   199 See below for complete usage.
   200 
   201 Now, provided you used a clone of the repository, you can send me your changes.
   202 See the section titled CONTRIBUTING, below, for how to submit changees.
   203 
   204 Preparing for packaging |
   205 ------------------------+
   206 
   207 If you plan on packaging crosstool-NG, you surely don't want to install it
   208 in your root file system. The install procedure of crosstool-NG honors the
   209 DESTDIR variable:
   210 
   211   ./configure --prefix=/usr
   212   make
   213   make DESTDIR=/packaging/place install
   214 
   215 Shell completion |
   216 -----------------+
   217 
   218 crosstool-NG comes with a shell script fragment that defines bash-compatible
   219 completion. That shell fragment is currently not installed automatically, but
   220 this is planned.
   221 
   222 To install the shell script fragment, you have two options:
   223  - install system-wide, most probably by copying ct-ng.comp into
   224    /etc/bash_completion.d/
   225  - install for a single user, by copying ct-ng.comp into ${HOME}/ and
   226    sourcing this file from your ${HOME}/.bashrc
   227 
   228 Contributed code |
   229 -----------------+
   230 
   231 Some people contibuted code that couldn't get merged for various reasons. This
   232 code is available as lzma-compressed patches, in the contrib/ sub-directory.
   233 These patches are to be applied to the source of crosstool-NG, prior to
   234 installing, using something like the following:
   235   lzcat contrib/foobar.patch.lzma |patch -p1
   236 
   237 There is no guarantee that a particuliar contribution applies to the current
   238 version of crosstool-ng, or that it will work at all. Use contributions at
   239 your own risk.
   240 
   241 
   242 ____________________________
   243                            /
   244 Configuring crosstool-NG  /
   245 _________________________/
   246 
   247 crosstool-NG is configured with a configurator presenting a menu-stuctured set
   248 of options. These options let you specify the way you want your toolchain
   249 built, where you want it installed, what architecture and specific processor it
   250 will support, the version of the components you want to use, etc... The
   251 value for those options are then stored in a configuration file.
   252 
   253 The configurator works the same way you configure your Linux kernel. It is
   254 assumed you now how to handle this.
   255 
   256 To enter the menu, type:
   257   ct-ng menuconfig
   258 
   259 Almost every config item has a help entry. Read them carefully.
   260 
   261 String and number options can refer to environment variables. In such a case,
   262 you must use the shell syntax: ${VAR}. You shall neither single- nor double-
   263 quote the string/number options.
   264 
   265 There are three environment variables that are computed by crosstool-NG, and
   266 that you can use:
   267 
   268 CT_TARGET:
   269   It represents the target tuple you are building for. You can use it for
   270   example in the installation/prefix directory, such as:
   271     /opt/x-tools/${CT_TARGET}
   272 
   273 CT_TOP_DIR:
   274   The top directory where crosstool-NG is running. You shouldn't need it in
   275   most cases. There is one case where you may need it: if you have local
   276   patches and you store them in your running directory, you can refer to them
   277   by using CT_TOP_DIR, such as:
   278     ${CT_TOP_DIR}/patches.myproject
   279 
   280 CT_VERSION:
   281   The version of crosstool-NG you are using. Not much use for you, but it's
   282   there if you need it.
   283 
   284 Interesting config options |
   285 ---------------------------+
   286 
   287 CT_LOCAL_TARBALLS_DIR:
   288   If you already have some tarballs in a direcotry, enter it here. That will
   289   speed up the retrieving phase, where crosstool-NG would otherwise download
   290   those tarballs.
   291 
   292 CT_PREFIX_DIR:
   293   This is where the toolchain will be installed in (and for now, where it
   294   will run from). Common use is to add the target tuple in the directory
   295   path, such as (see above):
   296     /opt/x-tools/${CT_TARGET}
   297 
   298 CT_TARGET_VENDOR:
   299   An identifier for your toolchain, will take place in the vendor part of the
   300   target tuple. It shall *not* contain spaces or dashes. Usually, keep it
   301   to a one-word string, or use underscores to separate words if you need.
   302   Avoid dots, commas, and special characters.
   303 
   304 CT_TARGET_ALIAS:
   305   An alias for the toolchian. It will be used as a prefix to the toolchain
   306   tools. For example, you will have ${CT_TARGET_ALIAS}-gcc
   307 
   308 Also, if you think you don't see enough versions, you can try to enable one of
   309 those:
   310 
   311 CT_OBSOLETE:
   312   Show obsolete versions or tools. Most of the time, you don't want to base
   313   your toolchain on too old a version (of gcc, for example). But at times, it
   314   can come handy to use such an old version for regression tests. Those old
   315   versions are hidden behind CT_OBSOLETE. Those versions (or features) are so
   316   marked because maintaining support for those in crosstool-NG would be too
   317   costly, time-wise, and time is dear.
   318 
   319 CT_EXPERIMENTAL:
   320   Show experimental versions or tools. Again, you might not want to base your
   321   toolchain on too recent tools (eg. gcc) for production. But if you need a
   322   feature present only in a recent version, or a new tool, you can find them
   323   hidden behind CT_EXPERIMENTAL. Those versions (or features) did not (yet)
   324   receive thorough testing in crosstool-NG, and/or are not mature enough to
   325   be blindly trusted.
   326 
   327 Re-building an existing toolchain |
   328 ----------------------------------+
   329 
   330 If you have an existing toolchain, you can re-use the options used to build it
   331 to create a new toolchain. That needs a very little bit of effort on your side
   332 but is quite easy. The options to build a toolchain are saved with the
   333 toolchain, and you can retrieve this configuration by running:
   334   ${CT_TARGET}-config
   335 
   336 This will dump the configuration to stdout, so to rebuild a toolchain with this
   337 configuration, the following is all you need to do:
   338   ${CT_TARGET}-config >.config
   339   ct-ng oldconfig
   340 
   341 Then, you can review and change the configuration by running:
   342   ct-ng menuconfig
   343 
   344 
   345 ________________________
   346                        /
   347 Running crosstool-NG  /
   348 _____________________/
   349 
   350 To build the toolchain, simply type:
   351   ct-ng build
   352 
   353 This will use the above configuration to retrieve, extract and patch the
   354 components, build, install and eventually test your newly built toolchain.
   355 
   356 You are then free to add the toolchain /bin directory in your PATH to use
   357 it at will.
   358 
   359 In any case, you can get some terse help. Just type:
   360   ct-ng help
   361 or:
   362   man 1 ct-ng
   363 
   364 Stopping and restarting a build |
   365 --------------------------------+
   366 
   367 If you want to stop the build after a step you are debugging, you can pass the
   368 variable STOP to make:
   369   ct-ng build STOP=some_step
   370 
   371 Conversely, if you want to restart a build at a specific step you are
   372 debugging, you can pass the RESTART variable to make:
   373   ct-ng build RESTART=some_step
   374 
   375 Alternatively, you can call make with the name of a step to just do that step:
   376   ct-ng libc_headers
   377 is equivalent to:
   378   ct-ng build RESTART=libc_headers STOP=libc_headers
   379 
   380 The shortcuts +step_name and step_name+ allow to respectively stop or restart
   381 at that step. Thus:
   382   ct-ng +libc_headers              and:    ct-ng libc_headers+
   383 are equivalent to:
   384   ct-ng build STOP=libc_headers    and:    ct-ng build RESTART=libc_headers
   385 
   386 To obtain the list of acceptable steps, please call:
   387   ct-ng list-steps
   388 
   389 Note that in order to restart a build, you'll have to say 'Y' to the config
   390 option CT_DEBUG_CT_SAVE_STEPS, and that the previous build effectively went
   391 that far.
   392 
   393 Building all toolchains at once |
   394 --------------------------------+
   395 
   396 You can build all samples; simply call:
   397   ct-ng build-all
   398 
   399 Overriding the number of // jobs |
   400 ---------------------------------+
   401 
   402 If you want to override the number of jobs to run in // (the -j option to
   403 make), you can either re-enter the menuconfig, or simply add it on the command
   404 line, as such:
   405   ct-ng build.4
   406 
   407 which tells crosstool-NG to override the number of // jobs to 4.
   408 
   409 You can see the actions that support overriding the number of // jobs in
   410 the help menu. Those are the ones with [.#] after them (eg. build[.#] or
   411 build-all[.#], and so on...).
   412 
   413 Note on // jobs |
   414 ----------------+
   415 
   416 The crosstool-NG script 'ct-ng' is a Makefile-script. It does *not* execute
   417 in parallel (there is not much to gain). When speaking of // jobs, we are
   418 refering to the number of // jobs when making the *components*. That is, we
   419 speak of the number of // jobs used to build gcc, glibc, and so on...
   420 
   421 Tools wrapper |
   422 --------------+
   423 
   424 Starting with gcc-4.3 come two new dependencies: GMP and MPFR. With gcc-4.4,
   425 come three new ones: GMP, PPL and CLooG/ppl. These are libraries that enable
   426 advanced features to gcc. Additionally, some of the libraries can be used by
   427 binutils and gdb. Unfortunately, not all systems on which crosstool-NG runs
   428 have all of those libraries. And for those that do, the versions of those
   429 libraries may be older than the version required by gcc.
   430 
   431 This is why crosstool-NG builds its own set of libraries as part of the
   432 toolchain.
   433 
   434 The libraries are built as shared libraries, because building them as static
   435 libraries has some short-comings. This poses no problem at build time, as
   436 crosstool-NG correctly points gcc (and binutils and gdb) to the correct
   437 place where our own version of the libraries are installed. But it poses
   438 a problem when gcc et al. are run: the place where the libraries are is most
   439 probably not known to the host dynamic linker. Still worse, if the host system
   440 has its own versions, then ld.so would load the wrong library!
   441 
   442 So we have to force the dynamic linker to load the correct version. We do this
   443 by using the LD_LIBRARY_PATH variable, that informs the dynamic linker where
   444 to look for shared libraries prior to searching its standard places. But we
   445 can't impose that burden on all the system (because it'd be a nightmare to
   446 configure, and because two toolchains on the same system may use different
   447 versions of the libraries); so we have to do it on a per-toolchain basis.
   448 
   449 So we rename all binaries of the toolchain (by adding a dot '.' as their first
   450 character), and add a small program, the so-called "tools wrapper", that
   451 correctly sets LD_LIBRARY_PATH prior to running the real tool.
   452 
   453 First, the wrapper was written as a POSIX-compliant shell script. That shell
   454 script is very simple, if not trivial, and works great. The only drawback is
   455 that it does not work on host systems that lack a shell, for example the
   456 MingW32 environment. To solve the issue, the wrapper has been re-written in C,
   457 and compiled at build time. This C wrapper is much more complex than the shell
   458 script, and although it sems to be working, it's been only lightly tested.
   459 Some of the expected short-comings with this C wrapper are;
   460  - multi-byte file names may not be handled correctly
   461  - it's really big for what it does
   462 
   463 So, the default wrapper installed with your toolchain is the shell script.
   464 If you know that your system is missing a shell, then you shall use the C
   465 wrapper (and report back whether it works, or does not work, for you).
   466 
   467 
   468 _______________________
   469                       /
   470 Using the toolchain  /
   471 ____________________/
   472 
   473 Using the toolchain is as simple as adding the toolchain's bin directory in
   474 your PATH, such as:
   475   export PATH="${PATH}:/your/toolchain/path/bin"
   476 
   477 and then using the target tuple to tell the build systems to use your
   478 toolchain:
   479   ./configure --target=your-target-tuple
   480 or
   481   make CC=your-target-tuple-gcc
   482 or
   483   make CROSS_COMPILE=your-target-tuple-
   484 and so on...
   485 
   486 It is strongly advised not to use the toolchain sys-root directory as an
   487 install directory for your programs/packages. If you do so, you will not be
   488 able to use your toolchain for another project. It is even strongly advised
   489 that your toolchain is chmod-ed to read-only once successfully build, so that
   490 you don't go polluting your toolchain with your programs/packages' files.
   491 
   492 Thus, when you build a program/package, install it in a separate directory,
   493 eg. /your/root. This directory is the /image/ of what would be in the root file
   494 system of your target, and will contain all that your programs/packages have
   495 installed.
   496 
   497 The 'populate' script |
   498 ----------------------+
   499 
   500 When your root directory is ready, it is still missing some important bits: the
   501 toolchain's libraries. To populate your root directory with those libs, just
   502 run:
   503   your-target-tuple-populate -s /your/root -d /your/root-populated
   504 
   505 This will copy /your/root into /your/root-populated, and put the needed and only
   506 the needed libraries there. Thus you don't polute /your/root with any cruft that
   507 would no longer be needed should you have to remove stuff. /your/root always
   508 contains only those things you install in it.
   509 
   510 You can then use /your/root-populated to build up your file system image, a
   511 tarball, or to NFS-mount it from your target, or whatever you need.
   512 
   513 The populate script accepts the following options:
   514 
   515  -s src_dir
   516     Use 'src_dir' as the un-populated root directory.
   517 
   518  -d dst_dir
   519     Put the populated root directory in 'dst_dir'.
   520 
   521  -l lib1 [...]
   522     Always add specified libraries.
   523 
   524  -L file
   525     Always add libraries listed in 'file'.
   526 
   527  -f
   528     Remove 'dst_dir' if it previously existed; continue even if any library
   529     specified with -l or -L is missing.
   530 
   531  -v
   532     Be verbose, and tell what's going on (you can see exactly where libs are
   533     coming from).
   534 
   535  -h
   536     Print the help.
   537 
   538 See 'your-target-tuple-populate -h' for more information on the options.
   539 
   540 Here is how populate works:
   541 
   542   1) performs some sanity checks:
   543      - src_dir and dst_dir are specified
   544      - src_dir exists
   545      - unless forced, dst_dir does not exist
   546      - src_dir != dst_dir
   547 
   548   2) copy src_dir to dst_dir
   549 
   550   3) add forced libraries to dst_dir
   551      - build the list from -l and -L options
   552      - get forced libraries from the sysroot (see below for heuristics)
   553        - abort on the first missing library, unless -f is specified
   554 
   555   4) add all missing libraries to dst_dir
   556      - scan dst_dir for every ELF files that are 'executable' or
   557        'shared object'
   558      - list the "NEEDED Shared library" fields
   559        - check if the library is already in dst_dir/lib or dst_dir/usr/lib
   560        - if not, get the library from the sysroot
   561          - if it's in sysroot/lib, copy it to dst_dir/lib
   562          - if it's in sysroot/usr/lib, copy it to dst_dir/usr/lib
   563          - in both cases, use the SONAME of the library to create the file
   564            in dst_dir
   565          - if it was not found in the sysroot, this is an error.
   566 
   567 
   568 ___________________
   569                   /
   570 Toolchain types  /
   571 ________________/
   572 
   573 There are four kinds of toolchains you could encounter.
   574 
   575 First off, you must understand the following: when it comes to compilers there
   576 are up to four machines involved:
   577   1) the machine configuring the toolchain components: the config machine
   578   2) the machine building the toolchain components:    the build machine
   579   3) the machine running the toolchain:                the host machine
   580   4) the machine the toolchain is generating code for: the target machine
   581 
   582 We can most of the time assume that the config machine and the build machine
   583 are the same. Most of the time, this will be true. The only time it isn't
   584 is if you're using distributed compilation (such as distcc). Let's forget
   585 this for the sake of simplicity.
   586 
   587 So we're left with three machines:
   588  - build
   589  - host
   590  - target
   591 
   592 Any toolchain will involve those three machines. You can be as pretty sure of
   593 this as "2 and 2 are 4". Here is how they come into play:
   594 
   595 1) build == host == target
   596     This is a plain native toolchain, targetting the exact same machine as the
   597     one it is built on, and running again on this exact same machine. You have
   598     to build such a toolchain when you want to use an updated component, such
   599     as a newer gcc for example.
   600     crosstool-NG calls it "native".
   601 
   602 2) build == host != target
   603     This is a classic cross-toolchain, which is expected to be run on the same
   604     machine it is compiled on, and generate code to run on a second machine,
   605     the target.
   606     crosstool-NG calls it "cross".
   607 
   608 3) build != host == target
   609     Such a toolchain is also a native toolchain, as it targets the same machine
   610     as it runs on. But it is build on another machine. You want such a
   611     toolchain when porting to a new architecture, or if the build machine is
   612     much faster than the host machine.
   613     crosstool-NG calls it "cross-native".
   614 
   615 4) build != host != target
   616     This one is called a canadian-toolchain (*), and is tricky. The three
   617     machines in play are different. You might want such a toolchain if you
   618     have a fast build machine, but the users will use it on another machine,
   619     and will produce code to run on a third machine.
   620     crosstool-NG calls it "canadian".
   621 
   622 crosstool-NG can build all these kinds of toolchains (or is aiming at it,
   623 anyway!)
   624 
   625 (*) The term Canadian Cross came about because at the time that these issues
   626     were all being hashed out, Canada had three national political parties.
   627     http://en.wikipedia.org/wiki/Cross_compiler
   628 
   629 
   630 ________________
   631                /
   632 Contributing  /
   633 _____________/
   634 
   635 Sending a bug report |
   636 ---------------------+
   637 
   638 If you need to send a bug report, please send a mail with subject
   639 prefixed with "[CT_NG]" with to following destinations:
   640     TO: yann.morin.1998 (at) anciens.enib.fr
   641     CC: crossgcc (at) sourceware.org
   642 
   643 Sending patches |
   644 ----------------+
   645 
   646 If you want to enhance crosstool-NG, there's a to-do list in the TODO file.
   647 
   648 Patches should come with the appropriate SoB line. A SoB line is typically
   649 something like:
   650    Signed-off-by: John DOE <john.doe@somewhere.net>
   651 
   652 The SoB line is clearly described in Documentation/SubmittingPatches , section
   653 12, of your favourite Linux kernel source tree.
   654 
   655 Then you'll need to correctly configure Mercurial. There are two extensions
   656 that you may find usefull:
   657   - mq        : http://mercurial.selenic.com/wiki/MqExtension
   658   - patchbomb : http://mercurial.selenic.com/wiki/PatchbombExtension
   659 
   660 Commit messages should look like (without leading pipes):
   661  |component: short, one-line description
   662  |
   663  |optional longer description
   664  |on multiple lines if needed
   665 
   666 Here is an example commit message (see revision a53a5e1d61db):
   667  |comp-libs/cloog: fix building
   668  |
   669  |For CLooG/PPL 0.15.3, the directory name was simply cloog-ppl.
   670  |For any later versions, the directory name does have the version, such as
   671  |cloog-ppl-0.15.4.
   672 
   673 Here's a typical hacking session:
   674   hg clone http://ymorin.is-a-geek.org/hg/crosstool-ng crosstool-ng
   675   cd crosstool-ng
   676   hg qinit
   677   hg qnew -D -U -e my_first_patch
   678   *edit patch description*
   679   *hack* *hack* *check* *fails* *hack* *hack* *check* *works*
   680   hg qref -D -e
   681   *edit patch description, serving as commit message*
   682   hg qnew -D -U -e my_second_patch
   683   *edit patch description*
   684   *hack* *hack* *check* *fails* *hack* *hack* *check* *works*
   685   hg qref -D -e
   686   *edit patch description, serving as commit message*
   687   hg email --outgoing --intro   \
   688            --from '"Your Full NAME" <your.email (at) your.domain>'   \
   689            --to '"Yann E. MORIN" <yann.morin.1998 (at) anciens.enib.fr>'    \
   690            --cc 'crossgcc (at) sourceware.org'
   691   *edit introductory message*
   692   *wait for feedback*
   693   *re-send if no answer for a few days*
   694 
   695 Note: replace '(at)' above with a plain '@'.
   696 
   697 
   698 _____________
   699             /
   700 Internals  /
   701 __________/
   702 
   703 Internally, crosstool-NG is script-based. To ease usage, the frontend is
   704 Makefile-based.
   705 
   706 Makefile front-end |
   707 -------------------+
   708 
   709 The entry point to crosstool-NG is the Makefile script "ct-ng". Calling this
   710 script with an action will act exactly as if the Makefile was in the current
   711 working directory and make was called with the action as rule. Thus:
   712   ct-ng menuconfig
   713 
   714 is equivalent to having the Makefile in CWD, and calling:
   715   make menuconfig
   716 
   717 Having ct-ng as it is avoids copying the Makefile everywhere, and acts as a
   718 traditional command.
   719 
   720 ct-ng loads sub- Makefiles from the library directory $(CT_LIB_DIR), as set up
   721 at configuration time with ./configure.
   722 
   723 ct-ng also searches for config files, sub-tools, samples, scripts and patches in
   724 that library directory.
   725 
   726 Because of a stupid make behavior/bug I was unable to track down, implicit make
   727 rules are disabled: installing with --local would triger those rules, and mconf
   728 was unbuildable.
   729 
   730 Kconfig parser |
   731 ---------------+
   732 
   733 The kconfig language is a hacked version, vampirised from the Linux kernel
   734 (http://www.kernel.org/), and (heavily) adapted to my needs.
   735 
   736 The list of the most notable changes (at least the ones I remember) follows:
   737 - the CONFIG_ prefix has been replaced with CT_
   738 - a leading | in prompts is skipped, and subsequent leading spaces are not
   739   trimmed
   740 - otherwise leading spaces are silently trimmed
   741 
   742 The kconfig parsers (conf and mconf) are not installed pre-built, but as
   743 source files. Thus you can have the directory where crosstool-NG is installed,
   744 exported (via NFS or whatever) and have clients with different architectures
   745 use the same crosstool-NG installation, and most notably, the same set of
   746 patches.
   747 
   748 Architecture-specific |
   749 ----------------------+
   750 
   751 Note: this chapter is not really well written, and might thus be a little bit
   752 complex to understand. To get a better grasp of what an architecture is, the
   753 reader is kindly encouraged to look at the "arch/" sub-directory, and to the
   754 existing architectures to see how things are laid out.
   755 
   756 An architecture is defined by:
   757 
   758  - a human-readable name, in lower case letters, with numbers as appropriate.
   759    The underscore is allowed; space and special characters are not.
   760      Eg.: arm, x86_64
   761  - a file in "config/arch/", named after the architecture's name, and suffixed
   762    with ".in".
   763      Eg.: config/arch/arm.in
   764  - a file in "scripts/build/arch/", named after the architecture's name, and
   765    suffixed with ".sh".
   766      Eg.: scripts/build/arch/arm.sh
   767 
   768 The architecture's ".in" file API:
   769  > the config option "ARCH_%arch%" (where %arch% is to be replaced with the
   770    actual architecture name).
   771    That config option must have *neither* a type, *nor* a prompt! Also, it can
   772    *not* depend on any other config option (EXPERIMENTAL is managed as above).
   773      Eg.:
   774        config ARCH_arm
   775    + mandatory:
   776        defines a (terse) help entry for this architecture:
   777        Eg.:
   778          config ARCH_arm
   779            help
   780              The ARM architecture.
   781    + optional:
   782        selects adequate associated config options.
   783        Note: 64-bit architectures *shall* select ARCH_64
   784        Eg.:
   785          config ARCH_arm
   786            select ARCH_SUPPORTS_BOTH_ENDIAN
   787            select ARCH_DEFAULT_LE
   788            help
   789              The ARM architecture.
   790        Eg.:
   791          config ARCH_x86_64
   792             select ARCH_64
   793             help
   794               The x86_64 architecture.
   795 
   796  > other target-specific options, at your discretion. Note however that to
   797    avoid name-clashing, such options shall be prefixed with "ARCH_%arch%",
   798    where %arch% is again replaced by the actual architecture name.
   799    (Note: due to historical reasons, and lack of time to clean up the code,
   800     I may have left some config options that do not completely conform to
   801     this, as the architecture name was written all upper case. However, the
   802     prefix is unique among architectures, and does not cause harm).
   803 
   804 The architecture's ".sh" file API:
   805  > the function "CT_DoArchTupleValues"
   806    + parameters: none
   807    + environment:
   808      - all variables from the ".config" file,
   809      - the two variables "target_endian_eb" and "target_endian_el" which are
   810        the endianness suffixes
   811    + return value: 0 upon success, !0 upon failure
   812    + provides:
   813      - mandatory
   814      - the environment variable CT_TARGET_ARCH
   815      - contains:
   816        the architecture part of the target tuple.
   817        Eg.: "armeb" for big endian ARM
   818             "i386" for an i386
   819    + provides:
   820      - optional
   821      - the environment variable CT_TARGET_SYS
   822      - contains:
   823        the sytem part of the target tuple.
   824        Eg.: "gnu" for glibc on most architectures
   825             "gnueabi" for glibc on an ARM EABI
   826      - defaults to:
   827        - for glibc-based toolchain: "gnu"
   828        - for uClibc-based toolchain: "uclibc"
   829    + provides:
   830      - optional
   831      - the environment variable CT_KERNEL_ARCH
   832      - contains:
   833        the architecture name as understandable by the Linux kernel build
   834        system.
   835        Eg.: "arm" for an ARM
   836             "powerpc" for a PowerPC
   837             "i386" for an x86
   838      - defaults to:
   839        ${CT_ARCH}
   840    + provides:
   841      - optional
   842      - the environment variables to configure the cross-gcc (defaults)
   843        - CT_ARCH_WITH_ARCH    : the gcc ./configure switch to select architecture level         ( "--with-arch=${CT_ARCH_ARCH}"   )
   844        - CT_ARCH_WITH_ABI     : the gcc ./configure switch to select ABI level                  ( "--with-abi=${CT_ARCH_ABI}"     )
   845        - CT_ARCH_WITH_CPU     : the gcc ./configure switch to select CPU instruction set        ( "--with-cpu=${CT_ARCH_CPU}"     )
   846        - CT_ARCH_WITH_TUNE    : the gcc ./configure switch to select scheduling                 ( "--with-tune=${CT_ARCH_TUNE}"   )
   847        - CT_ARCH_WITH_FPU     : the gcc ./configure switch to select FPU type                   ( "--with-fpu=${CT_ARCH_FPU}"     )
   848        - CT_ARCH_WITH_FLOAT   : the gcc ./configure switch to select floating point arithmetics ( "--with-float=soft" or /empty/  )
   849    + provides:
   850      - optional
   851      - the environment variables to pass to the cross-gcc to build target binaries (defaults)
   852        - CT_ARCH_ARCH_CFLAG   : the gcc switch to select architecture level                     ( "-march=${CT_ARCH_ARCH}"            )
   853        - CT_ARCH_ABI_CFLAG    : the gcc switch to select ABI level                              ( "-mabi=${CT_ARCH_ABI}"              )
   854        - CT_ARCH_CPU_CFLAG    : the gcc switch to select CPU instruction set                    ( "-mcpu=${CT_ARCH_CPU}"              )
   855        - CT_ARCH_TUNE_CFLAG   : the gcc switch to select scheduling                             ( "-mtune=${CT_ARCH_TUNE}"            )
   856        - CT_ARCH_FPU_CFLAG    : the gcc switch to select FPU type                               ( "-mfpu=${CT_ARCH_FPU}"              )
   857        - CT_ARCH_FLOAT_CFLAG  : the gcc switch to choose floating point arithmetics             ( "-msoft-float" or /empty/           )
   858        - CT_ARCH_ENDIAN_CFLAG : the gcc switch to choose big or little endian                   ( "-mbig-endian" or "-mlittle-endian" )
   859      - default to:
   860        see above.
   861    + provides:
   862      - optional
   863      - the environement variables to configure the core and final compiler, specific to this architecture:
   864        - CT_ARCH_CC_CORE_EXTRA_CONFIG   : additional, architecture specific core gcc ./configure flags
   865        - CT_ARCH_CC_EXTRA_CONFIG        : additional, architecture specific final gcc ./configure flags
   866      - default to:
   867        - all empty
   868    + provides:
   869      - optional
   870      - the architecture-specific CFLAGS and LDFLAGS:
   871        - CT_ARCH_TARGET_CLFAGS
   872        - CT_ARCH_TARGET_LDFLAGS
   873      - default to:
   874        - all empty
   875 
   876 You can have a look at "config/arch/arm.in" and "scripts/build/arch/arm.sh" for
   877 a quite complete example of what an actual architecture description looks like.
   878 
   879 Kernel specific |
   880 ----------------+
   881 
   882 A kernel is defined by:
   883 
   884  - a human-readable name, in lower case letters, with numbers as appropriate.
   885    The underscore is allowed; space and special characters are not (although
   886    they are internally replaced with underscores.
   887      Eg.: linux, bare-metal
   888  - a file in "config/kernel/", named after the kernel name, and suffixed with
   889    ".in".
   890      Eg.: config/kernel/linux.in, config/kernel/bare-metal.in
   891  - a file in "scripts/build/kernel/", named after the kernel name, and suffixed
   892    with ".sh".
   893      Eg.: scripts/build/kernel/linux.sh, scripts/build/kernel/bare-metal.sh
   894 
   895 The kernel's ".in" file must contain:
   896  > an optional lines containing exactly "# EXPERIMENTAL", starting on the
   897    first column, and without any following space or other character.
   898    If this line is present, then this kernel is considered EXPERIMENTAL,
   899    and correct dependency on EXPERIMENTAL will be set.
   900 
   901  > the config option "KERNEL_%kernel_name%" (where %kernel_name% is to be
   902    replaced with the actual kernel name, with all special characters and
   903    spaces replaced by underscores).
   904    That config option must have *neither* a type, *nor* a prompt! Also, it can
   905    *not* depends on EXPERIMENTAL.
   906      Eg.: KERNEL_linux, KERNEL_bare_metal
   907    + mandatory:
   908        defines a (terse) help entry for this kernel.
   909        Eg.:
   910          config KERNEL_bare_metal
   911            help
   912              Build a compiler for use without any kernel.
   913    + optional:
   914        selects adequate associated config options.
   915        Eg.:
   916          config KERNEL_bare_metal
   917            select BARE_METAL
   918            help
   919              Build a compiler for use without any kernel.
   920 
   921  > other kernel specific options, at your discretion. Note however that, to
   922    avoid name-clashing, such options should be prefixed with
   923    "KERNEL_%kernel_name%", where %kernel_name% is again tp be replaced with
   924    the actual kernel name.
   925    (Note: due to historical reasons, and lack of time to clean up the code,
   926     I may have left some config options that do not completely conform to
   927     this, as the kernel name was written all upper case. However, the prefix
   928     is unique among kernels, and does not cause harm).
   929 
   930 The kernel's ".sh" file API:
   931  > is a bash script fragment
   932 
   933  > defines the function CT_DoKernelTupleValues
   934    + see the architecture's CT_DoArchTupleValues, except for:
   935    + set the environment variable CT_TARGET_KERNEL, the kernel part of the
   936      target tuple
   937    + return value: ignored
   938 
   939  > defines the function "do_kernel_get":
   940    + parameters: none
   941    + environment:
   942       - all variables from the ".config" file.
   943    + return value: 0 for success, !0 for failure.
   944    + behavior: download the kernel's sources, and store the tarball into
   945      "${CT_TARBALLS_DIR}". To this end, a functions is available, that
   946      abstracts downloading tarballs:
   947      - CT_DoGet <tarball_base_name> <URL1 [URL...]>
   948        Eg.: CT_DoGet linux-2.6.26.5 ftp://ftp.kernel.org/pub/linux/kernel/v2.6
   949      Note: retrieving sources from svn, cvs, git and the likes is not supported
   950      by CT_DoGet. You'll have to do this by hand, as it is done for eglibc in
   951      "scripts/build/libc/eglibc.sh"
   952 
   953  > defines the function "do_kernel_extract":
   954    + parameters: none
   955    + environment:
   956       - all variables from the ".config" file,
   957    + return value: 0 for success, !0 for failure.
   958    + behavior: extract the kernel's tarball into "${CT_SRC_DIR}", and apply
   959      required patches. To this end, a function is available, that abstracts
   960      extracting tarballs:
   961      - CT_ExtractAndPatch <tarball_base_name>
   962        Eg.: CT_ExtractAndPatch linux-2.6.26.5
   963 
   964  > defines the function "do_kernel_headers":
   965    + parameters: none
   966    + environment:
   967       - all variables from the ".config" file,
   968    + return value: 0 for success, !0 for failure.
   969    + behavior: install the kernel headers (if any) in "${CT_SYSROOT_DIR}/usr/include"
   970 
   971  > defines any kernel-specific helper functions
   972    These functions, if any, must be prefixed with "do_kernel_%CT_KERNEL%_",
   973    where '%CT_KERNEL%' is to be replaced with the actual kernel name, to avoid
   974    any name-clashing.
   975 
   976 You can have a look at "config/kernel/linux.in" and "scripts/build/kernel/linux.sh"
   977 as an example of what a complex kernel description looks like.
   978 
   979 Adding a new version of a component |
   980 ------------------------------------+
   981 
   982 When a new component, such as the Linux kernel, gcc or any other is released,
   983 adding the new version to crosstool-NG is quite easy. There is a script that
   984 will do all that for you:
   985   scripts/addToolVersion.sh
   986 
   987 Run it with no option to get some help.
   988 
   989 Build scripts |
   990 --------------+
   991 
   992 To Be Written later...