docs/overview.txt
author "Yann E. MORIN" <yann.morin.1998@anciens.enib.fr>
Fri Oct 02 22:10:38 2009 +0200 (2009-10-02)
changeset 1551 8c40b842e798
parent 1513 0a405ac9d9ce
child 1554 3b8df55987ae
permissions -rw-r--r--
libc/glibc: fix building for seemingly native toolchains

Build glibc with -O2 as a fix/workaround to building
seemingly-native toolchains.

See:
- docs/overview.txt
- docs/known-issues.txt
- http://sourceware.org/ml/crossgcc/2009-09/msg00055.html
     1 File.........: overview.txt
     2 Content......: Overview of how crosstool-NG works.
     3 Copyrigth....: (C) 2007 Yann E. MORIN <yann.morin.1998@anciens.enib.fr>
     4 License......: Creative Commons Attribution Share Alike (CC-by-sa), v2.5
     5 
     6 ____________________
     7                    /
     8 Table Of Content  /
     9 _________________/
    10 
    11 
    12 Introduction
    13 History
    14 Installing crosstool-NG
    15   Install method
    16   The hacker's way
    17   Preparing for packaging
    18   Shell completion
    19   Contributed code
    20 Configuring crosstool-NG
    21   Interesting config options
    22   Re-building an existing toolchain
    23 Running crosstool-NG
    24   Stopping and restarting a build
    25   Testing all toolchains at once
    26   Overriding the number of // jobs
    27   Note on // jobs
    28   Tools wrapper
    29 Using the toolchain
    30 Toolchain types
    31   Seemingly-native toolchains
    32 Internals
    33   Makefile front-end
    34   Kconfig parser
    35   Architecture-specific
    36   Adding a new version of a component
    37   Build scripts
    38 
    39 ________________
    40                /
    41 Introduction  /
    42 _____________/
    43 
    44 crosstool-NG aims at building toolchains. Toolchains are an essential component
    45 in a software development project. It will compile, assemble and link the code
    46 that is being developed. Some pieces of the toolchain will eventually end up
    47 in the resulting binary/ies: static libraries are but an example.
    48 
    49 So, a toolchain is a very sensitive piece of software, as any bug in one of the
    50 components, or a poorly configured component, can lead to execution problems,
    51 ranging from poor performance, to applications ending unexpectedly, to
    52 mis-behaving software (which more than often is hard to detect), to hardware
    53 damage, or even to human risks (which is more than regrettable).
    54 
    55 Toolchains are made of different piece of software, each being quite complex
    56 and requiring specially crafted options to build and work seamlessly. This
    57 is usually not that easy, even in the not-so-trivial case of native toolchains.
    58 The work reaches a higher degree of complexity when it comes to cross-
    59 compilation, where it can become quite a nightmare...
    60 
    61 Some cross-toolchains exist on the internet, and can be used for general
    62 development, but they have a number of limitations:
    63   - they can be general purpose, in that they are configured for the majority:
    64     no optimisation for your specific target,
    65   - they can be prepared for a specific target and thus are not easy to use,
    66     nor optimised for, or even supporting your target,
    67   - they often are using aging components (compiler, C library, etc...) not
    68     supporting special features of your shiny new processor;
    69 On the other side, these toolchain offer some advantages:
    70   - they are ready to use and quite easy to install and setup,
    71   - they are proven if used by a wide community.
    72 
    73 But once you want to get all the juice out of your specific hardware, you will
    74 want to build your own toolchain. This is where crosstool-NG comes into play.
    75 
    76 There are also a number of tools that build toolchains for specific needs,
    77 which are not really scalable. Examples are:
    78   - buildroot (buildroot.uclibc.org) whose main purpose is to build root file
    79     systems, hence the name. But once you have your toolchain with buildroot,
    80     part of it is installed in the root-to-be, so if you want to build a whole
    81     new root, you either have to save the existing one as a template and
    82     restore it later, or restart again from scratch. This is not convenient,
    83   - ptxdist (www.pengutronix.de/software/ptxdist), whose purpose is very
    84     similar to buildroot,
    85   - other projects (openembedded.org for example), which are again used to
    86     build root file systems.
    87 
    88 crosstool-NG is really targeted at building toolchains, and only toolchains.
    89 It is then up to you to use it the way you want.
    90 
    91 ___________
    92           /
    93 History  /
    94 ________/
    95 
    96 crosstool was first 'conceived' by Dan Kegel, who offered it to the community
    97 as a set of scripts, a repository of patches, and some pre-configured, general
    98 purpose setup files to be used to configure crosstool. This is available at
    99 http://www.kegel.com/crosstool, and the subversion repository is hosted on
   100 google at http://code.google.com/p/crosstool/.
   101 
   102 I once managed to add support for uClibc-based toolchains, but it did not make
   103 into mainline, mostly because I didn't have time to port the patch forward to
   104 the new versions, due in part to the big effort it was taking.
   105 
   106 So I decided to clean up crosstool in the state it was, re-order the things
   107 in place, add appropriate support for what I needed, that is uClibc support
   108 and a menu-driven configuration, named the new implementation crosstool-NG,
   109 (standing for crosstool Next Generation, as many other comunity projects do,
   110 and as a wink at the TV series "Star Trek: The Next Generation" ;-) ) and
   111 made it available to the community, in case it was of interest to any one.
   112 
   113 ___________________________
   114                           /
   115 Installing crosstool-NG  /
   116 ________________________/
   117 
   118 There are two ways you can use crosstool-NG:
   119  - build and install it, then get rid of the sources like you'd do for most
   120    programs,
   121  - or only build it and run from the source directory.
   122 
   123 The former should be used if you got crosstool-NG from a packaged tarball, see
   124 "Install method", below, while the latter is most useful for developpers that
   125 checked the code out from SVN, and want to submit patches, see "The Hacker's
   126 way", below.
   127 
   128 Install method |
   129 ---------------+
   130 
   131 If you go for the install, then you just follow the classical, but yet easy
   132 ./configure way:
   133   ./configure --prefix=/some/place
   134   make
   135   make install
   136   export PATH="${PATH}:/some/place/bin"
   137 
   138 You can then get rid of crosstool-NG source. Next create a directory to serve
   139 as a working place, cd in there and run:
   140   ct-ng help
   141 
   142 See below for complete usage.
   143 
   144 The Hacker's way |
   145 -----------------+
   146 
   147 If you go the hacker's way, then the usage is a bit different, although very
   148 simple:
   149   ./configure --local
   150   make
   151 
   152 Now, *do not* remove crosstool-NG sources. They are needed to run crosstool-NG!
   153 Stay in the directory holding the sources, and run:
   154   ./ct-ng help
   155 
   156 See below for complete usage.
   157 
   158 Now, provided you checked-out the code, you can send me your interesting changes
   159 by running:
   160   svn diff
   161 
   162 and mailing me the result! :-P
   163 
   164 Preparing for packaging |
   165 ------------------------+
   166 
   167 If you plan on packaging crosstool-NG, you surely don't want to install it
   168 in your root file system. The install procedure of crosstool-NG honors the
   169 DESTDIR variable:
   170 
   171   ./configure --prefix=/usr
   172   make
   173   make DESTDIR=/packaging/place install
   174 
   175 Shell completion |
   176 -----------------+
   177 
   178 crosstool-NG comes with a shell script fragment that defines bash-compatible
   179 completion. That shell fragment is currently not installed automatically, but
   180 this is planned.
   181 
   182 To install the shell script fragment, you have two options:
   183  - install system-wide, most probably by copying ct-ng.comp into
   184    /etc/bash_completion.d/
   185  - install for a single user, by copying ct-ng.comp into ${HOME}/ and
   186    sourcing this file from your ${HOME}/.bashrc
   187 
   188 Contributed code |
   189 -----------------+
   190 
   191 Some people contibuted code that couldn't get merged for various reasons. This
   192 code is available as patches in the contrib/ sub-directory. These patches are
   193 to be applied to the source of crosstool-NG, prior to installing.
   194 
   195 An easy way to use contributed code is to pass the --with-contrib= option to
   196 ./configure. The possible values depend upon which contributions are packaged
   197 with your version, but you can get with it with passing one of those two
   198 special values:
   199   --with-contrib=list
   200     will list all available contributions
   201 
   202   --with-contrib=all
   203     will select all avalaible contributions
   204 
   205 There is no guarantee that a particuliar contribution applies to the current
   206 version of crosstool-ng, or that it will work at all. Use contributions at
   207 your own risk.
   208 
   209 ____________________________
   210                            /
   211 Configuring crosstool-NG  /
   212 _________________________/
   213 
   214 crosstool-NG is configured with a configurator presenting a menu-stuctured set
   215 of options. These options let you specify the way you want your toolchain
   216 built, where you want it installed, what architecture and specific processor it
   217 will support, the version of the components you want to use, etc... The
   218 value for those options are then stored in a configuration file.
   219 
   220 The configurator works the same way you configure your Linux kernel. It is
   221 assumed you now how to handle this.
   222 
   223 To enter the menu, type:
   224   ct-ng menuconfig
   225 
   226 Almost every config item has a help entry. Read them carefully.
   227 
   228 String and number options can refer to environment variables. In such a case,
   229 you must use the shell syntax: ${VAR}. You shall neither single- nor double-
   230 quote the string/number options.
   231 
   232 There are three environment variables that are computed by crosstool-NG, and
   233 that you can use:
   234 
   235 CT_TARGET:
   236   It represents the target tuple you are building for. You can use it for
   237   example in the installation/prefix directory, such as:
   238     /opt/x-tools/${CT_TARGET}
   239 
   240 CT_TOP_DIR:
   241   The top directory where crosstool-NG is running. You shouldn't need it in
   242   most cases. There is one case where you may need it: if you have local
   243   patches and you store them in your running directory, you can refer to them
   244   by using CT_TOP_DIR, such as:
   245     ${CT_TOP_DIR}/patches.myproject
   246 
   247 CT_VERSION:
   248   The version of crosstool-NG you are using. Not much use for you, but it's
   249   there if you need it.
   250 
   251 Interesting config options |
   252 ---------------------------+
   253 
   254 CT_LOCAL_TARBALLS_DIR:
   255   If you already have some tarballs in a direcotry, enter it here. That will
   256   speed up the retrieving phase, where crosstool-NG would otherwise download
   257   those tarballs.
   258 
   259 CT_PREFIX_DIR:
   260   This is where the toolchain will be installed in (and for now, where it
   261   will run from). Common use is to add the target tuple in the directory
   262   path, such as (see above):
   263     /opt/x-tools/${CT_TARGET}
   264 
   265 CT_TARGET_VENDOR:
   266   An identifier for your toolchain, will take place in the vendor part of the
   267   target tuple. It shall *not* contain spaces or dashes. Usually, keep it
   268   to a one-word string, or use underscores to separate words if you need.
   269   Avoid dots, commas, and special characters.
   270 
   271 CT_TARGET_ALIAS:
   272   An alias for the toolchian. It will be used as a prefix to the toolchain
   273   tools. For example, you will have ${CT_TARGET_ALIAS}-gcc
   274 
   275 Also, if you think you don't see enough versions, you can try to enable one of
   276 those:
   277 
   278 CT_OBSOLETE:
   279   Show obsolete versions or tools. Most of the time, you don't want to base
   280   your toolchain on too old a version (of gcc, for example). But at times, it
   281   can come handy to use such an old version for regression tests. Those old
   282   versions are hidden behind CT_OBSOLETE. Those versions (or features) are so
   283   marked because maintaining support for those in crosstool-NG would be too
   284   costly, time-wise, and time is dear.
   285 
   286 CT_EXPERIMENTAL:
   287   Show experimental versions or tools. Again, you might not want to base your
   288   toolchain on too recent tools (eg. gcc) for production. But if you need a
   289   feature present only in a recent version, or a new tool, you can find them
   290   hidden behind CT_EXPERIMENTAL. Those versions (or features) did not (yet)
   291   receive thorough testing in crosstool-NG, and/or are not mature enough to
   292   be blindly trusted.
   293 
   294 Re-building an existing toolchain |
   295 ----------------------------------+
   296 
   297 If you have an existing toolchain, you can re-use the options used to build it
   298 to create a new toolchain. That needs a very little bit of effort on your side
   299 but is quite easy. The options to build a toolchain are saved with the
   300 toolchain, and you can retrieve this configuration by running:
   301   ${CT_TARGET}-config
   302 
   303 This will dump the configuration to stdout, so to rebuild a toolchain with this
   304 configuration, the following is all you need to do:
   305   ${CT_TARGET}-config >.config
   306   ct-ng oldconfig
   307 
   308 Then, you can review and change the configuration by running:
   309   ct-ng menuconfig
   310 
   311 ________________________
   312                        /
   313 Running crosstool-NG  /
   314 _____________________/
   315 
   316 To build the toolchain, simply type:
   317   ct-ng build
   318 
   319 This will use the above configuration to retrieve, extract and patch the
   320 components, build, install and eventually test your newly built toolchain.
   321 
   322 You are then free to add the toolchain /bin directory in your PATH to use
   323 it at will.
   324 
   325 In any case, you can get some terse help. Just type:
   326   ct-ng help
   327 or:
   328   man 1 ct-ng
   329 
   330 Stopping and restarting a build |
   331 --------------------------------+
   332 
   333 If you want to stop the build after a step you are debugging, you can pass the
   334 variable STOP to make:
   335   ct-ng STOP=some_step
   336 
   337 Conversely, if you want to restart a build at a specific step you are
   338 debugging, you can pass the RESTART variable to make:
   339   ct-ng RESTART=some_step
   340 
   341 Alternatively, you can call make with the name of a step to just do that step:
   342   ct-ng libc_headers
   343 is equivalent to:
   344   ct-ng RESTART=libc_headers STOP=libc_headers
   345 
   346 The shortcuts +step_name and step_name+ allow to respectively stop or restart
   347 at that step. Thus:
   348   ct-ng +libc_headers        and:    ct-ng libc_headers+
   349 are equivalent to:
   350   ct-ng STOP=libc_headers    and:    ct-ng RESTART=libc_headers
   351 
   352 To obtain the list of acceptable steps, please call:
   353   ct-ng list-steps
   354 
   355 Note that in order to restart a build, you'll have to say 'Y' to the config
   356 option CT_DEBUG_CT_SAVE_STEPS, and that the previous build effectively went
   357 that far.
   358 
   359 Building all toolchains at once |
   360 --------------------------------+
   361 
   362 You can build all samples; simply call:
   363   ct-ng build-all
   364 
   365 Overriding the number of // jobs |
   366 ---------------------------------+
   367 
   368 If you want to override the number of jobs to run in // (the -j option to
   369 make), you can either re-enter the menuconfig, or simply add it on the command
   370 line, as such:
   371   ct-ng build.4
   372 
   373 which tells crosstool-NG to override the number of // jobs to 4.
   374 
   375 You can see the actions that support overriding the number of // jobs in
   376 the help menu. Those are the ones with [.#] after them (eg. build[.#] or
   377 build-all[.#], and so on...).
   378 
   379 Note on // jobs |
   380 ----------------+
   381 
   382 The crosstool-NG script 'ct-ng' is a Makefile-script. It does *not* execute
   383 in parallel (there is not much to gain). When speaking of // jobs, we are
   384 refering to the number of // jobs when making the *components*. That is, we
   385 speak of the number of // jobs used to build gcc, glibc, and so on...
   386 
   387 Tools wrapper |
   388 --------------+
   389 
   390 Starting with gcc-4.3 come two new dependencies: GMP and MPFR. With gcc-4.4,
   391 come three new ones: GMP, PPL and CLooG/ppl. These are libraries that enable
   392 advanced features to gcc. Additionally, some of the libraries can be used by
   393 binutils and gdb. Unfortunately, not all systems on which crosstool-NG runs
   394 have all of those libraries. And for those that do, the versions of those
   395 libraries may be older than the version required by gcc.
   396 
   397 This is why crosstool-NG builds its own set of libraries as part of the
   398 toolchain.
   399 
   400 The libraries are built as shared libraries, because building them as static
   401 libraries has some short-comings. This poses no problem at build time, as
   402 crosstool-NG correctly points gcc (and binutils and gdb) to the correct
   403 place where our own version of the libraries are installed. But it poses
   404 a problem when gcc et al. are run: the place where the libraries are is most
   405 probably not known to the host dynamic linker. Still worse, if the host system
   406 has its own versions, then ld.so would load the wrong library!
   407 
   408 So we have to force the dynamic linker to load the correct version. We do this
   409 by using the LD_LIBRARY_PATH variable, that informs the dynamic linker where
   410 to look for shared libraries prior to searching its standard places. But we
   411 can't impose that burden on all the system (because it'd be a nightmare to
   412 configure, and because two toolchains on the same system may use different
   413 versions of the libraries); so we have to do it on a per-toolchain basis.
   414 
   415 So we rename all binaries of the toolchain (by adding a dot '.' as their first
   416 character), and add a small program, the so-called "tools wrapper", that
   417 correctly sets LD_LIBRARY_PATH prior to running the real tool.
   418 
   419 First, the wrapper was written as a POSIX-compliant shell script. That shell
   420 script is very simple, if not trivial, and works great. The only drawback is
   421 that it does not work on host systems that lack a shell, for example the
   422 MingW32 environment. To solve the issue, the wrapper has been re-written in C,
   423 and compiled at build time. This C wrapper is much more complex than the shell
   424 script, and although it sems to be working, it's been only lightly tested.
   425 Some of the expected short-comings with this C wrapper are;
   426  - multi-byte file names may not be handled correctly
   427  - it's really big for what it does
   428 
   429 So, the default wrapper installed with your toolchain is the shell script.
   430 If you know that your system is missing a shell, then you shall use the C
   431 wrapper (and report back whether it works, or does not work, for you).
   432 
   433 
   434 _______________________
   435                       /
   436 Using the toolchain  /
   437 ____________________/
   438 
   439 Using the toolchain is as simple as adding the toolchain's bin directory in
   440 your PATH, such as:
   441   export PATH="${PATH}:/your/toolchain/path/bin"
   442 
   443 and then using the target tuple to tell the build systems to use your
   444 toolchain:
   445   ./configure --target=your-target-tuple
   446 or
   447   make CC=your-target-tuple-gcc
   448 or
   449   make CROSS_COMPILE=your-target-tuple-
   450 and so on...
   451 
   452 It is strongly advised not to use the toolchain sys-root directory as an
   453 install directory for your programs/packages. If you do so, you will not be
   454 able to use your toolchain for another project. It is even strongly advised
   455 that your toolchain is chmod-ed to read-only once successfully build, so that
   456 you don't go polluting your toolchain with your programs/packages' files.
   457 
   458 Thus, when you build a program/package, install it in a separate directory,
   459 eg. /your/root. This directory is the /image/ of what would be in the root file
   460 system of your target, and will contain all that your programs/packages have
   461 installed.
   462 
   463 The 'populate' script |
   464 ----------------------+
   465 
   466 When your root directory is ready, it is still missing some important bits: the
   467 toolchain's libraries. To populate your root directory with those libs, just
   468 run:
   469   your-target-tuple-populate -s /your/root -d /your/root-populated
   470 
   471 This will copy /your/root into /your/root-populated, and put the needed and only
   472 the needed libraries there. Thus you don't polute /your/root with any cruft that
   473 would no longer be needed should you have to remove stuff. /your/root always
   474 contains only those things you install in it.
   475 
   476 You can then use /your/root-populated to build up your file system image, a
   477 tarball, or to NFS-mount it from your target, or whatever you need.
   478 
   479 The populate script accepts the following options:
   480 
   481  -s src_dir
   482     Use 'src_dir' as the un-populated root directory.
   483 
   484  -d dst_dir
   485     Put the populated root directory in 'dst_dir'.
   486 
   487  -l lib1 [...]
   488     Always add specified libraries.
   489 
   490  -L file
   491     Always add libraries listed in 'file'.
   492 
   493  -f
   494     Remove 'dst_dir' if it previously existed; continue even if any library
   495     specified with -l or -L is missing.
   496 
   497  -v
   498     Be verbose, and tell what's going on (you can see exactly where libs are
   499     coming from).
   500 
   501  -h
   502     Print the help.
   503 
   504 See 'your-target-tuple-populate -h' for more information on the options.
   505 
   506 Here is how populate works:
   507 
   508   1) performs some sanity checks:
   509      - src_dir and dst_dir are specified
   510      - src_dir exists
   511      - unless forced, dst_dir does not exist
   512      - src_dir != dst_dir
   513 
   514   2) copy src_dir to dst_dir
   515 
   516   3) add forced libraries to dst_dir
   517      - build the list from -l and -L options
   518      - get forced libraries from the sysroot (see below for heuristics)
   519        - abort on the first missing library, unless -f is specified
   520 
   521   4) add all missing libraries to dst_dir
   522      - scan dst_dir for every ELF files that are 'executable' or
   523        'shared object'
   524      - list the "NEEDED Shared library" fields
   525        - check if the library is already in dst_dir/lib or dst_dir/usr/lib
   526        - if not, get the library from the sysroot
   527          - if it's in sysroot/lib, copy it to dst_dir/lib
   528          - if it's in sysroot/usr/lib, copy it to dst_dir/usr/lib
   529          - in both cases, use the SONAME of the library to create the file
   530            in dst_dir
   531          - if it was not found in the sysroot, this is an error.
   532 
   533 ___________________
   534                   /
   535 Toolchain types  /
   536 ________________/
   537 
   538 There are four kinds of toolchains you could encounter.
   539 
   540 First off, you must understand the following: when it comes to compilers there
   541 are up to four machines involved:
   542   1) the machine configuring the toolchain components: the config machine
   543   2) the machine building the toolchain components:    the build machine
   544   3) the machine running the toolchain:                the host machine
   545   4) the machine the toolchain is generating code for: the target machine
   546 
   547 We can most of the time assume that the config machine and the build machine
   548 are the same. Most of the time, this will be true. The only time it isn't
   549 is if you're using distributed compilation (such as distcc). Let's forget
   550 this for the sake of simplicity.
   551 
   552 So we're left with three machines:
   553  - build
   554  - host
   555  - target
   556 
   557 Any toolchain will involve those three machines. You can be as pretty sure of
   558 this as "2 and 2 are 4". Here is how they come into play:
   559 
   560 1) build == host == target
   561     This is a plain native toolchain, targetting the exact same machine as the
   562     one it is built on, and running again on this exact same machine. You have
   563     to build such a toolchain when you want to use an updated component, such
   564     as a newer gcc for example.
   565     crosstool-NG calls it "native".
   566 
   567 2) build == host != target
   568     This is a classic cross-toolchain, which is expected to be run on the same
   569     machine it is compiled on, and generate code to run on a second machine,
   570     the target.
   571     crosstool-NG calls it "cross".
   572 
   573 3) build != host == target
   574     Such a toolchain is also a native toolchain, as it targets the same machine
   575     as it runs on. But it is build on another machine. You want such a
   576     toolchain when porting to a new architecture, or if the build machine is
   577     much faster than the host machine.
   578     crosstool-NG calls it "cross-native".
   579 
   580 4) build != host != target
   581     This one is called a canadian-toolchain (*), and is tricky. The three
   582     machines in play are different. You might want such a toolchain if you
   583     have a fast build machine, but the users will use it on another machine,
   584     and will produce code to run on a third machine.
   585     crosstool-NG calls it "canadian".
   586 
   587 crosstool-NG can build all these kinds of toolchains (or is aiming at it,
   588 anyway!)
   589 
   590 (*) The term Canadian Cross came about because at the time that these issues
   591     were all being hashed out, Canada had three national political parties.
   592     http://en.wikipedia.org/wiki/Cross_compiler
   593 
   594 Seemingly-native toolchains |
   595 ----------------------------+
   596 
   597 Seemingly-native toolchains are toolchains that target the same architecture
   598 as the one it is built on, and on which it will run, but the machine tuple
   599 may be different (eg i686 vs. i386, or x86_64-unknown-linux-gnu vs.
   600 x86_64-pc-linux-gnu). This also applies if the target architecture is of the
   601 same kind (eg. x86 vs. x86_64, or ppc vs. ppc64).
   602 
   603 Such toolchain is tricky to build, as the configure scripts may incorrectly
   604 assume that files (headers and libs) from the build (or host) machine can be
   605 used by the cross-compiler it is going to build. The problem seems to arise
   606 only with glibc (and eglibc?) starting with version 2.7.
   607 
   608 
   609 _____________
   610             /
   611 Internals  /
   612 __________/
   613 
   614 Internally, crosstool-NG is script-based. To ease usage, the frontend is
   615 Makefile-based.
   616 
   617 Makefile front-end |
   618 -------------------+
   619 
   620 The entry point to crosstool-NG is the Makefile script "ct-ng". Calling this
   621 script with an action will act exactly as if the Makefile was in the current
   622 working directory and make was called with the action as rule. Thus:
   623   ct-ng menuconfig
   624 
   625 is equivalent to having the Makefile in CWD, and calling:
   626   make menuconfig
   627 
   628 Having ct-ng as it is avoids copying the Makefile everywhere, and acts as a
   629 traditional command.
   630 
   631 ct-ng loads sub- Makefiles from the library directory $(CT_LIB_DIR), as set up
   632 at configuration time with ./configure.
   633 
   634 ct-ng also searches for config files, sub-tools, samples, scripts and patches in
   635 that library directory.
   636 
   637 Because of a stupid make behavior/bug I was unable to track down, implicit make
   638 rules are disabled: installing with --local would triger those rules, and mconf
   639 was unbuildable.
   640 
   641 Kconfig parser |
   642 ---------------+
   643 
   644 The kconfig language is a hacked version, vampirised from the Linux kernel
   645 (http://www.kernel.org/), and (heavily) adapted to my needs.
   646 
   647 The list of the most notable changes (at least the ones I remember) follows:
   648 - the CONFIG_ prefix has been replaced with CT_
   649 - a leading | in prompts is skipped, and subsequent leading spaces are not
   650   trimmed
   651 - otherwise leading spaces are silently trimmed
   652 
   653 The kconfig parsers (conf and mconf) are not installed pre-built, but as
   654 source files. Thus you can have the directory where crosstool-NG is installed,
   655 exported (via NFS or whatever) and have clients with different architectures
   656 use the same crosstool-NG installation, and most notably, the same set of
   657 patches.
   658 
   659 Architecture-specific |
   660 ----------------------+
   661 
   662 Note: this chapter is not really well written, and might thus be a little bit
   663 complex to understand. To get a better grasp of what an architecture is, the
   664 reader is kindly encouraged to look at the "arch/" sub-directory, and to the
   665 existing architectures to see how things are laid out.
   666 
   667 An architecture is defined by:
   668 
   669  - a human-readable name, in lower case letters, with numbers as appropriate.
   670    The underscore is allowed; space and special characters are not.
   671      Eg.: arm, x86_64
   672  - a file in "config/arch/", named after the architecture's name, and suffixed
   673    with ".in".
   674      Eg.: config/arch/arm.in
   675  - a file in "scripts/build/arch/", named after the architecture's name, and
   676    suffixed with ".sh".
   677      Eg.: scripts/build/arch/arm.sh
   678 
   679 The architecture's ".in" file API:
   680  > the config option "ARCH_%arch%" (where %arch% is to be replaced with the
   681    actual architecture name).
   682    That config option must have *neither* a type, *nor* a prompt! Also, it can
   683    *not* depend on any other config option (EXPERIMENTAL is managed as above).
   684      Eg.:
   685        config ARCH_arm
   686    + mandatory:
   687        defines a (terse) help entry for this architecture:
   688        Eg.:
   689          config ARCH_arm
   690            help
   691              The ARM architecture.
   692    + optional:
   693        selects adequate associated config options.
   694        Note: 64-bit architectures *shall* select ARCH_64
   695        Eg.:
   696          config ARCH_arm
   697            select ARCH_SUPPORTS_BOTH_ENDIAN
   698            select ARCH_DEFAULT_LE
   699            help
   700              The ARM architecture.
   701        Eg.:
   702          config ARCH_x86_64
   703             select ARCH_64
   704             help
   705               The x86_64 architecture.
   706 
   707  > other target-specific options, at your discretion. Note however that to
   708    avoid name-clashing, such options shall be prefixed with "ARCH_%arch%",
   709    where %arch% is again replaced by the actual architecture name.
   710    (Note: due to historical reasons, and lack of time to clean up the code,
   711     I may have left some config options that do not completely conform to
   712     this, as the architecture name was written all upper case. However, the
   713     prefix is unique among architectures, and does not cause harm).
   714 
   715 The architecture's ".sh" file API:
   716  > the function "CT_DoArchTupleValues"
   717    + parameters: none
   718    + environment:
   719      - all variables from the ".config" file,
   720      - the two variables "target_endian_eb" and "target_endian_el" which are
   721        the endianness suffixes
   722    + return value: 0 upon success, !0 upon failure
   723    + provides:
   724      - mandatory
   725      - the environment variable CT_TARGET_ARCH
   726      - contains:
   727        the architecture part of the target tuple.
   728        Eg.: "armeb" for big endian ARM
   729             "i386" for an i386
   730    + provides:
   731      - optional
   732      - the environment variable CT_TARGET_SYS
   733      - contains:
   734        the sytem part of the target tuple.
   735        Eg.: "gnu" for glibc on most architectures
   736             "gnueabi" for glibc on an ARM EABI
   737      - defaults to:
   738        - for glibc-based toolchain: "gnu"
   739        - for uClibc-based toolchain: "uclibc"
   740    + provides:
   741      - optional
   742      - the environment variable CT_KERNEL_ARCH
   743      - contains:
   744        the architecture name as understandable by the Linux kernel build
   745        system.
   746        Eg.: "arm" for an ARM
   747             "powerpc" for a PowerPC
   748             "i386" for an x86
   749      - defaults to:
   750        ${CT_ARCH}
   751    + provides:
   752      - optional
   753      - the environment variables to configure the cross-gcc (defaults)
   754        - CT_ARCH_WITH_ARCH    : the gcc ./configure switch to select architecture level         ( "--with-arch=${CT_ARCH_ARCH}"   )
   755        - CT_ARCH_WITH_ABI     : the gcc ./configure switch to select ABI level                  ( "--with-abi=${CT_ARCH_ABI}"     )
   756        - CT_ARCH_WITH_CPU     : the gcc ./configure switch to select CPU instruction set        ( "--with-cpu=${CT_ARCH_CPU}"     )
   757        - CT_ARCH_WITH_TUNE    : the gcc ./configure switch to select scheduling                 ( "--with-tune=${CT_ARCH_TUNE}"   )
   758        - CT_ARCH_WITH_FPU     : the gcc ./configure switch to select FPU type                   ( "--with-fpu=${CT_ARCH_FPU}"     )
   759        - CT_ARCH_WITH_FLOAT   : the gcc ./configure switch to select floating point arithmetics ( "--with-float=soft" or /empty/  )
   760    + provides:
   761      - optional
   762      - the environment variables to pass to the cross-gcc to build target binaries (defaults)
   763        - CT_ARCH_ARCH_CFLAG   : the gcc switch to select architecture level                     ( "-march=${CT_ARCH_ARCH}"            )
   764        - CT_ARCH_ABI_CFLAG    : the gcc switch to select ABI level                              ( "-mabi=${CT_ARCH_ABI}"              )
   765        - CT_ARCH_CPU_CFLAG    : the gcc switch to select CPU instruction set                    ( "-mcpu=${CT_ARCH_CPU}"              )
   766        - CT_ARCH_TUNE_CFLAG   : the gcc switch to select scheduling                             ( "-mtune=${CT_ARCH_TUNE}"            )
   767        - CT_ARCH_FPU_CFLAG    : the gcc switch to select FPU type                               ( "-mfpu=${CT_ARCH_FPU}"              )
   768        - CT_ARCH_FLOAT_CFLAG  : the gcc switch to choose floating point arithmetics             ( "-msoft-float" or /empty/           )
   769        - CT_ARCH_ENDIAN_CFLAG : the gcc switch to choose big or little endian                   ( "-mbig-endian" or "-mlittle-endian" )
   770      - default to:
   771        see above.
   772    + provides:
   773      - optional
   774      - the environement variables to configure the core and final compiler, specific to this architecture:
   775        - CT_ARCH_CC_CORE_EXTRA_CONFIG   : additional, architecture specific core gcc ./configure flags
   776        - CT_ARCH_CC_EXTRA_CONFIG        : additional, architecture specific final gcc ./configure flags
   777      - default to:
   778        - all empty
   779    + provides:
   780      - optional
   781      - the architecture-specific CFLAGS and LDFLAGS:
   782        - CT_ARCH_TARGET_CLFAGS
   783        - CT_ARCH_TARGET_LDFLAGS
   784      - default to:
   785        - all empty
   786 
   787 You can have a look at "config/arch/arm.in" and "scripts/build/arch/arm.sh" for
   788 a quite complete example of what an actual architecture description looks like.
   789 
   790 Kernel specific |
   791 ----------------+
   792 
   793 A kernel is defined by:
   794 
   795  - a human-readable name, in lower case letters, with numbers as appropriate.
   796    The underscore is allowed; space and special characters are not (although
   797    they are internally replaced with underscores.
   798      Eg.: linux, bare-metal
   799  - a file in "config/kernel/", named after the kernel name, and suffixed with
   800    ".in".
   801      Eg.: config/kernel/linux.in, config/kernel/bare-metal.in
   802  - a file in "scripts/build/kernel/", named after the kernel name, and suffixed
   803    with ".sh".
   804      Eg.: scripts/build/kernel/linux.sh, scripts/build/kernel/bare-metal.sh
   805 
   806 The kernel's ".in" file must contain:
   807  > an optional lines containing exactly "# EXPERIMENTAL", starting on the
   808    first column, and without any following space or other character.
   809    If this line is present, then this kernel is considered EXPERIMENTAL,
   810    and correct dependency on EXPERIMENTAL will be set.
   811 
   812  > the config option "KERNEL_%kernel_name%" (where %kernel_name% is to be
   813    replaced with the actual kernel name, with all special characters and
   814    spaces replaced by underscores).
   815    That config option must have *neither* a type, *nor* a prompt! Also, it can
   816    *not* depends on EXPERIMENTAL.
   817      Eg.: KERNEL_linux, KERNEL_bare_metal
   818    + mandatory:
   819        defines a (terse) help entry for this kernel.
   820        Eg.:
   821          config KERNEL_bare_metal
   822            help
   823              Build a compiler for use without any kernel.
   824    + optional:
   825        selects adequate associated config options.
   826        Eg.:
   827          config KERNEL_bare_metal
   828            select BARE_METAL
   829            help
   830              Build a compiler for use without any kernel.
   831 
   832  > other kernel specific options, at your discretion. Note however that, to
   833    avoid name-clashing, such options should be prefixed with
   834    "KERNEL_%kernel_name%", where %kernel_name% is again tp be replaced with
   835    the actual kernel name.
   836    (Note: due to historical reasons, and lack of time to clean up the code,
   837     I may have left some config options that do not completely conform to
   838     this, as the kernel name was written all upper case. However, the prefix
   839     is unique among kernels, and does not cause harm).
   840 
   841 The kernel's ".sh" file API:
   842  > is a bash script fragment
   843 
   844  > defines the function CT_DoKernelTupleValues
   845    + see the architecture's CT_DoArchTupleValues, except for:
   846    + set the environment variable CT_TARGET_KERNEL, the kernel part of the
   847      target tuple
   848    + return value: ignored
   849 
   850  > defines the function "do_kernel_get":
   851    + parameters: none
   852    + environment:
   853       - all variables from the ".config" file.
   854    + return value: 0 for success, !0 for failure.
   855    + behavior: download the kernel's sources, and store the tarball into
   856      "${CT_TARBALLS_DIR}". To this end, a functions is available, that
   857      abstracts downloading tarballs:
   858      - CT_DoGet <tarball_base_name> <URL1 [URL...]>
   859        Eg.: CT_DoGet linux-2.6.26.5 ftp://ftp.kernel.org/pub/linux/kernel/v2.6
   860      Note: retrieving sources from svn, cvs, git and the likes is not supported
   861      by CT_DoGet. You'll have to do this by hand, as it is done for eglibc in
   862      "scripts/build/libc/eglibc.sh"
   863 
   864  > defines the function "do_kernel_extract":
   865    + parameters: none
   866    + environment:
   867       - all variables from the ".config" file,
   868    + return value: 0 for success, !0 for failure.
   869    + behavior: extract the kernel's tarball into "${CT_SRC_DIR}", and apply
   870      required patches. To this end, a function is available, that abstracts
   871      extracting tarballs:
   872      - CT_ExtractAndPatch <tarball_base_name>
   873        Eg.: CT_ExtractAndPatch linux-2.6.26.5
   874 
   875  > defines the function "do_kernel_headers":
   876    + parameters: none
   877    + environment:
   878       - all variables from the ".config" file,
   879    + return value: 0 for success, !0 for failure.
   880    + behavior: install the kernel headers (if any) in "${CT_SYSROOT_DIR}/usr/include"
   881 
   882  > defines any kernel-specific helper functions
   883    These functions, if any, must be prefixed with "do_kernel_%CT_KERNEL%_",
   884    where '%CT_KERNEL%' is to be replaced with the actual kernel name, to avoid
   885    any name-clashing.
   886 
   887 You can have a look at "config/kernel/linux.in" and "scripts/build/kernel/linux.sh"
   888 as an example of what a complex kernel description looks like.
   889 
   890 Adding a new version of a component |
   891 ------------------------------------+
   892 
   893 When a new component, such as the Linux kernel, gcc or any other is released,
   894 adding the new version to crosstool-NG is quite easy. There is a script that
   895 will do all that for you:
   896   scripts/addToolVersion.sh
   897 
   898 Run it with no option to get some help.
   899 
   900 Build scripts |
   901 --------------+
   902 
   903 To Be Written later...