docs/overview.txt
author "Yann E. MORIN" <yann.morin.1998@anciens.enib.fr>
Fri Jul 13 14:25:06 2007 +0000 (2007-07-13)
changeset 246 372bc84bea36
parent 227 15f44768d3f7
child 276 835178a32fc1
permissions -rw-r--r--
Document CT_OBSOLETE, CT_EXPERIMENTAL and CT_BROKEN.
yann@1
     1
File.........: overview.txt
yann@197
     2
Content......: Overview of how crosstool-NG works.
yann@92
     3
Copyrigth....: (C) 2007 Yann E. MORIN <yann.morin.1998@anciens.enib.fr>
yann@192
     4
License......: Creative Commons Attribution Share Alike (CC-by-sa), v2.5
yann@92
     5
yann@1
     6
________________
yann@1
     7
               /
yann@1
     8
Introduction  /
yann@1
     9
_____________/
yann@1
    10
yann@1
    11
crosstool-NG aims at building toolchains. Toolchains are an essential component
yann@1
    12
in a software development project. It will compile, assemble and link the code
yann@1
    13
that is being developped. Some pieces of the toolchain will eventually end up
yann@1
    14
in the resulting binary/ies: static libraries are but an example.
yann@1
    15
yann@1
    16
So, a toolchain is a very sensitive piece of software, as any bug in one of the
yann@1
    17
components, or a poorly configured component, can lead to execution problems,
yann@1
    18
ranging from poor performance, to applications ending unexpectedly, to
yann@1
    19
mis-behaving software (which more than often is hard to detect), to hardware
yann@1
    20
damage, or even to human risks (which is more than regretable).
yann@1
    21
yann@1
    22
Toolchains are made of different piece of software, each being quite complex
yann@1
    23
and requiring specially crafted options to build and work seamlessly. This
yann@1
    24
is usually not that easy, even in the not-so-trivial case of native toolchains.
yann@1
    25
The work reaches a higher degree of complexity when it comes to cross-
yann@40
    26
compilation, where it can become quite a nightmare...
yann@1
    27
yann@40
    28
Some cross-toolchains exist on the internet, and can be used for general
yann@1
    29
development, but they have a number of limitations:
yann@1
    30
  - they can be general purpose, in that they are configured for the majority:
yann@1
    31
    no optimisation for your specific target,
yann@1
    32
  - they can be prepared for a specific target and thus are not easy to use,
yann@1
    33
    nor optimised for, or even supporting your target,
yann@1
    34
  - they often are using ageing components (compiler, C library, etc...) not
yann@1
    35
    supporting special features of your shiny new processor;
yann@1
    36
On the other side, these toolchain offer some advantages:
yann@1
    37
  - they are ready to use and quite easy to install and setup,
yann@1
    38
  - they are proven if used by a wide community.
yann@1
    39
yann@1
    40
But once you want to get all the juice out of your specific hardware, you will
yann@197
    41
want to build your own toolchain. This is where crosstool-NG comes into play.
yann@1
    42
yann@1
    43
There are also a number of tools that builds toolchains for specific needs,
yann@1
    44
which is not really scalable. Examples are:
yann@1
    45
  - buildroot (buildroot.uclibc.org) whose main puprpose is to build root file
yann@1
    46
    systems, hence the name. But once you have your toolchain with buildroot,
yann@1
    47
    part of it is installed in the root-to-be, so if you want to build a whole
yann@1
    48
    new root, you either have to save the existing one as a template and
yann@1
    49
    restore it later, or restart again from scratch. This is not convenient,
yann@1
    50
  - ptxdist (www.pengutronix.de/software/ptxdist), whose purpose is very
yann@1
    51
    similar to buildroot,
yann@1
    52
  - other projects (openembeded.org for example), which is again used to
yann@1
    53
    build root file systems.
yann@1
    54
yann@1
    55
crosstool-NG is really targetted at building toolchains, and only toolchains.
yann@1
    56
It is then up to you to use it the way you want.
yann@1
    57
yann@1
    58
___________
yann@1
    59
          /
yann@1
    60
History  /
yann@1
    61
________/
yann@1
    62
yann@1
    63
crosstool was first 'conceived' by Dan Kegel, which offered it to the community,
yann@1
    64
as a set of scripts, a repository of patches, and some pre-configured, general
yann@1
    65
purpose setup files to be used to configure crosstool. This is available at
yann@203
    66
http://www.kegel.com/crosstool, and the subversion repository is hosted on
yann@203
    67
google at http://code.google.com/p/crosstool/.
yann@1
    68
yann@1
    69
At the time of writing, crosstool only supports building with one C library,
yann@1
    70
namely glibc, and one C compiler, gcc; it is cripled with historical support
yann@92
    71
for legacy components, and is some kind of a mess to upgrade. Also, submited
yann@203
    72
patches take a loooong time before they are integrated mainline.
yann@1
    73
yann@1
    74
I once managed to add support for uClibc-based toolchains, but it did not make
yann@1
    75
into mainline, mostly because I don't have time to port the patch forward to
yann@1
    76
the new versions, due in part to the big effort it was taking.
yann@1
    77
yann@1
    78
So I decided to clean up crosstool in the state it was, re-order the things
yann@1
    79
in place, and add appropriate support for what I needed, that is uClibc
yann@203
    80
support. That was a disaster, as inclusion into mainline is slow as hell,
yann@203
    81
and the changes were so numerous.
yann@1
    82
yann@1
    83
The only option left to me was rewrite crosstool from scratch. I decided to go
yann@197
    84
this way, and name the new implementation crosstool-NG, standing for crosstool
yann@197
    85
Next Generation, as many other comunity projects do, and as a wink at the TV
yann@197
    86
series "Star Trek: The Next Generation". ;-)
yann@1
    87
yann@168
    88
____________________________
yann@168
    89
                           /
yann@168
    90
Configuring crosstool-NG  /
yann@168
    91
_________________________/
yann@168
    92
yann@168
    93
crosstool-NG is configured the same way you configure your Linux kernel: by
yann@168
    94
using a curses-based menu. It is assumed you now how to handle this.
yann@168
    95
yann@168
    96
To enter the menu, type:
yann@192
    97
  ct-ng menuconfig
yann@168
    98
yann@203
    99
Almost every config item has a help entry. Read them carefully.
yann@168
   100
yann@168
   101
String and number options can refer to environment variables. In such a case,
yann@192
   102
you must use the shell syntax: ${VAR}. You shall neither single- nor double-
yann@192
   103
quote the string options.
yann@168
   104
yann@192
   105
There are three environment variables that are computed by crosstool-NG, and
yann@168
   106
that you can use:
yann@168
   107
yann@168
   108
CT_TARGET:
yann@168
   109
  It represents the target triplet you are building for. You can use it for
yann@168
   110
  example in the installation/prefix directory, such as:
yann@168
   111
    /opt/x-tools/${CT_TARGET}
yann@168
   112
yann@168
   113
CT_TOP_DIR:
yann@182
   114
  The top directory where crosstool-NG is running. You shouldn't need it in
yann@182
   115
  most cases. There is one case where you may need it: if you have local
yann@182
   116
  patches and you store them in your running directory, you can refer to them
yann@168
   117
  by using CT_TOP_DIR, such as:
yann@168
   118
    ${CT_TOP_DIR}/patches.myproject
yann@168
   119
yann@168
   120
CT_VERSION:
yann@192
   121
  The version of crosstool-NG you are using. Not much use for you, but it's
yann@168
   122
  there if you need it.
yann@168
   123
yann@168
   124
yann@168
   125
Interesting config options |
yann@168
   126
---------------------------*
yann@168
   127
yann@168
   128
CT_LOCAL_TARBALLS_DIR:
yann@168
   129
  If you already have sone tarballs in a direcotry, enter it here. That will
yann@197
   130
  speed up the retrieving phase, where crosstool-NG would otherwise download
yann@168
   131
  those tarballs.
yann@168
   132
yann@168
   133
CT_PREFIX_DIR:
yann@168
   134
  This is where the toolchain will be installed in (and for now, where it
yann@168
   135
  will run from).
yann@168
   136
yann@168
   137
CT_TARGET_VENDOR:
yann@168
   138
  An identifier for your toolchain, will take place in the vendor part of the
yann@168
   139
  target triplet. It shall *not* contain spaces or dashes. Usually, keep it
yann@168
   140
  to a one-word string, or use underscores to separate words if you need.
yann@168
   141
  Avoid dots, commas, and special characters.
yann@168
   142
yann@168
   143
CT_TARGET_ALIAS:
yann@168
   144
  An alias for the toolchian. It will be used as a prefix to the toolchain
yann@168
   145
  tools. For example, you will have ${CT_TARGET_ALIAS}-gcc
yann@168
   146
yann@246
   147
Also, if you think you don't see enough versions, you can try to enable one of
yann@246
   148
those:
yann@246
   149
yann@246
   150
CT_OBSOLETE:
yann@246
   151
  Show obsolete versions or tools. Most of the time, you don't want to base
yann@246
   152
  your toolchain on too old a version (of gcc, for example). But at times, it
yann@246
   153
  can come handy to use such an old version for regression tests. Those old
yann@246
   154
  versions are hidden behind CT_BSOLETE.
yann@246
   155
yann@246
   156
CT_EXPERIMENTAL:
yann@246
   157
  Show experimental versions or tools. Again, you might not want to base your
yann@246
   158
  toolchain on too recent tools (eg. gcc) for production. But if you need a
yann@246
   159
  feature present only in a recent version, or a new tool, you can find them
yann@246
   160
  hidden behind CT_EXPERIMENTAL.
yann@246
   161
yann@246
   162
CT_BROKEN:
yann@246
   163
  Show broken versions or tools. Some usefull tools are currently broken: they
yann@246
   164
  won't compile, run, or worse, cause defects when running. But if you are
yann@246
   165
  brave enough, you can try and debug them. They are hidden behind CT_BROKEN,
yann@246
   166
  which itself is hiddent behind EXPERIMENTAL.
yann@246
   167
yann@168
   168
________________________
yann@168
   169
                       /
yann@168
   170
Running crosstool-NG  /
yann@168
   171
_____________________/
yann@1
   172
yann@197
   173
crosstool-NG is configured by a configurator presenting a menu-stuctured set of
yann@1
   174
options. These options let you specify the way you want your toolchain built,
yann@1
   175
where you want it installed, what architecture and specific processor it
yann@1
   176
will support, the version of the components you want to use, etc... The
yann@1
   177
value for those options are then stored in a configuration file.
yann@1
   178
yann@168
   179
To build the toolchain, simply type:
yann@203
   180
  ct-ng build
yann@135
   181
yann@135
   182
This will use the above configuration to retrieve, extract and patch the
yann@135
   183
components, build, install and eventually test your newly built toolchain.
yann@1
   184
yann@1
   185
You are then free to add the toolchain /bin directory in your PATH to use
yann@1
   186
it at will.
yann@1
   187
yann@135
   188
In any case, you can get some terse help. Just type:
yann@192
   189
  ct-ng help
yann@203
   190
or:
yann@203
   191
  man 1 ct-ng
yann@135
   192
yann@135
   193
yann@135
   194
Stoping and restarting a build |
yann@135
   195
-------------------------------*
yann@135
   196
yann@135
   197
If you want to stop the build after a step you are debugging, you can pass the
yann@135
   198
variable STOP to make:
yann@192
   199
  ct-ng STOP=some_step
yann@135
   200
yann@135
   201
Conversely, if you want to restart a build at a specific step you are
yann@135
   202
debugging, you can pass the RESTART variable to make:
yann@192
   203
  ct-ng RESTART=some_step
yann@135
   204
yann@136
   205
Alternatively, you can call make with the name of a step to just do that step:
yann@192
   206
  ct-ng libc_headers
yann@136
   207
is equivalent to:
yann@192
   208
  ct-ng RESTART=libs_headers STOP=libc_headers
yann@136
   209
yann@136
   210
The shortcuts -step_name and step_name- allow to respectively stop or restart
yann@136
   211
at that step. Thus:
yann@192
   212
  ct-ng -libc_headers        and:    ct-ng libc_headers-
yann@136
   213
are equivalent to:
yann@192
   214
  ct-ng STOP=libc_headers    and:    ct-ng RESTART=libc_headers
yann@136
   215
yann@181
   216
To obtain the list of acceptable steps, please call:
yann@192
   217
  ct-ng liststeps
yann@181
   218
yann@168
   219
Note that in order to restart a build, you'll have to say 'Y' to the config
yann@168
   220
option CT_DEBUG_CT_SAVE_STEPS, and that the previous build effectively went
yann@168
   221
that far.
yann@92
   222
yann@92
   223
yann@168
   224
Testing all toolchains at once |
yann@168
   225
-------------------------------*
yann@92
   226
yann@168
   227
You can test-build all samples; simply call:
yann@192
   228
  ct-ng regtest
yann@40
   229
yann@227
   230
_______________________
yann@227
   231
                      /
yann@227
   232
Using the toolchain  /
yann@227
   233
____________________/
yann@227
   234
yann@227
   235
Using the toolchain is as simple as adding the toolchain's bin directory in
yann@227
   236
your PATH, such as:
yann@227
   237
  export PATH="${PATH}:/your/toolchain/path/bin"
yann@227
   238
yann@227
   239
and then using the target triplet to tell the build systems to use your
yann@227
   240
toolchain:
yann@227
   241
  ./configure --target=your-target-triplet
yann@227
   242
  make CC=your-target-triplet-gcc
yann@227
   243
  make CROSS_COMPILE=your-target-triplet-
yann@227
   244
  and so on...
yann@227
   245
yann@227
   246
When your root directory is ready, it is still missing some important bits: the
yann@227
   247
toolchain's libraries. To populate your root directory with those libs, just
yann@227
   248
run:
yann@227
   249
  your-target-triplet-populate -s /your/root -d /your/root-populated
yann@227
   250
yann@227
   251
This will copy /your/root into /your/root-populated, and put the needed and only
yann@227
   252
the needed libraries there. Thus you don't polute /your/root with any cruft that
yann@227
   253
would no longer be needed should you have to remove stuff. /your/root always
yann@227
   254
contains only those things you install in it.
yann@227
   255
yann@227
   256
You can then use /your/root-populated to build up your file system image, a
yann@227
   257
tarball, or to NFS-mount it from your target, or whatever you need.
yann@227
   258
yann@40
   259
___________________
yann@40
   260
                  /
yann@40
   261
Toolchain types  /
yann@40
   262
________________/
yann@40
   263
yann@40
   264
There are four kinds of toolchains you could encounter.
yann@40
   265
yann@40
   266
First off, you must understand the following: when it comes to compilers there
yann@40
   267
are up to four machines involved:
yann@40
   268
  1) the machine configuring the toolchain components: the config machine
yann@40
   269
  2) the machine building the toolchain components:    the build machine
yann@40
   270
  3) the machine running the toolchain:                the host machine
yann@203
   271
  4) the machine the toolchain is generating code for: the target machine
yann@40
   272
yann@40
   273
We can most of the time assume that the config machine and the build machine
yann@40
   274
are the same. Most of the time, this will be true. The only time it isn't
yann@40
   275
is if you're using distributed compilation (such as distcc). Let's forget
yann@40
   276
this for the sake of simplicity.
yann@40
   277
yann@40
   278
So we're left with three machines:
yann@40
   279
 - build
yann@40
   280
 - host
yann@40
   281
 - target
yann@40
   282
yann@40
   283
Any toolchain will involve those three machines. You can be as pretty sure of
yann@40
   284
this as "2 and 2 are 4". Here is how they come into play:
yann@40
   285
yann@40
   286
1) build == host == target
yann@40
   287
    This is a plain native toolchain, targetting the exact same machine as the
yann@40
   288
    one it is built on, and running again on this exact same machine. You have
yann@40
   289
    to build such a toolchain when you want to use an updated component, such
yann@40
   290
    as a newer gcc for example.
yann@197
   291
    crosstool-NG calls it "native".
yann@40
   292
yann@40
   293
2) build == host != target
yann@40
   294
    This is a classic cross-toolchain, which is expected to be run on the same
yann@40
   295
    machine it is compiled on, and generate code to run on a second machine,
yann@40
   296
    the target.
yann@197
   297
    crosstool-NG calls it "cross".
yann@40
   298
yann@40
   299
3) build != host == target
yann@40
   300
    Such a toolchain is also a native toolchain, as it targets the same machine
yann@40
   301
    as it runs on. But it is build on another machine. You want such a
yann@40
   302
    toolchain when porting to a new architecture, or if the build machine is
yann@40
   303
    much faster than the host machine.
yann@197
   304
    crosstool-NG calls it "cross-native".
yann@40
   305
yann@40
   306
4) build != host != target
yann@92
   307
    This one is called a canadian-toolchain (*), and is tricky. The three
yann@40
   308
    machines in play are different. You might want such a toolchain if you
yann@40
   309
    have a fast build machine, but the users will use it on another machine,
yann@40
   310
    and will produce code to run on a third machine.
yann@197
   311
    crosstool-NG calls it "canadian".
yann@40
   312
yann@197
   313
crosstool-NG can build all these kinds of toolchains (or is aiming at it,
yann@197
   314
anyway!)
yann@40
   315
yann@40
   316
(*) The term Canadian Cross came about because at the time that these issues
yann@40
   317
    were all being hashed out, Canada had three national political parties.
yann@40
   318
    http://en.wikipedia.org/wiki/Cross_compiler
yann@40
   319
yann@1
   320
_____________
yann@1
   321
            /
yann@1
   322
Internals  /
yann@1
   323
__________/
yann@1
   324
yann@92
   325
Internally, crosstool-NG is script-based. To ease usage, the frontend is
yann@92
   326
Makefile-based.
yann@92
   327
yann@92
   328
Makefile front-end |
yann@92
   329
-------------------*
yann@92
   330
yann@203
   331
The entry point to crosstool-NG is the Makefile script "ct-ng". Calling this
yann@203
   332
script with an action will act exactly as if the Makefile was in the current
yann@203
   333
working directory and make was called with the action as rule. Thus:
yann@203
   334
  ct-ng menuconfig
yann@203
   335
is equivalent to having the Makefile in CWD, and calling:
yann@203
   336
  make menuconfig
yann@203
   337
yann@203
   338
Having ct-ng as it is avoids copying the Makefile everywhere, and acts as a
yann@203
   339
traditional command.
yann@203
   340
yann@203
   341
ct-ng loads sub- Makefiles from the library directory $(CT_LIB_DIR), as set up
yann@203
   342
at configuration time with ./configure.
yann@203
   343
yann@203
   344
ct-ng also search for config files, sub-tools, samples, scripts and patches in
yann@203
   345
that library directory.
yann@92
   346
yann@182
   347
Kconfig parser |
yann@182
   348
---------------*
yann@92
   349
yann@92
   350
The kconfig language is a hacked version, vampirised from the toybox project
yann@182
   351
by Rob LANDLEY (http://www.landley.net/code/toybox/), itself coming from the
yann@182
   352
Linux kernel (http://www.linux.org/ http://www.kernel.org/), and (heavily)
yann@182
   353
adapted to my needs.
yann@92
   354
yann@203
   355
The kconfig parsers (conf and mconf) are not installed pre-built, but as
yann@203
   356
source files. Thus you can have the directory where crosstool-NG is installed,
yann@203
   357
exported (via NFS or whatever) and have clients with different architectures
yann@203
   358
use the same crosstool-NG installation, and most notably, the same set of
yann@203
   359
patches.
yann@203
   360
yann@203
   361
Build scripts |
yann@203
   362
--------------*
yann@203
   363
yann@203
   364
To Be Written later...