docs/overview.txt
author Yann E. MORIN
Sun Jun 14 21:43:55 2009 +0200 (2009-06-14)
changeset 1405 8b86a6f004ae
parent 1300 7acdd1de5b50
child 1406 515580cad631
permissions -rw-r--r--
[populate] fix documentation

This patch fixes the existing documentation:
- document -l and -L
- use syntax more in line with the help entry
yann@1
     1
File.........: overview.txt
yann@197
     2
Content......: Overview of how crosstool-NG works.
yann@92
     3
Copyrigth....: (C) 2007 Yann E. MORIN <yann.morin.1998@anciens.enib.fr>
yann@192
     4
License......: Creative Commons Attribution Share Alike (CC-by-sa), v2.5
yann@92
     5
yann@628
     6
____________________
yann@628
     7
                   /
yann@628
     8
Table Of Content  /
yann@628
     9
_________________/
yann@628
    10
yann@628
    11
yann@628
    12
Introduction
yann@628
    13
History
yann@628
    14
Installing crosstool-NG
yann@628
    15
  Install method
yann@628
    16
  The hacker's way
yann@1048
    17
  Preparing for packaging
yann@837
    18
  Shell completion
yann@628
    19
  Contributed code
yann@628
    20
Configuring crosstool-NG
yann@628
    21
  Interesting config options
yann@628
    22
  Re-building an existing toolchain
yann@628
    23
Running crosstool-NG
yann@628
    24
  Stopping and restarting a build
yann@628
    25
  Testing all toolchains at once
yann@628
    26
  Overriding the number of // jobs
yann@628
    27
Using the toolchain
yann@628
    28
Toolchain types
yann@628
    29
Internals
yann@628
    30
  Makefile front-end
yann@628
    31
  Kconfig parser
yann@628
    32
  Architecture-specific
yann@628
    33
  Adding a new version of a component
yann@628
    34
  Build scripts
yann@628
    35
yann@1
    36
________________
yann@1
    37
               /
yann@1
    38
Introduction  /
yann@1
    39
_____________/
yann@1
    40
yann@1
    41
crosstool-NG aims at building toolchains. Toolchains are an essential component
yann@1
    42
in a software development project. It will compile, assemble and link the code
rpjday@436
    43
that is being developed. Some pieces of the toolchain will eventually end up
yann@1
    44
in the resulting binary/ies: static libraries are but an example.
yann@1
    45
yann@1
    46
So, a toolchain is a very sensitive piece of software, as any bug in one of the
yann@1
    47
components, or a poorly configured component, can lead to execution problems,
yann@1
    48
ranging from poor performance, to applications ending unexpectedly, to
yann@1
    49
mis-behaving software (which more than often is hard to detect), to hardware
rpjday@436
    50
damage, or even to human risks (which is more than regrettable).
yann@1
    51
yann@1
    52
Toolchains are made of different piece of software, each being quite complex
yann@1
    53
and requiring specially crafted options to build and work seamlessly. This
yann@1
    54
is usually not that easy, even in the not-so-trivial case of native toolchains.
yann@1
    55
The work reaches a higher degree of complexity when it comes to cross-
yann@40
    56
compilation, where it can become quite a nightmare...
yann@1
    57
yann@40
    58
Some cross-toolchains exist on the internet, and can be used for general
yann@1
    59
development, but they have a number of limitations:
yann@1
    60
  - they can be general purpose, in that they are configured for the majority:
yann@1
    61
    no optimisation for your specific target,
yann@1
    62
  - they can be prepared for a specific target and thus are not easy to use,
yann@1
    63
    nor optimised for, or even supporting your target,
rpjday@436
    64
  - they often are using aging components (compiler, C library, etc...) not
yann@1
    65
    supporting special features of your shiny new processor;
yann@1
    66
On the other side, these toolchain offer some advantages:
yann@1
    67
  - they are ready to use and quite easy to install and setup,
yann@1
    68
  - they are proven if used by a wide community.
yann@1
    69
yann@1
    70
But once you want to get all the juice out of your specific hardware, you will
yann@197
    71
want to build your own toolchain. This is where crosstool-NG comes into play.
yann@1
    72
rpjday@436
    73
There are also a number of tools that build toolchains for specific needs,
rpjday@436
    74
which are not really scalable. Examples are:
rpjday@436
    75
  - buildroot (buildroot.uclibc.org) whose main purpose is to build root file
yann@1
    76
    systems, hence the name. But once you have your toolchain with buildroot,
yann@1
    77
    part of it is installed in the root-to-be, so if you want to build a whole
yann@1
    78
    new root, you either have to save the existing one as a template and
yann@1
    79
    restore it later, or restart again from scratch. This is not convenient,
yann@1
    80
  - ptxdist (www.pengutronix.de/software/ptxdist), whose purpose is very
yann@1
    81
    similar to buildroot,
yann@702
    82
  - other projects (openembedded.org for example), which are again used to
yann@1
    83
    build root file systems.
yann@1
    84
rpjday@436
    85
crosstool-NG is really targeted at building toolchains, and only toolchains.
yann@1
    86
It is then up to you to use it the way you want.
yann@1
    87
yann@1
    88
___________
yann@1
    89
          /
yann@1
    90
History  /
yann@1
    91
________/
yann@1
    92
rpjday@436
    93
crosstool was first 'conceived' by Dan Kegel, who offered it to the community
yann@1
    94
as a set of scripts, a repository of patches, and some pre-configured, general
yann@1
    95
purpose setup files to be used to configure crosstool. This is available at
yann@203
    96
http://www.kegel.com/crosstool, and the subversion repository is hosted on
yann@203
    97
google at http://code.google.com/p/crosstool/.
yann@1
    98
yann@1
    99
I once managed to add support for uClibc-based toolchains, but it did not make
yann@437
   100
into mainline, mostly because I didn't have time to port the patch forward to
yann@1
   101
the new versions, due in part to the big effort it was taking.
yann@1
   102
yann@1
   103
So I decided to clean up crosstool in the state it was, re-order the things
yann@437
   104
in place, add appropriate support for what I needed, that is uClibc support
yann@437
   105
and a menu-driven configuration, named the new implementation crosstool-NG,
yann@437
   106
(standing for crosstool Next Generation, as many other comunity projects do,
yann@437
   107
and as a wink at the TV series "Star Trek: The Next Generation" ;-) ) and
yann@437
   108
made it available to the community, in case it was of interest to any one.
yann@1
   109
yann@294
   110
___________________________
yann@294
   111
                          /
yann@294
   112
Installing crosstool-NG  /
yann@294
   113
________________________/
yann@294
   114
yann@294
   115
There are two ways you can use crosstool-NG:
yann@294
   116
 - build and install it, then get rid of the sources like you'd do for most
yann@294
   117
   programs,
yann@294
   118
 - or only build it and run from the source directory.
yann@294
   119
yann@294
   120
The former should be used if you got crosstool-NG from a packaged tarball, see
rpjday@436
   121
"Install method", below, while the latter is most useful for developpers that
yann@294
   122
checked the code out from SVN, and want to submit patches, see "The Hacker's
yann@294
   123
way", below.
yann@294
   124
yann@294
   125
Install method |
yann@294
   126
---------------+
yann@294
   127
yann@294
   128
If you go for the install, then you just follow the classical, but yet easy
yann@294
   129
./configure way:
yann@294
   130
  ./configure --prefix=/some/place
yann@294
   131
  make
yann@294
   132
  make install
yann@294
   133
  export PATH="${PATH}:/some/place/bin"
yann@294
   134
yann@294
   135
You can then get rid of crosstool-NG source. Next create a directory to serve
yann@294
   136
as a working place, cd in there and run:
yann@294
   137
  ct-ng help
yann@294
   138
yann@294
   139
See below for complete usage.
yann@294
   140
yann@294
   141
The Hacker's way |
yann@294
   142
-----------------+
yann@294
   143
yann@294
   144
If you go the hacker's way, then the usage is a bit different, although very
yann@294
   145
simple:
yann@294
   146
  ./configure --local
yann@294
   147
  make
yann@1048
   148
  make install
yann@294
   149
yann@294
   150
Now, *do not* remove crosstool-NG sources. They are needed to run crosstool-NG!
yann@294
   151
Stay in the directory holding the sources, and run:
yann@294
   152
  ./ct-ng help
yann@294
   153
yann@294
   154
See below for complete usage.
yann@294
   155
yann@294
   156
Now, provided you checked-out the code, you can send me your interesting changes
yann@294
   157
by running:
yann@294
   158
  svn diff
yann@294
   159
yann@294
   160
and mailing me the result! :-P
yann@294
   161
yann@1048
   162
Preparing for packaging |
yann@1048
   163
------------------------+
yann@1048
   164
yann@1048
   165
If you plan on packaging crosstool-NG, you surely don't want to install it
yann@1048
   166
in your root file system. The install procedure of crosstool-NG honors the
yann@1048
   167
DESTDIR variable:
yann@1048
   168
yann@1048
   169
  ./configure --prefix=/usr
yann@1048
   170
  make
rpjday@1291
   171
  make DESTDIR=/packaging/place install
yann@1048
   172
yann@837
   173
Shell completion |
yann@837
   174
-----------------+
yann@837
   175
yann@837
   176
crosstool-NG comes with a shell script fragment that defines bash-compatible
yann@837
   177
completion. That shell fragment is currently not installed automatically, but
yann@837
   178
this is planned.
yann@837
   179
yann@837
   180
To install the shell script fragment, you have two options:
yann@837
   181
 - install system-wide, most probably by copying ct-ng.comp into
yann@837
   182
   /etc/bash_completion.d/
yann@837
   183
 - install for a single user, by copying ct-ng.comp into ${HOME}/ and
yann@837
   184
   sourcing this file from your ${HOME}/.bashrc
yann@837
   185
yann@456
   186
Contributed code |
yann@456
   187
-----------------+
yann@456
   188
yann@456
   189
Some people contibuted code that couldn't get merged for various reasons. This
yann@456
   190
code is available as patches in the contrib/ sub-directory. These patches are
yann@456
   191
to be applied to the source of crosstool-NG, prior to installing.
yann@456
   192
yann@620
   193
An easy way to use contributed code is to pass the --with-contrib= option to
yann@620
   194
./configure. The possible values depend upon which contributions are packaged
yann@620
   195
with your version, but you can get with it with passing one of those two
yann@620
   196
special values:
yann@620
   197
  --with-contrib=list
yann@620
   198
    will list all available contributions
yann@620
   199
yann@620
   200
  --with-contrib=all
yann@620
   201
    will select all avalaible contributions
yann@620
   202
yann@620
   203
There is no guarantee that a particuliar contribution applies to the current
yann@620
   204
version of crosstool-ng, or that it will work at all. Use contributions at
yann@620
   205
your own risk.
yann@620
   206
yann@168
   207
____________________________
yann@168
   208
                           /
yann@168
   209
Configuring crosstool-NG  /
yann@168
   210
_________________________/
yann@168
   211
yann@620
   212
crosstool-NG is configured with a configurator presenting a menu-stuctured set
yann@620
   213
of options. These options let you specify the way you want your toolchain
yann@620
   214
built, where you want it installed, what architecture and specific processor it
yann@277
   215
will support, the version of the components you want to use, etc... The
yann@277
   216
value for those options are then stored in a configuration file.
yann@277
   217
yann@476
   218
The configurator works the same way you configure your Linux kernel. It is
yann@277
   219
assumed you now how to handle this.
yann@168
   220
yann@168
   221
To enter the menu, type:
yann@192
   222
  ct-ng menuconfig
yann@168
   223
yann@203
   224
Almost every config item has a help entry. Read them carefully.
yann@168
   225
yann@168
   226
String and number options can refer to environment variables. In such a case,
yann@192
   227
you must use the shell syntax: ${VAR}. You shall neither single- nor double-
yann@294
   228
quote the string/number options.
yann@168
   229
yann@192
   230
There are three environment variables that are computed by crosstool-NG, and
yann@168
   231
that you can use:
yann@168
   232
yann@168
   233
CT_TARGET:
yann@335
   234
  It represents the target tuple you are building for. You can use it for
yann@168
   235
  example in the installation/prefix directory, such as:
yann@168
   236
    /opt/x-tools/${CT_TARGET}
yann@168
   237
yann@168
   238
CT_TOP_DIR:
yann@182
   239
  The top directory where crosstool-NG is running. You shouldn't need it in
yann@182
   240
  most cases. There is one case where you may need it: if you have local
yann@182
   241
  patches and you store them in your running directory, you can refer to them
yann@168
   242
  by using CT_TOP_DIR, such as:
yann@168
   243
    ${CT_TOP_DIR}/patches.myproject
yann@168
   244
yann@168
   245
CT_VERSION:
yann@192
   246
  The version of crosstool-NG you are using. Not much use for you, but it's
yann@168
   247
  there if you need it.
yann@168
   248
yann@168
   249
Interesting config options |
yann@476
   250
---------------------------+
yann@168
   251
yann@168
   252
CT_LOCAL_TARBALLS_DIR:
yann@277
   253
  If you already have some tarballs in a direcotry, enter it here. That will
yann@197
   254
  speed up the retrieving phase, where crosstool-NG would otherwise download
yann@168
   255
  those tarballs.
yann@168
   256
yann@168
   257
CT_PREFIX_DIR:
yann@168
   258
  This is where the toolchain will be installed in (and for now, where it
yann@437
   259
  will run from). Common use is to add the target tuple in the directory
yann@277
   260
  path, such as (see above):
yann@277
   261
    /opt/x-tools/${CT_TARGET}
yann@168
   262
yann@168
   263
CT_TARGET_VENDOR:
yann@168
   264
  An identifier for your toolchain, will take place in the vendor part of the
yann@335
   265
  target tuple. It shall *not* contain spaces or dashes. Usually, keep it
yann@168
   266
  to a one-word string, or use underscores to separate words if you need.
yann@168
   267
  Avoid dots, commas, and special characters.
yann@168
   268
yann@168
   269
CT_TARGET_ALIAS:
yann@168
   270
  An alias for the toolchian. It will be used as a prefix to the toolchain
yann@168
   271
  tools. For example, you will have ${CT_TARGET_ALIAS}-gcc
yann@168
   272
yann@246
   273
Also, if you think you don't see enough versions, you can try to enable one of
yann@246
   274
those:
yann@246
   275
yann@246
   276
CT_OBSOLETE:
yann@246
   277
  Show obsolete versions or tools. Most of the time, you don't want to base
yann@246
   278
  your toolchain on too old a version (of gcc, for example). But at times, it
yann@246
   279
  can come handy to use such an old version for regression tests. Those old
yann@1300
   280
  versions are hidden behind CT_OBSOLETE. Those versions (or features) are so
yann@1300
   281
  marked because maintaining support for those in crosstool-NG would be too
yann@1300
   282
  costly, time-wise, and time is dear.
yann@246
   283
yann@246
   284
CT_EXPERIMENTAL:
yann@246
   285
  Show experimental versions or tools. Again, you might not want to base your
yann@246
   286
  toolchain on too recent tools (eg. gcc) for production. But if you need a
yann@246
   287
  feature present only in a recent version, or a new tool, you can find them
yann@1300
   288
  hidden behind CT_EXPERIMENTAL. Those versions (or features) did not (yet)
yann@1300
   289
  receive thorough testing in crosstool-NG, and/or are not mature enough to
yann@1300
   290
  be blindly trusted.
yann@246
   291
yann@276
   292
Re-building an existing toolchain |
yann@276
   293
----------------------------------+
yann@276
   294
yann@276
   295
If you have an existing toolchain, you can re-use the options used to build it
yann@276
   296
to create a new toolchain. That needs a very little bit of effort on your side
yann@894
   297
but is quite easy. The options to build a toolchain are saved with the
yann@894
   298
toolchain, and you can retrieve this configuration by running:
yann@894
   299
  ${CT_TARGET}-config
yann@276
   300
yann@894
   301
This will dump the configuration to stdout, so to rebuild a toolchain with this
yann@894
   302
configuration, the following is all you need to do:
yann@894
   303
  ${CT_TARGET}-config >.config
yann@1098
   304
  ct-ng oldconfig
yann@276
   305
yann@894
   306
Then, you can review and change the configuration by running:
yann@894
   307
  ct-ng menuconfig
yann@276
   308
yann@168
   309
________________________
yann@168
   310
                       /
yann@168
   311
Running crosstool-NG  /
yann@168
   312
_____________________/
yann@1
   313
yann@168
   314
To build the toolchain, simply type:
yann@203
   315
  ct-ng build
yann@135
   316
yann@135
   317
This will use the above configuration to retrieve, extract and patch the
yann@135
   318
components, build, install and eventually test your newly built toolchain.
yann@1
   319
yann@1
   320
You are then free to add the toolchain /bin directory in your PATH to use
yann@1
   321
it at will.
yann@1
   322
yann@135
   323
In any case, you can get some terse help. Just type:
yann@192
   324
  ct-ng help
yann@203
   325
or:
yann@203
   326
  man 1 ct-ng
yann@135
   327
rpjday@436
   328
Stopping and restarting a build |
yann@476
   329
--------------------------------+
yann@135
   330
yann@135
   331
If you want to stop the build after a step you are debugging, you can pass the
yann@135
   332
variable STOP to make:
yann@192
   333
  ct-ng STOP=some_step
yann@135
   334
yann@135
   335
Conversely, if you want to restart a build at a specific step you are
yann@135
   336
debugging, you can pass the RESTART variable to make:
yann@192
   337
  ct-ng RESTART=some_step
yann@135
   338
yann@136
   339
Alternatively, you can call make with the name of a step to just do that step:
yann@192
   340
  ct-ng libc_headers
yann@136
   341
is equivalent to:
yann@620
   342
  ct-ng RESTART=libc_headers STOP=libc_headers
yann@136
   343
yann@304
   344
The shortcuts +step_name and step_name+ allow to respectively stop or restart
yann@136
   345
at that step. Thus:
yann@304
   346
  ct-ng +libc_headers        and:    ct-ng libc_headers+
yann@136
   347
are equivalent to:
yann@192
   348
  ct-ng STOP=libc_headers    and:    ct-ng RESTART=libc_headers
yann@136
   349
yann@181
   350
To obtain the list of acceptable steps, please call:
yann@544
   351
  ct-ng list-steps
yann@181
   352
yann@168
   353
Note that in order to restart a build, you'll have to say 'Y' to the config
yann@168
   354
option CT_DEBUG_CT_SAVE_STEPS, and that the previous build effectively went
yann@168
   355
that far.
yann@92
   356
yann@1025
   357
Building all toolchains at once |
yann@1025
   358
--------------------------------+
yann@92
   359
yann@1025
   360
You can build all samples; simply call:
yann@1025
   361
  ct-ng build-all
yann@40
   362
yann@335
   363
Overriding the number of // jobs |
yann@476
   364
---------------------------------+
yann@335
   365
yann@335
   366
If you want to override the number of jobs to run in // (the -j option to
yann@335
   367
make), you can either re-enter the menuconfig, or simply add it on the command
yann@335
   368
line, as such:
yann@335
   369
  ct-ng build.4
yann@335
   370
yann@335
   371
which tells crosstool-NG to override the number of // jobs to 4.
yann@335
   372
yann@335
   373
You can see the actions that support overriding the number of // jobs in
yann@335
   374
the help menu. Those are the ones with [.#] after them (eg. build[.#] or
yann@1025
   375
build-all[.#], and so on...).
yann@1025
   376
yann@1025
   377
Note on // jobs |
yann@1025
   378
----------------+
yann@1025
   379
yann@1025
   380
The crosstool-NG script 'ct-ng' is a Makefile-script. It does *not* execute
yann@1025
   381
in parallel (there is not much to gain). When speaking of // jobs, we are
yann@1025
   382
refering to the number of // jobs when making the *components*. That is, we
yann@1025
   383
speak of the number of // jobs used to build gcc, glibc, and so on...
yann@1025
   384
yann@335
   385
yann@227
   386
_______________________
yann@227
   387
                      /
yann@227
   388
Using the toolchain  /
yann@227
   389
____________________/
yann@227
   390
yann@227
   391
Using the toolchain is as simple as adding the toolchain's bin directory in
yann@227
   392
your PATH, such as:
yann@227
   393
  export PATH="${PATH}:/your/toolchain/path/bin"
yann@227
   394
yann@335
   395
and then using the target tuple to tell the build systems to use your
yann@227
   396
toolchain:
yann@335
   397
  ./configure --target=your-target-tuple
yann@294
   398
or
yann@335
   399
  make CC=your-target-tuple-gcc
yann@294
   400
or
yann@335
   401
  make CROSS_COMPILE=your-target-tuple-
yann@294
   402
and so on...
yann@227
   403
yann@476
   404
It is strongly advised not to use the toolchain sys-root directory as an
yann@620
   405
install directory for your programs/packages. If you do so, you will not be
yann@476
   406
able to use your toolchain for another project. It is even strongly advised
yann@476
   407
that your toolchain is chmod-ed to read-only once successfully build, so that
yann@620
   408
you don't go polluting your toolchain with your programs/packages' files.
yann@476
   409
yann@476
   410
Thus, when you build a program/package, install it in a separate directory,
yann@476
   411
eg. /your/root. This directory is the /image/ of what would be in the root file
yann@620
   412
system of your target, and will contain all that your programs/packages have
yann@476
   413
installed.
yann@476
   414
Yann@1405
   415
The 'populate' script |
Yann@1405
   416
----------------------+
Yann@1405
   417
yann@227
   418
When your root directory is ready, it is still missing some important bits: the
yann@227
   419
toolchain's libraries. To populate your root directory with those libs, just
yann@227
   420
run:
yann@335
   421
  your-target-tuple-populate -s /your/root -d /your/root-populated
yann@227
   422
yann@227
   423
This will copy /your/root into /your/root-populated, and put the needed and only
yann@227
   424
the needed libraries there. Thus you don't polute /your/root with any cruft that
yann@227
   425
would no longer be needed should you have to remove stuff. /your/root always
yann@227
   426
contains only those things you install in it.
yann@227
   427
yann@227
   428
You can then use /your/root-populated to build up your file system image, a
yann@227
   429
tarball, or to NFS-mount it from your target, or whatever you need.
yann@227
   430
Yann@1405
   431
The populate script accepts the following options:
yann@294
   432
Yann@1405
   433
 -s src_dir
Yann@1405
   434
    Use 'src_dir' as the un-populated root directory.
yann@294
   435
Yann@1405
   436
 -d dst_dir
Yann@1405
   437
    Put the populated root directory in 'dst_dir'.
Yann@1405
   438
Yann@1405
   439
 -l lib1 [...]
Yann@1405
   440
    Always add specified libraries.
Yann@1405
   441
Yann@1405
   442
 -L file
Yann@1405
   443
    Always add libraries listed in 'file'.
yann@294
   444
yann@294
   445
 -f
Yann@1405
   446
    Remove 'dst_dir' if it previously existed; continue even if any library
Yann@1405
   447
    specified with -l or -L is missing.
yann@294
   448
yann@294
   449
 -v
yann@294
   450
    Be verbose, and tell what's going on (you can see exactly where libs are
yann@294
   451
    coming from).
yann@294
   452
yann@294
   453
 -h
Yann@1405
   454
    Print the help.
Yann@1405
   455
Yann@1405
   456
See 'your-target-tuple-populate -h' for more information on the options.
yann@294
   457
yann@40
   458
___________________
yann@40
   459
                  /
yann@40
   460
Toolchain types  /
yann@40
   461
________________/
yann@40
   462
yann@40
   463
There are four kinds of toolchains you could encounter.
yann@40
   464
yann@40
   465
First off, you must understand the following: when it comes to compilers there
yann@40
   466
are up to four machines involved:
yann@40
   467
  1) the machine configuring the toolchain components: the config machine
yann@40
   468
  2) the machine building the toolchain components:    the build machine
yann@40
   469
  3) the machine running the toolchain:                the host machine
yann@203
   470
  4) the machine the toolchain is generating code for: the target machine
yann@40
   471
yann@40
   472
We can most of the time assume that the config machine and the build machine
yann@40
   473
are the same. Most of the time, this will be true. The only time it isn't
yann@40
   474
is if you're using distributed compilation (such as distcc). Let's forget
yann@40
   475
this for the sake of simplicity.
yann@40
   476
yann@40
   477
So we're left with three machines:
yann@40
   478
 - build
yann@40
   479
 - host
yann@40
   480
 - target
yann@40
   481
yann@40
   482
Any toolchain will involve those three machines. You can be as pretty sure of
yann@40
   483
this as "2 and 2 are 4". Here is how they come into play:
yann@40
   484
yann@40
   485
1) build == host == target
yann@40
   486
    This is a plain native toolchain, targetting the exact same machine as the
yann@40
   487
    one it is built on, and running again on this exact same machine. You have
yann@40
   488
    to build such a toolchain when you want to use an updated component, such
yann@40
   489
    as a newer gcc for example.
yann@197
   490
    crosstool-NG calls it "native".
yann@40
   491
yann@40
   492
2) build == host != target
yann@40
   493
    This is a classic cross-toolchain, which is expected to be run on the same
yann@40
   494
    machine it is compiled on, and generate code to run on a second machine,
yann@40
   495
    the target.
yann@197
   496
    crosstool-NG calls it "cross".
yann@40
   497
yann@40
   498
3) build != host == target
yann@40
   499
    Such a toolchain is also a native toolchain, as it targets the same machine
yann@40
   500
    as it runs on. But it is build on another machine. You want such a
yann@40
   501
    toolchain when porting to a new architecture, or if the build machine is
yann@40
   502
    much faster than the host machine.
yann@197
   503
    crosstool-NG calls it "cross-native".
yann@40
   504
yann@40
   505
4) build != host != target
yann@92
   506
    This one is called a canadian-toolchain (*), and is tricky. The three
yann@40
   507
    machines in play are different. You might want such a toolchain if you
yann@40
   508
    have a fast build machine, but the users will use it on another machine,
yann@40
   509
    and will produce code to run on a third machine.
yann@197
   510
    crosstool-NG calls it "canadian".
yann@40
   511
yann@197
   512
crosstool-NG can build all these kinds of toolchains (or is aiming at it,
yann@197
   513
anyway!)
yann@40
   514
yann@40
   515
(*) The term Canadian Cross came about because at the time that these issues
yann@40
   516
    were all being hashed out, Canada had three national political parties.
yann@40
   517
    http://en.wikipedia.org/wiki/Cross_compiler
yann@40
   518
yann@1
   519
_____________
yann@1
   520
            /
yann@1
   521
Internals  /
yann@1
   522
__________/
yann@1
   523
yann@92
   524
Internally, crosstool-NG is script-based. To ease usage, the frontend is
yann@92
   525
Makefile-based.
yann@92
   526
yann@92
   527
Makefile front-end |
yann@476
   528
-------------------+
yann@92
   529
yann@203
   530
The entry point to crosstool-NG is the Makefile script "ct-ng". Calling this
yann@203
   531
script with an action will act exactly as if the Makefile was in the current
yann@203
   532
working directory and make was called with the action as rule. Thus:
yann@203
   533
  ct-ng menuconfig
yann@294
   534
yann@203
   535
is equivalent to having the Makefile in CWD, and calling:
yann@203
   536
  make menuconfig
yann@203
   537
yann@203
   538
Having ct-ng as it is avoids copying the Makefile everywhere, and acts as a
yann@203
   539
traditional command.
yann@203
   540
yann@203
   541
ct-ng loads sub- Makefiles from the library directory $(CT_LIB_DIR), as set up
yann@203
   542
at configuration time with ./configure.
yann@203
   543
yann@437
   544
ct-ng also searches for config files, sub-tools, samples, scripts and patches in
yann@203
   545
that library directory.
yann@92
   546
yann@294
   547
Because of a stupid make behavior/bug I was unable to track down, implicit make
yann@294
   548
rules are disabled: installing with --local would triger those rules, and mconf
yann@294
   549
was unbuildable.
yann@294
   550
yann@182
   551
Kconfig parser |
yann@476
   552
---------------+
yann@92
   553
yann@965
   554
The kconfig language is a hacked version, vampirised from the Linux kernel
yann@965
   555
(http://www.kernel.org/), and (heavily) adapted to my needs.
yann@92
   556
yann@1040
   557
The list of the most notable changes (at least the ones I remember) follows:
yann@1040
   558
- the CONFIG_ prefix has been replaced with CT_
yann@1040
   559
- a leading | in prompts is skipped, and subsequent leading spaces are not
yann@1040
   560
  trimmed
yann@1040
   561
- otherwise leading spaces are silently trimmed
yann@1040
   562
yann@203
   563
The kconfig parsers (conf and mconf) are not installed pre-built, but as
yann@203
   564
source files. Thus you can have the directory where crosstool-NG is installed,
yann@203
   565
exported (via NFS or whatever) and have clients with different architectures
yann@203
   566
use the same crosstool-NG installation, and most notably, the same set of
yann@203
   567
patches.
yann@203
   568
yann@381
   569
Architecture-specific |
yann@476
   570
----------------------+
yann@381
   571
yann@628
   572
Note: this chapter is not really well written, and might thus be a little bit
yann@628
   573
complex to understand. To get a better grasp of what an architecture is, the
yann@628
   574
reader is kindly encouraged to look at the "arch/" sub-directory, and to the
yann@628
   575
existing architectures to see how things are laid out.
yann@628
   576
yann@381
   577
An architecture is defined by:
yann@381
   578
yann@381
   579
 - a human-readable name, in lower case letters, with numbers as appropriate.
yann@628
   580
   The underscore is allowed; space and special characters are not.
yann@628
   581
     Eg.: arm, x86_64
yann@903
   582
 - a file in "config/arch/", named after the architecture's name, and suffixed
yann@903
   583
   with ".in".
yann@903
   584
     Eg.: config/arch/arm.in
yann@903
   585
 - a file in "scripts/build/arch/", named after the architecture's name, and
yann@903
   586
   suffixed with ".sh".
yann@903
   587
     Eg.: scripts/build/arch/arm.sh
yann@628
   588
yann@903
   589
The architecture's ".in" file API:
yann@628
   590
 > the config option "ARCH_%arch%" (where %arch% is to be replaced with the
yann@628
   591
   actual architecture name).
yann@628
   592
   That config option must have *neither* a type, *nor* a prompt! Also, it can
yann@628
   593
   *not* depend on any other config option (EXPERIMENTAL is managed as above).
yann@628
   594
     Eg.:
yann@628
   595
       config ARCH_arm
yann@630
   596
   + mandatory:
yann@702
   597
       defines a (terse) help entry for this architecture:
yann@630
   598
       Eg.:
yann@630
   599
         config ARCH_arm
yann@630
   600
           help
yann@630
   601
             The ARM architecture.
yann@628
   602
   + optional:
yann@628
   603
       selects adequate associated config options.
yann@1038
   604
       Note: 64-bit architectures *shall* select ARCH_64
yann@628
   605
       Eg.:
yann@628
   606
         config ARCH_arm
yann@628
   607
           select ARCH_SUPPORTS_BOTH_ENDIAN
yann@628
   608
           select ARCH_DEFAULT_LE
yann@630
   609
           help
yann@630
   610
             The ARM architecture.
yann@1038
   611
       Eg.:
yann@1038
   612
         config ARCH_x86_64
yann@1038
   613
            select ARCH_64
yann@1038
   614
            help
yann@1038
   615
              The x86_64 architecture.
yann@628
   616
yann@628
   617
 > other target-specific options, at your discretion. Note however that to
yann@628
   618
   avoid name-clashing, such options shall be prefixed with "ARCH_%arch%",
yann@628
   619
   where %arch% is again replaced by the actual architecture name.
yann@628
   620
   (Note: due to historical reasons, and lack of time to clean up the code,
yann@628
   621
    I may have left some config options that do not completely conform to
yann@628
   622
    this, as the architecture name was written all upper case. However, the
yann@628
   623
    prefix is unique among architectures, and does not cause harm).
yann@381
   624
yann@903
   625
The architecture's ".sh" file API:
yann@965
   626
 > the function "CT_DoArchTupleValues"
yann@381
   627
   + parameters: none
yann@381
   628
   + environment:
yann@901
   629
     - all variables from the ".config" file,
yann@901
   630
     - the two variables "target_endian_eb" and "target_endian_el" which are
yann@901
   631
       the endianness suffixes
yann@381
   632
   + return value: 0 upon success, !0 upon failure
yann@381
   633
   + provides:
yann@391
   634
     - mandatory
yann@383
   635
     - the environment variable CT_TARGET_ARCH
yann@389
   636
     - contains:
yann@389
   637
       the architecture part of the target tuple.
yann@389
   638
       Eg.: "armeb" for big endian ARM
yann@389
   639
            "i386" for an i386
yann@389
   640
   + provides:
yann@391
   641
     - optional
yann@389
   642
     - the environment variable CT_TARGET_SYS
yann@456
   643
     - contains:
yann@383
   644
       the sytem part of the target tuple.
yann@383
   645
       Eg.: "gnu" for glibc on most architectures
yann@383
   646
            "gnueabi" for glibc on an ARM EABI
yann@383
   647
     - defaults to:
yann@383
   648
       - for glibc-based toolchain: "gnu"
yann@383
   649
       - for uClibc-based toolchain: "uclibc"
yann@383
   650
   + provides:
yann@383
   651
     - optional
yann@391
   652
     - the environment variable CT_KERNEL_ARCH
yann@383
   653
     - contains:
yann@391
   654
       the architecture name as understandable by the Linux kernel build
yann@391
   655
       system.
yann@391
   656
       Eg.: "arm" for an ARM
yann@391
   657
            "powerpc" for a PowerPC
yann@391
   658
            "i386" for an x86
yann@383
   659
     - defaults to:
yann@391
   660
       ${CT_ARCH}
yann@391
   661
   + provides:
yann@391
   662
     - optional
yann@767
   663
     - the environment variables to configure the cross-gcc (defaults)
yann@767
   664
       - CT_ARCH_WITH_ARCH    : the gcc ./configure switch to select architecture level         ( "--with-arch=${CT_ARCH_ARCH}"   )
yann@767
   665
       - CT_ARCH_WITH_ABI     : the gcc ./configure switch to select ABI level                  ( "--with-abi=${CT_ARCH_ABI}"     )
yann@767
   666
       - CT_ARCH_WITH_CPU     : the gcc ./configure switch to select CPU instruction set        ( "--with-cpu=${CT_ARCH_CPU}"     )
yann@767
   667
       - CT_ARCH_WITH_TUNE    : the gcc ./configure switch to select scheduling                 ( "--with-tune=${CT_ARCH_TUNE}"   )
yann@767
   668
       - CT_ARCH_WITH_FPU     : the gcc ./configure switch to select FPU type                   ( "--with-fpu=${CT_ARCH_FPU}"     )
yann@767
   669
       - CT_ARCH_WITH_FLOAT   : the gcc ./configure switch to select floating point arithmetics ( "--with-float=soft" or /empty/  )
yann@391
   670
   + provides:
yann@391
   671
     - optional
yann@767
   672
     - the environment variables to pass to the cross-gcc to build target binaries (defaults)
yann@391
   673
       - CT_ARCH_ARCH_CFLAG   : the gcc switch to select architecture level                     ( "-march=${CT_ARCH_ARCH}"            )
yann@456
   674
       - CT_ARCH_ABI_CFLAG    : the gcc switch to select ABI level                              ( "-mabi=${CT_ARCH_ABI}"              )
yann@391
   675
       - CT_ARCH_CPU_CFLAG    : the gcc switch to select CPU instruction set                    ( "-mcpu=${CT_ARCH_CPU}"              )
yann@391
   676
       - CT_ARCH_TUNE_CFLAG   : the gcc switch to select scheduling                             ( "-mtune=${CT_ARCH_TUNE}"            )
yann@391
   677
       - CT_ARCH_FPU_CFLAG    : the gcc switch to select FPU type                               ( "-mfpu=${CT_ARCH_FPU}"              )
yann@391
   678
       - CT_ARCH_FLOAT_CFLAG  : the gcc switch to choose floating point arithmetics             ( "-msoft-float" or /empty/           )
yann@391
   679
       - CT_ARCH_ENDIAN_CFLAG : the gcc switch to choose big or little endian                   ( "-mbig-endian" or "-mlittle-endian" )
yann@391
   680
     - default to:
yann@391
   681
       see above.
yann@767
   682
   + provides:
yann@767
   683
     - optional
yann@767
   684
     - the environement variables to configure the core and final compiler, specific to this architecture:
yann@767
   685
       - CT_ARCH_CC_CORE_EXTRA_CONFIG   : additional, architecture specific core gcc ./configure flags
yann@767
   686
       - CT_ARCH_CC_EXTRA_CONFIG        : additional, architecture specific final gcc ./configure flags
yann@767
   687
     - default to:
yann@767
   688
       - all empty
yann@767
   689
   + provides:
yann@767
   690
     - optional
yann@767
   691
     - the architecture-specific CFLAGS and LDFLAGS:
yann@767
   692
       - CT_ARCH_TARGET_CLFAGS
yann@767
   693
       - CT_ARCH_TARGET_LDFLAGS
yann@767
   694
     - default to:
yann@767
   695
       - all empty
yann@628
   696
yann@903
   697
You can have a look at "config/arch/arm.in" and "scripts/build/arch/arm.sh" for
yann@903
   698
a quite complete example of what an actual architecture description looks like.
yann@901
   699
yann@890
   700
Kernel specific |
yann@890
   701
----------------+
yann@890
   702
yann@890
   703
A kernel is defined by:
yann@890
   704
yann@890
   705
 - a human-readable name, in lower case letters, with numbers as appropriate.
yann@890
   706
   The underscore is allowed; space and special characters are not (although
yann@890
   707
   they are internally replaced with underscores.
yann@890
   708
     Eg.: linux, bare-metal
yann@890
   709
 - a file in "config/kernel/", named after the kernel name, and suffixed with
yann@890
   710
   ".in".
yann@890
   711
     Eg.: config/kernel/linux.in, config/kernel/bare-metal.in
yann@901
   712
 - a file in "scripts/build/kernel/", named after the kernel name, and suffixed
yann@901
   713
   with ".sh".
yann@901
   714
     Eg.: scripts/build/kernel/linux.sh, scripts/build/kernel/bare-metal.sh
yann@890
   715
yann@890
   716
The kernel's ".in" file must contain:
yann@890
   717
 > an optional lines containing exactly "# EXPERIMENTAL", starting on the
yann@890
   718
   first column, and without any following space or other character.
yann@890
   719
   If this line is present, then this kernel is considered EXPERIMENTAL,
yann@890
   720
   and correct dependency on EXPERIMENTAL will be set.
yann@901
   721
yann@890
   722
 > the config option "KERNEL_%kernel_name%" (where %kernel_name% is to be
yann@890
   723
   replaced with the actual kernel name, with all special characters and
yann@890
   724
   spaces replaced by underscores).
yann@890
   725
   That config option must have *neither* a type, *nor* a prompt! Also, it can
yann@890
   726
   *not* depends on EXPERIMENTAL.
yann@890
   727
     Eg.: KERNEL_linux, KERNEL_bare_metal
yann@890
   728
   + mandatory:
yann@890
   729
       defines a (terse) help entry for this kernel.
yann@890
   730
       Eg.:
yann@890
   731
         config KERNEL_bare_metal
yann@890
   732
           help
yann@890
   733
             Build a compiler for use without any kernel.
yann@890
   734
   + optional:
yann@890
   735
       selects adequate associated config options.
yann@890
   736
       Eg.:
yann@890
   737
         config KERNEL_bare_metal
yann@890
   738
           select BARE_METAL
yann@890
   739
           help
yann@890
   740
             Build a compiler for use without any kernel.
yann@890
   741
yann@890
   742
 > other kernel specific options, at your discretion. Note however that, to
yann@890
   743
   avoid name-clashing, such options should be prefixed with
yann@890
   744
   "KERNEL_%kernel_name%", where %kernel_name% is again tp be replaced with
yann@890
   745
   the actual kernel name.
yann@890
   746
   (Note: due to historical reasons, and lack of time to clean up the code,
yann@890
   747
    I may have left some config options that do not completely conform to
yann@890
   748
    this, as the kernel name was written all upper case. However, the prefix
yann@890
   749
    is unique among kernels, and does not cause harm).
yann@890
   750
yann@901
   751
The kernel's ".sh" file API:
yann@901
   752
 > is a bash script fragment
yann@901
   753
yann@965
   754
 > defines the function CT_DoKernelTupleValues
yann@965
   755
   + see the architecture's CT_DoArchTupleValues, except for:
yann@965
   756
   + set the environment variable CT_TARGET_KERNEL, the kernel part of the
yann@965
   757
     target tuple
yann@965
   758
   + return value: ignored
yann@965
   759
yann@901
   760
 > defines the function "do_kernel_get":
yann@901
   761
   + parameters: none
yann@901
   762
   + environment:
yann@901
   763
      - all variables from the ".config" file.
yann@901
   764
   + return value: 0 for success, !0 for failure.
yann@901
   765
   + behavior: download the kernel's sources, and store the tarball into
yann@901
   766
     "${CT_TARBALLS_DIR}". To this end, a functions is available, that
yann@901
   767
     abstracts downloading tarballs:
yann@901
   768
     - CT_DoGet <tarball_base_name> <URL1 [URL...]>
yann@901
   769
       Eg.: CT_DoGet linux-2.6.26.5 ftp://ftp.kernel.org/pub/linux/kernel/v2.6
yann@901
   770
     Note: retrieving sources from svn, cvs, git and the likes is not supported
yann@901
   771
     by CT_DoGet. You'll have to do this by hand, as it is done for eglibc in
yann@901
   772
     "scripts/build/libc/eglibc.sh"
yann@901
   773
yann@901
   774
 > defines the function "do_kernel_extract":
yann@901
   775
   + parameters: none
yann@901
   776
   + environment:
yann@901
   777
      - all variables from the ".config" file,
yann@901
   778
   + return value: 0 for success, !0 for failure.
yann@901
   779
   + behavior: extract the kernel's tarball into "${CT_SRC_DIR}", and apply
yann@901
   780
     required patches. To this end, a function is available, that abstracts
yann@901
   781
     extracting tarballs:
yann@901
   782
     - CT_ExtractAndPatch <tarball_base_name>
yann@901
   783
       Eg.: CT_ExtractAndPatch linux-2.6.26.5
yann@901
   784
yann@901
   785
 > defines the function "do_kernel_headers":
yann@901
   786
   + parameters: none
yann@901
   787
   + environment:
yann@901
   788
      - all variables from the ".config" file,
yann@901
   789
   + return value: 0 for success, !0 for failure.
yann@901
   790
   + behavior: install the kernel headers (if any) in "${CT_SYSROOT_DIR}/usr/include"
yann@901
   791
yann@901
   792
 > defines any kernel-specific helper functions
yann@901
   793
   These functions, if any, must be prefixed with "do_kernel_%CT_KERNEL%_",
yann@901
   794
   where '%CT_KERNEL%' is to be replaced with the actual kernel name, to avoid
yann@901
   795
   any name-clashing.
yann@901
   796
yann@901
   797
You can have a look at "config/kernel/linux.in" and "scripts/build/kernel/linux.sh"
yann@903
   798
as an example of what a complex kernel description looks like.
yann@901
   799
yann@620
   800
Adding a new version of a component |
yann@476
   801
------------------------------------+
yann@476
   802
yann@476
   803
When a new component, such as the Linux kernel, gcc or any other is released,
yann@476
   804
adding the new version to crosstool-NG is quite easy. There is a script that
yann@476
   805
will do all that for you:
yann@1095
   806
  scripts/addToolVersion.sh
yann@476
   807
yann@476
   808
Run it with no option to get some help.
yann@381
   809
yann@203
   810
Build scripts |
yann@476
   811
--------------+
yann@203
   812
yann@203
   813
To Be Written later...