More configuration options for dynamic alloc

This makes the internal logic much simpler, and also keeps the datatypes
more similar between STATIC/POINTER cases. It will still be a bit cumbersome
to use because of variable length array member. Macros PB_BYTES_ARRAY_T(n) and
PB_BYTES_ARRAY_T_ALLOCSIZE(n) have been added to make life a bit easier.

This has the drawback that it is no longer as easy to use externally allocated
byte array as input for bytes field in pointer mode. However, this is still
easy to do using callbacks, so it shouldn't be a large issue.

.gitignore

@@ -5,35 +5,24 @@
 *.pb.c
 *.pb.h
 *.pb
+*.pyc
+*_pb2.py
 *~
 *.tar.gz
+.sconsign.dblite
+config.log
+.sconf_temp
+tests/build
 julkaisu.txt
+dist
 docs/*.html
 docs/generator_flow.png
-example/client
+examples/simple/simple
-example/server
+examples/network_server/client
-example_avr_double/decode_double
+examples/network_server/server
-example_avr_double/encode_double
+examples/using_double_on_avr/decode_double
-example_avr_double/test_conversions
+examples/using_double_on_avr/encode_double
-example_unions/decode
+examples/using_double_on_avr/test_conversions
-example_unions/encode
+examples/using_union_messages/decode
+examples/using_union_messages/encode
 generator/nanopb_pb2.pyc
-tests/decode_unittests
-tests/encode_unittests
-tests/test_compiles
-tests/test_decode1
-tests/test_decode2
-tests/test_decode3
-tests/test_decode3_buf
-tests/test_decode_callbacks
-tests/test_encode1
-tests/test_encode2
-tests/test_encode3
-tests/test_encode3_buf
-tests/test_encode_callbacks
-tests/test_missing_fields
-tests/test_multiple_files
-tests/bc_decode
-tests/bc_encode
-tests/breakpoints
-

CHANGELOG

@@ -1,64 +0,0 @@
-git master
- Fixed error message bugs (issues 52, 56)
- Sanitize #ifndef filename (issue 50)
- Performance improvements
- Add compile-time option PB_BUFFER_ONLY
- Add Java package name to nanopb.proto
- Check for sizeof(double) == 8 (issue 54)
- Added generator option to ignore some fields. (issue 51)
- Added generator option to make message structs packed. (issue 49)
- Add more test cases.
-
-nanopb-0.1.8
- Fix bugs in the enum short names introduced in 0.1.7 (issues 42, 43)
- Fix STATIC_ASSERT macro when using multiple .proto files. (issue 41)
- Fix missing initialization of istream.errmsg
- Make tests/Makefile work for non-gcc compilers (issue 40)
-
-nanopb-0.1.7
- Remove "skip" mode from pb_istream_t callbacks. Example implementation had a bug. (issue 37)
- Add option to use shorter names for enum values (issue 38)
- Improve options support in generator (issues 12, 30)
- Add nanopb version number to generated files (issue 36)
- Add extern "C" to generated headers (issue 35)
- Add names for structs to allow forward declaration (issue 39)
- Add buffer size check in example (issue 34)
- Fix build warnings on MS compilers (issue 33)
-
-nanopb-0.1.6
- Reorganize the field decoder interface (issue 2)
- Improve performance in submessage decoding (issue 28)
- Implement error messages in the decoder side (issue 7)
- Extended testcases (alltypes test is now complete).
- Fix some compiler warnings (issues 25, 26, 27, 32).
-
-nanopb-0.1.5
- Fix bug in decoder with packed arrays (issue 23).
- Extended testcases.
- Fix some compiler warnings.
-
-nanopb-0.1.4
- Add compile-time options for easy-to-use >255 field support.
- Improve the detection of missing required fields.
- Added example on how to handle union messages.
- Fix generator error with .proto without messages.
- Fix problems that stopped the code from compiling with some compilers.
- Fix some compiler warnings.
-
-nanopb-0.1.3
- Refactor the field encoder interface.
- Improve generator error messages (issue 5)
- Add descriptor.proto into the #include exclusion list
- Fix some compiler warnings.
-
-nanopb-0.1.2
- Make the generator to generate include for other .proto files (issue 4).
- Fixed generator not working on Windows (issue 3)
-
-nanopb-0.1.1
- Fixed bug in encoder with 'bytes' fields (issue 1).
- Fixed a bug in the generator that caused a compiler error on sfixed32 and sfixed64 fields.
- Extended testcases.
-
-nanopb-0.1.0
- First stable release.

CHANGELOG.txt

@@ -0,0 +1,136 @@
+nanopb-0.2.6 (2014-02-15)
+ Fix generator error with bytes callback fields (issue 99)
+ Fix warnings about large integer constants (issue 102)
+ Add comments to where STATIC_ASSERT is used (issue 96)
+ Add warning about unknown field names on .options (issue 105)
+ Move descriptor.proto to google/protobuf subdirectory (issue 104)
+ Improved tests
+
+nanopb-0.2.5 (2014-01-01)
+ Fix a bug with encoding negative values in int32 fields (issue 97)
+ Create binary packages of the generator + dependencies (issue 47)
+ Add support for pointer-type fields to the encoder (part of issue 80)
+ Fixed path in FindNanopb.cmake (issue 94)
+ Improved tests
+
+nanopb-0.2.4 (2013-11-07)
+ Remove the deprecated NANOPB_INTERNALS functions from public API.
+ Document the security model.
+ Check array and bytes max sizes when encoding (issue 90)
+ Add #defines for maximum encoded message size (issue 89)
+ Add #define tags for extension fields (issue 93)
+ Fix MISRA C violations (issue 91)
+ Clean up pb_field_t definition with typedefs.
+
+nanopb-0.2.3 (2013-09-18)
+ Improve compatibility by removing ternary operator from initializations (issue 88)
+ Fix build error on Visual C++ (issue 84, patch by Markus Schwarzenberg)
+ Don't stop on unsupported extension fields (issue 83)
+ Add an example pb_syshdr.h file for non-C99 compilers
+ Reorganize tests and examples into subfolders (issue 63)
+ Switch from Makefiles to scons for building the tests
+ Make the tests buildable on Windows
+
+nanopb-0.2.2 (2013-08-18)
+ Add support for extension fields (issue 17)
+ Fix unknown fields in empty message (issue 78)
+ Include the field tags in the generated .pb.h file.
+ Add pb_decode_delimited and pb_encode_delimited wrapper functions (issue 74)
+ Add a section in top of pb.h for changing compilation settings (issue 76)
+ Documentation improvements (issues 12, 77 and others)
+ Improved tests
+
+nanopb-0.2.1 (2013-04-14)
+ NOTE: The default callback function signature has changed.
+       If you don't want to update your code, define PB_OLD_CALLBACK_STYLE.
+ 
+ Change the callback function to use void** (issue 69)
+ Add support for defining the nanopb options in a separate file (issue 12)
+ Add support for packed structs in IAR and MSVC (in addition to GCC) (issue 66)
+ Implement error message support for the encoder side (issue 7)
+ Handle unterminated strings when encoding (issue 68)
+ Fix bug with empty strings in repeated string callbacks (issue 73)
+ Fix regression in 0.2.0 with optional callback fields (issue 70)
+ Fix bugs with empty message types (issues 64, 65)
+ Fix some compiler warnings on clang (issue 67)
+ Some portability improvements (issues 60, 62)
+ Various new generator options
+ Improved tests
+
+nanopb-0.2.0 (2013-03-02)
+ NOTE: This release requires you to regenerate all .pb.c
+       files. Files generated by older versions will not
+       compile anymore.
+
+ Reformat generated .pb.c files using macros (issue 58)
+ Rename PB_HTYPE_ARRAY -> PB_HTYPE_REPEATED
+ Separate PB_HTYPE to PB_ATYPE and PB_HTYPE
+ Move STATIC_ASSERTs to .pb.c file
+ Added CMake file (by Pavel Ilin)
+ Add option to give file extension to generator (by Michael Haberler)
+ Documentation updates
+
+nanopb-0.1.9 (2013-02-13)
+ Fixed error message bugs (issues 52, 56)
+ Sanitize #ifndef filename (issue 50)
+ Performance improvements
+ Add compile-time option PB_BUFFER_ONLY
+ Add Java package name to nanopb.proto
+ Check for sizeof(double) == 8 (issue 54)
+ Added generator option to ignore some fields. (issue 51)
+ Added generator option to make message structs packed. (issue 49)
+ Add more test cases.
+
+nanopb-0.1.8 (2012-12-13)
+ Fix bugs in the enum short names introduced in 0.1.7 (issues 42, 43)
+ Fix STATIC_ASSERT macro when using multiple .proto files. (issue 41)
+ Fix missing initialization of istream.errmsg
+ Make tests/Makefile work for non-gcc compilers (issue 40)
+
+nanopb-0.1.7 (2012-11-11)
+ Remove "skip" mode from pb_istream_t callbacks. Example implementation had a bug. (issue 37)
+ Add option to use shorter names for enum values (issue 38)
+ Improve options support in generator (issues 12, 30)
+ Add nanopb version number to generated files (issue 36)
+ Add extern "C" to generated headers (issue 35)
+ Add names for structs to allow forward declaration (issue 39)
+ Add buffer size check in example (issue 34)
+ Fix build warnings on MS compilers (issue 33)
+
+nanopb-0.1.6 (2012-09-02)
+ Reorganize the field decoder interface (issue 2)
+ Improve performance in submessage decoding (issue 28)
+ Implement error messages in the decoder side (issue 7)
+ Extended testcases (alltypes test is now complete).
+ Fix some compiler warnings (issues 25, 26, 27, 32).
+
+nanopb-0.1.5 (2012-08-04)
+ Fix bug in decoder with packed arrays (issue 23).
+ Extended testcases.
+ Fix some compiler warnings.
+
+nanopb-0.1.4 (2012-07-05)
+ Add compile-time options for easy-to-use >255 field support.
+ Improve the detection of missing required fields.
+ Added example on how to handle union messages.
+ Fix generator error with .proto without messages.
+ Fix problems that stopped the code from compiling with some compilers.
+ Fix some compiler warnings.
+
+nanopb-0.1.3 (2012-06-12)
+ Refactor the field encoder interface.
+ Improve generator error messages (issue 5)
+ Add descriptor.proto into the #include exclusion list
+ Fix some compiler warnings.
+
+nanopb-0.1.2 (2012-02-15)
+ Make the generator to generate include for other .proto files (issue 4).
+ Fixed generator not working on Windows (issue 3)
+
+nanopb-0.1.1 (2012-01-14)
+ Fixed bug in encoder with 'bytes' fields (issue 1).
+ Fixed a bug in the generator that caused a compiler error on sfixed32 and sfixed64 fields.
+ Extended testcases.
+
+nanopb-0.1.0 (2012-01-06)
+ First stable release.

README

@@ -1,11 +0,0 @@
-Nanopb is a small code-size Protocol Buffers implementation.
-
-Homepage: http://kapsi.fi/~jpa/nanopb/
-
-To compile the library, you'll need these libraries:
-protobuf-compiler python-protobuf libprotobuf-dev
-
-The only runtime dependencies are memset() and memcpy().
-
-To run the tests, run make under the tests folder.
-If it completes without error, everything is fine.

README.txt

@@ -0,0 +1,61 @@
+Nanopb is a small code-size Protocol Buffers implementation in ansi C. It is
+especially suitable for use in microcontrollers, but fits any memory
+restricted system.
+
+Homepage: http://kapsi.fi/~jpa/nanopb/
+
+
+
+
+Using the nanopb library
+========================
+To use the nanopb library, you need to do two things:
+
+1) Compile your .proto files for nanopb, using protoc.
+2) Include pb_encode.c and pb_decode.c in your project.
+
+The easiest way to get started is to study the project in "examples/simple".
+It contains a Makefile, which should work directly under most Linux systems.
+However, for any other kind of build system, see the manual steps in
+README.txt in that folder.
+
+
+
+Using the Protocol Buffers compiler (protoc)
+============================================
+The nanopb generator is implemented as a plugin for the Google's own protoc
+compiler. This has the advantage that there is no need to reimplement the
+basic parsing of .proto files. However, it does mean that you need the
+Google's protobuf library in order to run the generator.
+
+If you have downloaded a binary package for nanopb (either Windows, Linux or
+Mac OS X version), the 'protoc' binary is included in the 'generator-bin'
+folder. In this case, you are ready to go. Simply run this command:
+
+    generator-bin/protoc --nanopb_out=. myprotocol.proto
+
+However, if you are using a git checkout or a plain source distribution, you
+need to provide your own version of protoc and the Google's protobuf library.
+On Linux, the necessary packages are protobuf-compiler and python-protobuf.
+On Windows, you can either build Google's protobuf library from source or use
+one of the binary distributions of it. In either case, if you use a separate
+protoc, you need to manually give the path to nanopb generator:
+
+    protoc --plugin=protoc-gen-nanopb=nanopb/generator/protoc-gen-nanopb ...
+
+
+
+Running the tests
+=================
+If you want to perform further development of the nanopb core, or to verify
+its functionality using your compiler and platform, you'll want to run the
+test suite. The build rules for the test suite are implemented using Scons,
+so you need to have that installed. To run the tests:
+
+    cd tests
+    scons
+
+This will show the progress of various test cases. If the output does not
+end in an error, the test cases were successful.
+
+

docs/Makefile

@@ -1,4 +1,4 @@
-all: index.html concepts.html reference.html \
+all: index.html concepts.html reference.html security.html \
 	generator_flow.png
 
 %.png: %.svg

docs/concepts.rst

@@ -10,47 +10,40 @@ The things outlined here are the underlying concepts of the nanopb design.
 
 Proto files
 ===========
-All Protocol Buffers implementations use .proto files to describe the message format.
+All Protocol Buffers implementations use .proto files to describe the message
-The point of these files is to be a portable interface description language.
+format. The point of these files is to be a portable interface description
+language.
 
 Compiling .proto files for nanopb
 ---------------------------------
-Nanopb uses the Google's protoc compiler to parse the .proto file, and then a python script to generate the C header and source code from it::
+Nanopb uses the Google's protoc compiler to parse the .proto file, and then a
+python script to generate the C header and source code from it::
 
     user@host:~$ protoc -omessage.pb message.proto
     user@host:~$ python ../generator/nanopb_generator.py message.pb
     Writing to message.h and message.c
     user@host:~$
 
-Compiling .proto files with nanopb options
+Modifying generator behaviour
-------------------------------------------
+-----------------------------
-Nanopb defines two extensions for message fields described in .proto files: *max_size* and *max_count*.
+Using generator options, you can set maximum sizes for fields in order to
-These are the maximum size of a string and maximum count of items in an array::
+allocate them statically. The preferred way to do this is to create an .options
+file with the same name as your .proto file::
 
-    required string name = 1 [(nanopb).max_size = 40];
+   # Foo.proto
-    repeated PhoneNumber phone = 4 [(nanopb).max_count = 5];
+   message Foo {
-
+      required string name = 1;
-To use these extensions, you need to place an import statement in the beginning of the file::
-
-    import "nanopb.proto";
-
-This file, in turn, requires the file *google/protobuf/descriptor.proto*. This is usually installed under */usr/include*. Therefore, to compile a .proto file which uses options, use a protoc command similar to::
-
-    protoc -I/usr/include -Inanopb/generator -I. -omessage.pb message.proto
-
-The options can be defined in file, message and field scopes::
-
-    option (nanopb_fileopt).max_size = 20; // File scope
-    message Message
-    {
-        option (nanopb_msgopt).max_size = 30; // Message scope
-        required string fieldsize = 1 [(nanopb).max_size = 40]; // Field scope
    }
 
-It is also possible to give the options on command line, but then they will affect the whole file. For example::
+::
 
-    user@host:~$ python ../generator/nanopb_generator.py -s 'max_size: 20' message.pb
+   # Foo.options
+   Foo.name max_size:16
 
+For more information on this, see the `Proto file options`_ section in the
+reference manual.
+
+.. _`Proto file options`: reference.html#proto-file-options
 
 Streams
 =======
@@ -181,7 +174,9 @@ Field callbacks
 ===============
 When a field has dynamic length, nanopb cannot statically allocate storage for it. Instead, it allows you to handle the field in whatever way you want, using a callback function.
 
-The `pb_callback_t`_ structure contains a function pointer and a *void* pointer you can use for passing data to the callback. If the function pointer is NULL, the field will be skipped. The actual behavior of the callback function is different in encoding and decoding modes.
+The `pb_callback_t`_ structure contains a function pointer and a *void* pointer called *arg* you can use for passing data to the callback. If the function pointer is NULL, the field will be skipped. A pointer to the *arg* is passed to the function, so that it can modify it and retrieve the value.
+
+The actual behavior of the callback function is different in encoding and decoding modes. In encoding mode, the callback is called once and should write out everything, including field tags. In decoding mode, the callback is called repeatedly for every data item.
 
 .. _`pb_callback_t`: reference.html#pb-callback-t
 
@@ -189,7 +184,7 @@ Encoding callbacks
 ------------------
 ::
 
-    bool (*encode)(pb_ostream_t *stream, const pb_field_t *field, const void *arg);
+    bool (*encode)(pb_ostream_t *stream, const pb_field_t *field, void * const *arg);
 
 When encoding, the callback should write out complete fields, including the wire type and field number tag. It can write as many or as few fields as it likes. For example, if you want to write out an array as *repeated* field, you should do it all in a single call.
 
@@ -203,7 +198,7 @@ If the callback is used in a submessage, it will be called multiple times during
 
 This callback writes out a dynamically sized string::
 
-    bool write_string(pb_ostream_t *stream, const pb_field_t *field, const void *arg)
+    bool write_string(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
     {
         char *str = get_string_from_somewhere();
         if (!pb_encode_tag_for_field(stream, field))
@@ -216,7 +211,7 @@ Decoding callbacks
 ------------------
 ::
 
-    bool (*decode)(pb_istream_t *stream, const pb_field_t *field, void *arg);
+    bool (*decode)(pb_istream_t *stream, const pb_field_t *field, void **arg);
 
 When decoding, the callback receives a length-limited substring that reads the contents of a single field. The field tag has already been read. For *string* and *bytes*, the length value has already been parsed, and is available at *stream->bytes_left*.
 
@@ -226,7 +221,7 @@ The callback will be called multiple times for repeated fields. For packed field
 
 This callback reads multiple integers and prints them::
 
-    bool read_ints(pb_istream_t *stream, const pb_field_t *field, void *arg)
+    bool read_ints(pb_istream_t *stream, const pb_field_t *field, void **arg)
     {
         while (stream->bytes_left)
         {
@@ -255,20 +250,56 @@ For example this submessage in the Person.proto file::
 generates this field description array for the structure *Person_PhoneNumber*::
 
  const pb_field_t Person_PhoneNumber_fields[3] = {
-    {1, PB_HTYPE_REQUIRED | PB_LTYPE_STRING,
+    PB_FIELD(  1, STRING  , REQUIRED, STATIC, Person_PhoneNumber, number, number, 0),
-    offsetof(Person_PhoneNumber, number), 0,
+    PB_FIELD(  2, ENUM    , OPTIONAL, STATIC, Person_PhoneNumber, type, number, &Person_PhoneNumber_type_default),
-    pb_membersize(Person_PhoneNumber, number), 0, 0},
-
-    {2, PB_HTYPE_OPTIONAL | PB_LTYPE_VARINT,
-    pb_delta(Person_PhoneNumber, type, number),
-    pb_delta(Person_PhoneNumber, has_type, type),
-    pb_membersize(Person_PhoneNumber, type), 0,
-    &Person_PhoneNumber_type_default},
-
     PB_LAST_FIELD
  };
 
 
+Extension fields
+================
+Protocol Buffers supports a concept of `extension fields`_, which are
+additional fields to a message, but defined outside the actual message.
+The definition can even be in a completely separate .proto file.
+
+The base message is declared as extensible by keyword *extensions* in
+the .proto file::
+
+ message MyMessage {
+     .. fields ..
+     extensions 100 to 199;
+ }
+
+For each extensible message, *nanopb_generator.py* declares an additional
+callback field called *extensions*. The field and associated datatype
+*pb_extension_t* forms a linked list of handlers. When an unknown field is
+encountered, the decoder calls each handler in turn until either one of them
+handles the field, or the list is exhausted.
+
+The actual extensions are declared using the *extend* keyword in the .proto,
+and are in the global namespace::
+
+ extend MyMessage {
+     optional int32 myextension = 100;
+ }
+
+For each extension, *nanopb_generator.py* creates a constant of type
+*pb_extension_type_t*. To link together the base message and the extension,
+you have to:
+
+1. Allocate storage for your field, matching the datatype in the .proto.
+   For example, for a *int32* field, you need a *int32_t* variable to store
+   the value.
+2. Create a *pb_extension_t* constant, with pointers to your variable and
+   to the generated *pb_extension_type_t*.
+3. Set the *message.extensions* pointer to point to the *pb_extension_t*.
+
+An example of this is available in *tests/test_encode_extensions.c* and
+*tests/test_decode_extensions.c*.
+
+.. _`extension fields`: https://developers.google.com/protocol-buffers/docs/proto#extensions
+
+
 Return values and error handling
 ================================
 
@@ -276,8 +307,8 @@ Most functions in nanopb return bool: *true* means success, *false* means failur
 
 The error messages help in guessing what is the underlying cause of the error. The most common error conditions are:
 
-1) Running out of memory. Because everything is allocated from the stack, nanopb can't detect this itself. Encoding or decoding the same type of a message always takes the same amount of stack space. Therefore, if it works once, it works always.
+1) Running out of memory, i.e. stack overflow.
-2) Invalid field description. These are usually stored as constants, so if it works under the debugger, it always does.
+2) Invalid field descriptors (would usually mean a bug in the generator).
 3) IO errors in your own stream callbacks.
 4) Errors that happen in your callback functions.
 5) Exceeding the max_size or bytes_left of a stream.

docs/index.rst

@@ -36,23 +36,26 @@ Features and limitations
 **Features**
 
 #) Pure C runtime
-#) Small code size (2–10 kB depending on processor)
+#) Small code size (2–10 kB depending on processor, plus any message definitions)
-#) Small ram usage (typically 200 bytes)
+#) Small ram usage (typically ~300 bytes, plus any message structs)
 #) Allows specifying maximum size for strings and arrays, so that they can be allocated statically.
 #) No malloc needed: everything can be allocated statically or on the stack.
 #) You can use either encoder or decoder alone to cut the code size in half.
+#) Support for most protobuf features, including: all data types, nested submessages, default values, repeated and optional fields, packed arrays, extension fields.
+#) Callback mechanism for handling messages larger than can fit in available RAM.
+#) Extensive set of tests.
 
 **Limitations**
 
 #) User must provide callbacks when decoding arrays or strings without maximum size. Malloc support could be added as a separate module.
-#) Some speed has been sacrificed for code size. For example varint calculations are always done in 64 bits.
+#) Some speed has been sacrificed for code size.
 #) Encoding is focused on writing to streams. For memory buffers only it could be made more efficient.
 #) The deprecated Protocol Buffers feature called "groups" is not supported.
 #) Fields in the generated structs are ordered by the tag number, instead of the natural ordering in .proto file.
 #) Unknown fields are not preserved when decoding and re-encoding a message.
 #) Reflection (runtime introspection) is not supported. E.g. you can't request a field by giving its name in a string.
 #) Numeric arrays are always encoded as packed, even if not marked as packed in .proto. This causes incompatibility with decoders that do not support packed format.
-#) Cyclic references between messages are not supported. They could be supported in callback-mode if there was an option in the generator to set the mode.
+#) Cyclic references between messages are supported only in callback mode.
 
 Getting started
 ===============
@@ -87,27 +90,38 @@ After that, buffer will contain the encoded message.
 The number of bytes in the message is stored in *stream.bytes_written*.
 You can feed the message to *protoc --decode=Example message.proto* to verify its validity.
 
-For complete examples of the simple cases, see *tests/test_decode1.c* and *tests/test_encode1.c*. For an example with network interface, see the *example* subdirectory.
+For a complete example of the simple case, see *example/simple.c*.
+For a more complex example with network interface, see the *example/network_server* subdirectory.
 
 Compiler requirements
 =====================
-Nanopb should compile with most ansi-C compatible compilers. It however requires a few header files to be available:
+Nanopb should compile with most ansi-C compatible compilers. It however
+requires a few header files to be available:
 
 #) *string.h*, with these functions: *strlen*, *memcpy*, *memset*
 #) *stdint.h*, for definitions of *int32_t* etc.
 #) *stddef.h*, for definition of *size_t*
 #) *stdbool.h*, for definition of *bool*
 
-If these header files do not come with your compiler, you should be able to find suitable replacements online. Mostly the requirements are very simple, just a few basic functions and typedefs.
+If these header files do not come with your compiler, you can use the
+file *extra/pb_syshdr.h* instead. It contains an example of how to provide
+the dependencies. You may have to edit it a bit to suit your custom platform.
 
-Debugging and testing
+To use the pb_syshdr.h, define *PB_SYSTEM_HEADER* as *"pb_syshdr.h"* (including the quotes).
-=====================
+Similarly, you can provide a custom include file, which should provide all the dependencies
-Extensive unittests are included under the *tests* folder. Just type *make* there to run the tests.
+listed above.
 
-This also generates a file called *breakpoints* which includes all lines returning *false* in nanopb. You can use this in gdb by typing *source breakpoints*, after which gdb will break on first nanopb error.
+Running the test cases
+======================
+Extensive unittests and test cases are included under the *tests* folder.
+
+To build the tests, you will need the `scons`__ build system. The tests should
+be runnable on most platforms. Windows and Linux builds are regularly tested.
+
+__ http://www.scons.org/
+
+In addition to the build system, you will also need a working Google Protocol
+Buffers *protoc* compiler, and the Python bindings for Protocol Buffers. On
+Debian-based systems, install the following packages: *protobuf-compiler*,
+*python-protobuf* and *libprotobuf-dev*.
 
-Wishlist
-========
-#) A specialized encoder for encoding to a memory buffer. Should serialize in reverse order to avoid having to determine submessage size beforehand.
-#) A cleaner rewrite of the Python-based source generator.
-#) Better performance for 16- and 8-bit platforms: use smaller datatypes where possible.

docs/menu.rst

@@ -3,8 +3,9 @@
     1) `Overview`_
     2) `Concepts`_
     3) `API reference`_
+    4) `Security model`_
     
 .. _`Overview`: index.html
 .. _`Concepts`: concepts.html
 .. _`API reference`: reference.html
-
+.. _`Security model`: security.html

docs/reference.rst

@@ -6,29 +6,192 @@ Nanopb: API reference
 
 .. contents ::
 
+
+
+
 Compilation options
 ===================
-The following options can be specified using -D switch given to the C compiler:
+The following options can be specified in one of two ways:
+
+1. Using the -D switch on the C compiler command line.
+2. By #defining them at the top of pb.h.
+
+You must have the same settings for the nanopb library and all code that
+includes pb.h.
+
+============================  ================================================
+__BIG_ENDIAN__                 Set this if your platform stores integers and
+                               floats in big-endian format. Mixed-endian
+                               systems (different layout for ints and floats)
+                               are currently not supported.
+NANOPB_INTERNALS               Set this to expose the field encoder functions
+                               that are hidden since nanopb-0.1.3. Starting
+                               with nanopb-0.2.4, this flag does nothing. Use
+                               the newer functions that have better interface.
+PB_ENABLE_MALLOC               Set this to enable dynamic allocation support
+                               in the decoder.
+PB_MAX_REQUIRED_FIELDS         Maximum number of required fields to check for
+                               presence. Default value is 64. Increases stack
+                               usage 1 byte per every 8 fields. Compiler
+                               warning will tell if you need this.
+PB_FIELD_16BIT                 Add support for tag numbers > 255 and fields
+                               larger than 255 bytes or 255 array entries.
+                               Increases code size 3 bytes per each field.
+                               Compiler error will tell if you need this.
+PB_FIELD_32BIT                 Add support for tag numbers > 65535 and fields
+                               larger than 65535 bytes or 65535 array entries.
+                               Increases code size 9 bytes per each field.
+                               Compiler error will tell if you need this.
+PB_NO_ERRMSG                   Disables the support for error messages; only
+                               error information is the true/false return
+                               value. Decreases the code size by a few hundred
+                               bytes.
+PB_BUFFER_ONLY                 Disables the support for custom streams. Only
+                               supports encoding and decoding with memory
+                               buffers. Speeds up execution and decreases code
+                               size slightly.
+PB_OLD_CALLBACK_STYLE          Use the old function signature (void\* instead
+                               of void\*\*) for callback fields. This was the
+                               default until nanopb-0.2.1.
+PB_SYSTEM_HEADER               Replace the standard header files with a single
+                               header file. It should define all the required
+                               functions and typedefs listed on the
+                               `overview page`_. Value must include quotes,
+                               for example *#define PB_SYSTEM_HEADER "foo.h"*.
+============================  ================================================
+
+The PB_MAX_REQUIRED_FIELDS, PB_FIELD_16BIT and PB_FIELD_32BIT settings allow
+raising some datatype limits to suit larger messages. Their need is recognized
+automatically by C-preprocessor #if-directives in the generated .pb.h files.
+The default setting is to use the smallest datatypes (least resources used).
+
+.. _`overview page`: index.html#compiler-requirements
+
+
+Proto file options
+==================
+The generator behaviour can be adjusted using these options, defined in the
+'nanopb.proto' file in the generator folder:
+
+============================  ================================================
+max_size                       Allocated size for *bytes* and *string* fields.
+max_count                      Allocated number of entries in arrays
+                               (*repeated* fields).
+type                           Type of the generated field. Default value
+                               is *FT_DEFAULT*, which selects automatically.
+                               You can use *FT_CALLBACK*, *FT_POINTER*,
+                               *FT_STATIC* or *FT_IGNORE* to force a callback
+                               field, a dynamically allocated field, a static
+                               field or to completely ignore the field.
+long_names                     Prefix the enum name to the enum value in
+                               definitions, i.e. *EnumName_EnumValue*. Enabled
+                               by default.
+packed_struct                  Make the generated structures packed.
+                               NOTE: This cannot be used on CPUs that break
+                               on unaligned accesses to variables.
+============================  ================================================
+
+These options can be defined for the .proto files before they are converted
+using the nanopb-generatory.py. There are three ways to define the options:
+
+1. Using a separate .options file.
+   This is the preferred way as of nanopb-0.2.1, because it has the best
+   compatibility with other protobuf libraries.
+2. Defining the options on the command line of nanopb_generator.py.
+   This only makes sense for settings that apply to a whole file.
+3. Defining the options in the .proto file using the nanopb extensions.
+   This is the way used in nanopb-0.1, and will remain supported in the
+   future. It however sometimes causes trouble when using the .proto file
+   with other protobuf libraries.
+
+The effect of the options is the same no matter how they are given. The most
+common purpose is to define maximum size for string fields in order to
+statically allocate them.
+
+Defining the options in a .options file
+---------------------------------------
+The preferred way to define options is to have a separate file
+'myproto.options' in the same directory as the 'myproto.proto'. ::
+
+    # myproto.proto
+    message MyMessage {
+        required string name = 1;
+        repeated int32 ids = 4;
+    }
+
+::
+
+    # myproto.options
+    MyMessage.name         max_size:40
+    MyMessage.ids          max_count:5
+
+The generator will automatically search for this file and read the
+options from it. The file format is as follows:
+
+* Lines starting with '#' or '//' are regarded as comments.
+* Blank lines are ignored.
+* All other lines should start with a field name pattern, followed by one or
+  more options. For example: *"MyMessage.myfield max_size:5 max_count:10"*.
+* The field name pattern is matched against a string of form *'Message.field'*.
+  For nested messages, the string is *'Message.SubMessage.field'*.
+* The field name pattern may use the notation recognized by Python fnmatch():
+
+  - *\** matches any part of string, like 'Message.\*' for all fields
+  - *\?* matches any single character
+  - *[seq]* matches any of characters 's', 'e' and 'q'
+  - *[!seq]* matches any other character
+
+* The options are written as *'option_name:option_value'* and several options
+  can be defined on same line, separated by whitespace.
+* Options defined later in the file override the ones specified earlier, so
+  it makes sense to define wildcard options first in the file and more specific
+  ones later.
+  
+If preferred, the name of the options file can be set using the command line
+switch *-f* to nanopb_generator.py.
+
+Defining the options on command line
+------------------------------------
+The nanopb_generator.py has a simple command line option *-s OPTION:VALUE*.
+The setting applies to the whole file that is being processed.
+
+Defining the options in the .proto file
+---------------------------------------
+The .proto file format allows defining custom options for the fields.
+The nanopb library comes with *nanopb.proto* which does exactly that, allowing
+you do define the options directly in the .proto file::
+
+    import "nanopb.proto";
+    
+    message MyMessage {
+        required string name = 1 [(nanopb).max_size = 40];
+        repeated int32 ids = 4   [(nanopb).max_count = 5];
+    }
+
+A small complication is that you have to set the include path of protoc so that
+nanopb.proto can be found. This file, in turn, requires the file
+*google/protobuf/descriptor.proto*. This is usually installed under
+*/usr/include*. Therefore, to compile a .proto file which uses options, use a
+protoc command similar to::
+
+    protoc -I/usr/include -Inanopb/generator -I. -omessage.pb message.proto
+
+The options can be defined in file, message and field scopes::
+
+    option (nanopb_fileopt).max_size = 20; // File scope
+    message Message
+    {
+        option (nanopb_msgopt).max_size = 30; // Message scope
+        required string fieldsize = 1 [(nanopb).max_size = 40]; // Field scope
+    }
+
+
+
+
+
+
 
-============================  ================================================================================================
-__BIG_ENDIAN__                 Set this if your platform stores integers and floats in big-endian format.
-                               Mixed-endian systems (different layout for ints and floats) are currently not supported.
-NANOPB_INTERNALS               Set this to expose the field encoder functions that are hidden since nanopb-0.1.3.
-PB_MAX_REQUIRED_FIELDS         Maximum number of required fields to check for presence. Default value is 64. Increases stack
-                               usage 1 byte per every 8 fields. Compiler warning will tell if you need this.
-PB_FIELD_16BIT                 Add support for tag numbers > 255 and fields larger than 255 bytes or 255 array entries.
-                               Increases code size 3 bytes per each field. Compiler error will tell if you need this.
-PB_FIELD_32BIT                 Add support for tag numbers > 65535 and fields larger than 65535 bytes or 65535 array entries.
-                               Increases code size 9 bytes per each field. Compiler error will tell if you need this.
-PB_NO_ERRMSG                   Disables the support for error messages; only error information is the true/false return value.
-                               Decreases the code size by a few hundred bytes.
-PB_BUFFER_ONLY                 Disables the support for custom streams. Only supports encoding to memory buffers.
-                               Speeds up execution and decreases code size slightly.
-============================  ================================================================================================
 
-The PB_MAX_REQUIRED_FIELDS, PB_FIELD_16BIT and PB_FIELD_32BIT settings allow raising some datatype limits to suit larger messages.
-Their need is recognized automatically by C-preprocessor #if-directives in the generated .pb.h files. The default setting is to use
-the smallest datatypes (least resources used).
 
 pb.h
 ====
@@ -37,22 +200,23 @@ pb_type_t
 ---------
 Defines the encoder/decoder behaviour that should be used for a field. ::
 
-    typedef enum { ... } pb_type_t;
+    typedef uint8_t pb_type_t;
 
-The low-order byte of the enumeration values defines the function that can be used for encoding and decoding the field data:
+The low-order nibble of the enumeration values defines the function that can be used for encoding and decoding the field data:
 
 ==================== ===== ================================================
 LTYPE identifier     Value Storage format
 ==================== ===== ================================================
 PB_LTYPE_VARINT      0x00  Integer.
 PB_LTYPE_SVARINT     0x01  Integer, zigzag encoded.
-PB_LTYPE_FIXED       0x02  Integer or floating point.
+PB_LTYPE_FIXED32     0x02  32-bit integer or floating point.
-PB_LTYPE_BYTES       0x03  Structure with *size_t* field and byte array.
+PB_LTYPE_FIXED64     0x03  64-bit integer or floating point.
-PB_LTYPE_STRING      0x04  Null-terminated string.
+PB_LTYPE_BYTES       0x04  Structure with *size_t* field and byte array.
-PB_LTYPE_SUBMESSAGE  0x05  Submessage structure.
+PB_LTYPE_STRING      0x05  Null-terminated string.
+PB_LTYPE_SUBMESSAGE  0x06  Submessage structure.
 ==================== ===== ================================================
 
-The high-order byte defines whether the field is required, optional, repeated or callback:
+The bits 4-5 define whether the field is required, optional or repeated:
 
 ==================== ===== ================================================
 HTYPE identifier     Value Field handling
@@ -60,13 +224,24 @@ HTYPE identifier     Value Field handling
 PB_HTYPE_REQUIRED    0x00  Verify that field exists in decoded message.
 PB_HTYPE_OPTIONAL    0x10  Use separate *has_<field>* boolean to specify
                            whether the field is present.
-PB_HTYPE_ARRAY       0x20  A repeated field with preallocated array.
+                           (Unless it is a callback)
+PB_HTYPE_REPEATED    0x20  A repeated field with preallocated array.
                            Separate *<field>_count* for number of items.
-PB_HTYPE_CALLBACK    0x30  A field with dynamic storage size, data is
+                           (Unless it is a callback)
-                           actually a pointer to a structure containing a
-                           callback function.
 ==================== ===== ================================================
 
+The bits 6-7 define the how the storage for the field is allocated:
+
+==================== ===== ================================================
+ATYPE identifier     Value Allocation method
+==================== ===== ================================================
+PB_ATYPE_STATIC      0x00  Statically allocated storage in the structure.
+PB_ATYPE_CALLBACK    0x40  A field with dynamic storage size. Struct field
+                           actually contains a pointer to a callback
+                           function.
+==================== ===== ================================================
+
+
 pb_field_t
 ----------
 Describes a single structure field with memory position in relation to others. The descriptions are usually autogenerated. ::
@@ -83,7 +258,7 @@ Describes a single structure field with memory position in relation to others. T
     } pb_packed;
 
 :tag:           Tag number of the field or 0 to terminate a list of fields.
-:type:          LTYPE and HTYPE of the field.
+:type:          LTYPE, HTYPE and ATYPE of the field.
 :data_offset:   Offset of field data, relative to the end of the previous field.
 :size_offset:   Offset of *bool* flag for optional fields or *size_t* count for arrays, relative to field data.
 :data_size:     Size of a single data entry, in bytes. For PB_LTYPE_BYTES, the size of the byte array inside the containing structure. For PB_HTYPE_CALLBACK, size of the C data type if known.
@@ -110,14 +285,16 @@ Part of a message structure, for fields with type PB_HTYPE_CALLBACK::
     typedef struct _pb_callback_t pb_callback_t;
     struct _pb_callback_t {
         union {
-            bool (*decode)(pb_istream_t *stream, const pb_field_t *field, void *arg);
+            bool (*decode)(pb_istream_t *stream, const pb_field_t *field, void **arg);
-            bool (*encode)(pb_ostream_t *stream, const pb_field_t *field, const void *arg);
+            bool (*encode)(pb_ostream_t *stream, const pb_field_t *field, void * const *arg);
         } funcs;
         
         void *arg;
     };
 
-The *arg* is passed to the callback when calling. It can be used to store any information that the callback might need.
+A pointer to the *arg* is passed to the callback when calling. It can be used to store any information that the callback might need.
+
+Previously the function received just the value of *arg* instead of a pointer to it. This old behaviour can be enabled by defining *PB_OLD_CALLBACK_STYLE*.
 
 When calling `pb_encode`_, *funcs.encode* is used, and similarly when calling `pb_decode`_, *funcs.decode* is used. The function pointers are stored in the same memory location but are of incompatible types. You can set the function pointer to NULL to skip the field.
 
@@ -132,6 +309,76 @@ Protocol Buffers wire types. These are used with `pb_encode_tag`_. ::
         PB_WT_32BIT  = 5
     } pb_wire_type_t;
 
+pb_extension_type_t
+-------------------
+Defines the handler functions and auxiliary data for a field that extends
+another message. Usually autogenerated by *nanopb_generator.py*::
+
+    typedef struct {
+        bool (*decode)(pb_istream_t *stream, pb_extension_t *extension,
+                   uint32_t tag, pb_wire_type_t wire_type);
+        bool (*encode)(pb_ostream_t *stream, const pb_extension_t *extension);
+        const void *arg;
+    } pb_extension_type_t;
+
+In the normal case, the function pointers are *NULL* and the decoder and
+encoder use their internal implementations. The internal implementations
+assume that *arg* points to a *pb_field_t* that describes the field in question.
+
+To implement custom processing of unknown fields, you can provide pointers
+to your own functions. Their functionality is mostly the same as for normal
+callback fields, except that they get called for any unknown field when decoding.
+
+pb_extension_t
+--------------
+Ties together the extension field type and the storage for the field value::
+
+    typedef struct {
+        const pb_extension_type_t *type;
+        void *dest;
+        pb_extension_t *next;
+    } pb_extension_t;
+
+:type:      Pointer to the structure that defines the callback functions.
+:dest:      Pointer to the variable that stores the field value
+            (as used by the default extension callback functions.)
+:next:      Pointer to the next extension handler, or *NULL*.
+
+PB_GET_ERROR
+------------
+Get the current error message from a stream, or a placeholder string if
+there is no error message::
+
+    #define PB_GET_ERROR(stream) (string expression)
+
+This should be used for printing errors, for example::
+
+    if (!pb_decode(...))
+    {
+        printf("Decode failed: %s\n", PB_GET_ERROR(stream));
+    }
+
+The macro only returns pointers to constant strings (in code memory),
+so that there is no need to release the returned pointer.
+
+PB_RETURN_ERROR
+---------------
+Set the error message and return false::
+
+    #define PB_RETURN_ERROR(stream,msg) (sets error and returns false)
+
+This should be used to handle error conditions inside nanopb functions
+and user callback functions::
+
+    if (error_condition)
+    {
+        PB_RETURN_ERROR(stream, "something went wrong");
+    }
+
+The *msg* parameter must be a constant string.
+
+
+
 pb_encode.h
 ===========
 
@@ -173,6 +420,17 @@ Encodes the contents of a structure as a protocol buffers message and writes it
 
 Normally pb_encode simply walks through the fields description array and serializes each field in turn. However, submessages must be serialized twice: first to calculate their size and then to actually write them to output. This causes some constraints for callback fields, which must return the same data on every call.
 
+pb_encode_delimited
+-------------------
+Calculates the length of the message, encodes it as varint and then encodes the message. ::
+
+    bool pb_encode_delimited(pb_ostream_t *stream, const pb_field_t fields[], const void *src_struct);
+
+(parameters are the same as for `pb_encode`_.)
+
+A common way to indicate the message length in Protocol Buffers is to prefix it with a varint.
+This function does this, and it is compatible with *parseDelimitedFrom* in Google's protobuf library.
+
 .. sidebar:: Encoding fields manually
 
     The functions with names *pb_encode_\** are used when dealing with callback fields. The typical reason for using callbacks is to have an array of unlimited size. In that case, `pb_encode`_ will call your callback function, which in turn will call *pb_encode_\** functions repeatedly to write out values.
@@ -281,6 +539,17 @@ In Protocol Buffers format, the submessage size must be written before the subme
 
 If the submessage contains callback fields, the callback function might misbehave and write out a different amount of data on the second call. This situation is recognized and *false* is returned, but garbage will be written to the output before the problem is detected.
 
+
+
+
+
+
+
+
+
+
+
+
 pb_decode.h
 ===========
 
@@ -324,6 +593,10 @@ In addition to EOF, the pb_decode implementation supports terminating a message
 
 For optional fields, this function applies the default value and sets *has_<field>* to false if the field is not present.
 
+If *PB_ENABLE_MALLOC* is defined, this function may allocate storage for any pointer type fields.
+In this case, you have to call `pb_release`_ to release the memory after you are done with the message.
+On error return `pb_decode` will release the memory itself.
+
 pb_decode_noinit
 ----------------
 Same as `pb_decode`_, except does not apply the default values to fields. ::
@@ -334,6 +607,35 @@ Same as `pb_decode`_, except does not apply the default values to fields. ::
 
 The destination structure should be filled with zeros before calling this function. Doing a *memset* manually can be slightly faster than using `pb_decode`_ if you don't need any default values.
 
+In addition to decoding a single message, this function can be used to merge two messages, so that
+values from previous message will remain if the new message does not contain a field.
+
+This function *will not* release the message even on error return. If you use *PB_ENABLE_MALLOC*,
+you will need to call `pb_release`_ yourself.
+
+pb_decode_delimited
+-------------------
+Same as `pb_decode`_, except that it first reads a varint with the length of the message. ::
+
+    bool pb_decode_delimited(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct);
+
+(parameters are the same as for `pb_decode`_.)
+
+A common method to indicate message size in Protocol Buffers is to prefix it with a varint.
+This function is compatible with *writeDelimitedTo* in the Google's Protocol Buffers library.
+
+pb_release
+----------
+Releases any dynamically allocated fields.
+
+    void pb_release(const pb_field_t fields[], void *dest_struct);
+
+:fields:        A field description array. Usually autogenerated.
+:dest_struct:   Pointer to structure where data will be stored.
+
+This function is only available if *PB_ENABLE_MALLOC* is defined. It will release any
+pointer type fields in the structure and set the pointers to NULL.
+
 pb_skip_varint
 --------------
 Skip a varint_ encoded integer without decoding it. ::

docs/security.rst

@@ -0,0 +1,79 @@
+======================
+Nanopb: Security model
+======================
+
+.. include :: menu.rst
+
+.. contents ::
+
+
+
+Importance of security in a Protocol Buffers library
+====================================================
+In the context of protocol buffers, security comes into play when decoding
+untrusted data. Naturally, if the attacker can modify the contents of a
+protocol buffers message, he can feed the application any values possible.
+Therefore the application itself must be prepared to receive untrusted values.
+
+Where nanopb plays a part is preventing the attacker from running arbitrary
+code on the target system. Mostly this means that there must not be any
+possibility to cause buffer overruns, memory corruption or invalid pointers
+by the means of crafting a malicious message.
+
+Division of trusted and untrusted data
+======================================
+The following data is regarded as **trusted**. It must be under the control of
+the application writer. Malicious data in these structures could cause
+security issues, such as execution of arbitrary code:
+
+1. Callback and extension fields in message structures given to pb_encode()
+   and pb_decode(). These fields are memory pointers, and are generated
+   depending on the .proto file.
+2. The automatically generated field definitions, i.e. *pb_field_t* lists.
+3. Contents of the *pb_istream_t* and *pb_ostream_t* structures (this does not
+   mean the contents of the stream itself, just the stream definition).
+
+The following data is regarded as **untrusted**. Invalid/malicious data in
+these will cause "garbage in, garbage out" behaviour. It will not cause
+buffer overflows, information disclosure or other security problems:
+
+1. All data read from *pb_istream_t*.
+2. All fields in message structures, except callbacks and extensions.
+   (Beginning with nanopb-0.2.4, in earlier versions the field sizes are partially unchecked.)
+
+Invariants
+==========
+The following invariants are maintained during operation, even if the
+untrusted data has been maliciously crafted:
+
+1. Nanopb will never read more than *bytes_left* bytes from *pb_istream_t*.
+2. Nanopb will never write more than *max_size* bytes to *pb_ostream_t*.
+3. Nanopb will never access memory out of bounds of the message structure.
+4. After pb_decode() returns successfully, the message structure will be
+   internally consistent:
+
+   - The *count* fields of arrays will not exceed the array size.
+   - The *size* field of bytes will not exceed the allocated size.
+   - All string fields will have null terminator.
+
+5. After pb_encode() returns successfully, the resulting message is a valid
+   protocol buffers message. (Except if user-defined callbacks write incorrect
+   data.)
+
+Further considerations
+======================
+Even if the nanopb library is free of any security issues, there are still
+several possible attack vectors that the application author must consider.
+The following list is not comprehensive:
+
+1. Stack usage may depend on the contents of the message. The message
+   definition places an upper bound on how much stack will be used. Tests
+   should be run with all fields present, to record the maximum possible
+   stack usage.
+2. Callbacks can do anything. The code for the callbacks must be carefully
+   checked if they are used with untrusted data.
+3. If using stream input, a maximum size should be set in *pb_istream_t* to
+   stop a denial of service attack from using an infinite message.
+4. If using network sockets as streams, a timeout should be set to stop
+   denial of service attacks.
+

example/Makefile

@@ -1,14 +0,0 @@
-CFLAGS=-ansi -Wall -Werror -I .. -g -O0
-DEPS=../pb_decode.c ../pb_decode.h ../pb_encode.c ../pb_encode.h ../pb.h
-
-all: server client
-
-clean:
-	rm -f server client fileproto.pb.c fileproto.pb.h
-
-%: %.c $(DEPS) fileproto.pb.h fileproto.pb.c
-	$(CC) $(CFLAGS) -o $@ $< ../pb_decode.c ../pb_encode.c fileproto.pb.c common.c
-
-fileproto.pb.c fileproto.pb.h: fileproto.proto ../generator/nanopb_generator.py
-	protoc -I. -I../generator -I/usr/include -ofileproto.pb $<
-	python ../generator/nanopb_generator.py fileproto.pb

example/fileproto.proto

@@ -1,26 +0,0 @@
-import "nanopb.proto";
-
-// This defines protocol for a simple server that lists files.
-//
-// If you come from high-level programming background, the hardcoded
-// maximum lengths may disgust you. However, if your microcontroller only
-// has a few kB of ram to begin with, setting reasonable limits for
-// filenames is ok.
-//
-// On the other hand, using the callback interface, it is not necessary
-// to set a limit on the number of files in the response.
-
-message ListFilesRequest {
-    optional string path = 1 [default = "/", (nanopb).max_size = 128];
-}
-
-message FileInfo {
-    required uint64 inode = 1;
-    required string name = 2 [(nanopb).max_size = 128];
-}
-
-message ListFilesResponse {
-    optional bool path_error = 1 [default = false];
-    repeated FileInfo file = 2;
-}
-

example_avr_double/Makefile

@@ -1,22 +0,0 @@
-CFLAGS=-Wall -Werror -I .. -g -O0
-DEPS=double_conversion.c ../pb_decode.c ../pb_decode.h ../pb_encode.c ../pb_encode.h ../pb.h
-
-all: run_tests
-
-clean:
-	rm -f test_conversions encode_double decode_double doubleproto.pb.c doubleproto.pb.h
-
-test_conversions: test_conversions.c double_conversion.c
-	$(CC) $(CFLAGS) -o $@ $^
-
-%: %.c $(DEPS) doubleproto.pb.h doubleproto.pb.c
-	$(CC) $(CFLAGS) -o $@ $< double_conversion.c ../pb_decode.c ../pb_encode.c doubleproto.pb.c
-
-doubleproto.pb.c doubleproto.pb.h: doubleproto.proto ../generator/nanopb_generator.py
-	protoc -I. -I../generator -I/usr/include -odoubleproto.pb $<
-	python ../generator/nanopb_generator.py doubleproto.pb
-
-run_tests: test_conversions encode_double decode_double
-	./test_conversions
-	./encode_double | ./decode_double
-    

example_unions/Makefile

@@ -1,17 +0,0 @@
-CFLAGS=-ansi -Wall -Werror -I .. -g -O0
-DEPS=../pb_decode.c ../pb_decode.h ../pb_encode.c ../pb_encode.h ../pb.h
-
-all: encode decode
-	./encode 1 | ./decode
-	./encode 2 | ./decode
-	./encode 3 | ./decode
-
-clean:
-	rm -f encode unionproto.pb.h unionproto.pb.c
-
-%: %.c $(DEPS) unionproto.pb.h unionproto.pb.c
-	$(CC) $(CFLAGS) -o $@ $< ../pb_decode.c ../pb_encode.c unionproto.pb.c
-
-unionproto.pb.h unionproto.pb.c: unionproto.proto ../generator/nanopb_generator.py
-	protoc -I. -I../generator -I/usr/include -ounionproto.pb $<
-	python ../generator/nanopb_generator.py unionproto.pb

examples/network_server/Makefile

@@ -0,0 +1,17 @@
+# Include the nanopb provided Makefile rules
+include ../../extra/nanopb.mk
+
+# Compiler flags to enable all warnings & debug info
+CFLAGS = -ansi -Wall -Werror -g -O0
+CFLAGS += -I$(NANOPB_DIR)
+
+all: server client
+
+.SUFFIXES:
+
+clean:
+	rm -f server client fileproto.pb.c fileproto.pb.h
+
+%: %.c common.c fileproto.pb.c
+	$(CC) $(CFLAGS) -o $@ $^ $(NANOPB_CORE)
+

examples/network_server/README.txt

@@ -0,0 +1,60 @@
+Nanopb example "network_server"
+===============================
+
+This example demonstrates the use of nanopb to communicate over network
+connections. It consists of a server that sends file listings, and of
+a client that requests the file list from the server.
+
+Example usage
+-------------
+
+user@host:~/nanopb/examples/network_server$ make        # Build the example
+protoc -ofileproto.pb fileproto.proto
+python ../../generator/nanopb_generator.py fileproto.pb
+Writing to fileproto.pb.h and fileproto.pb.c
+cc -ansi -Wall -Werror -I .. -g -O0 -I../.. -o server server.c
+    ../../pb_decode.c ../../pb_encode.c fileproto.pb.c common.c
+cc -ansi -Wall -Werror -I .. -g -O0 -I../.. -o client client.c
+    ../../pb_decode.c ../../pb_encode.c fileproto.pb.c common.c
+
+user@host:~/nanopb/examples/network_server$ ./server &  # Start the server on background
+[1] 24462
+
+petteri@oddish:~/nanopb/examples/network_server$ ./client /bin  # Request the server to list /bin
+Got connection.
+Listing directory: /bin
+1327119    bzdiff
+1327126    bzless
+1327147    ps
+1327178    ntfsmove
+1327271    mv
+1327187    mount
+1327259    false
+1327266    tempfile
+1327285    zfgrep
+1327165    gzexe
+1327204    nc.openbsd
+1327260    uname
+
+
+Details of implementation
+-------------------------
+fileproto.proto contains the portable Google Protocol Buffers protocol definition.
+It could be used as-is to implement a server or a client in any other language, for
+example Python or Java.
+
+fileproto.options contains the nanopb-specific options for the protocol file. This
+sets the amount of space allocated for file names when decoding messages.
+
+common.c/h contains functions that allow nanopb to read and write directly from
+network socket. This way there is no need to allocate a separate buffer to store
+the message.
+
+server.c contains the code to open a listening socket, to respond to clients and
+to list directory contents.
+
+client.c contains the code to connect to a server, to send a request and to print
+the response message.
+
+The code is implemented using the POSIX socket api, but it should be easy enough
+to port into any other socket api, such as lwip.

examples/network_server/client.c

@@ -23,7 +23,7 @@
 #include "fileproto.pb.h"
 #include "common.h"
 
-bool printfile_callback(pb_istream_t *stream, const pb_field_t *field, void *arg)
+bool printfile_callback(pb_istream_t *stream, const pb_field_t *field, void **arg)
 {
     FileInfo fileinfo;
     

examples/network_server/fileproto.options

@@ -0,0 +1,13 @@
+# This file defines the nanopb-specific options for the messages defined
+# in fileproto.proto.
+#
+# If you come from high-level programming background, the hardcoded
+# maximum lengths may disgust you. However, if your microcontroller only
+# has a few kB of ram to begin with, setting reasonable limits for
+# filenames is ok.
+#
+# On the other hand, using the callback interface, it is not necessary
+# to set a limit on the number of files in the response.
+
+ListFilesRequest.path 	max_size:128
+FileInfo.name 		max_size:128

examples/network_server/fileproto.proto

@@ -0,0 +1,18 @@
+// This defines protocol for a simple server that lists files.
+//
+// See also the nanopb-specific options in fileproto.options.
+
+message ListFilesRequest {
+    optional string path = 1 [default = "/"];
+}
+
+message FileInfo {
+    required uint64 inode = 1;
+    required string name = 2;
+}
+
+message ListFilesResponse {
+    optional bool path_error = 1 [default = false];
+    repeated FileInfo file = 2;
+}
+

examples/network_server/server.c

@@ -23,9 +23,9 @@
 #include "fileproto.pb.h"
 #include "common.h"
 
-bool listdir_callback(pb_ostream_t *stream, const pb_field_t *field, const void *arg)
+bool listdir_callback(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
 {
-    DIR *dir = (DIR*) arg;
+    DIR *dir = (DIR*) *arg;
     struct dirent *file;
     FileInfo fileinfo;
     

examples/simple/Makefile

@@ -0,0 +1,21 @@
+# Include the nanopb provided Makefile rules
+include ../../extra/nanopb.mk
+
+# Compiler flags to enable all warnings & debug info
+CFLAGS = -Wall -Werror -g -O0
+CFLAGS += -I$(NANOPB_DIR)
+
+# C source code files that are required
+CSRC  = simple.c                   # The main program
+CSRC += simple.pb.c                # The compiled protocol definition
+CSRC += $(NANOPB_DIR)/pb_encode.c  # The nanopb encoder
+CSRC += $(NANOPB_DIR)/pb_decode.c  # The nanopb decoder
+
+# Build rule for the main program
+simple: $(CSRC)
+	$(CC) $(CFLAGS) -osimple $(CSRC)
+
+# Build rule for the protocol
+simple.pb.c: simple.proto
+	$(PROTOC) $(PROTOC_OPTS) --nanopb_out=. simple.proto
+

examples/simple/README.txt

@@ -0,0 +1,29 @@
+Nanopb example "simple"
+=======================
+
+This example demonstrates the very basic use of nanopb. It encodes and
+decodes a simple message.
+
+The code uses four different API functions:
+
+  * pb_ostream_from_buffer() to declare the output buffer that is to be used
+  * pb_encode() to encode a message
+  * pb_istream_from_buffer() to declare the input buffer that is to be used
+  * pb_decode() to decode a message
+
+Example usage
+-------------
+
+On Linux, simply type "make" to build the example. After that, you can
+run it with the command: ./simple
+
+On other platforms, you first have to compile the protocol definition using
+the following command::
+
+  ../../generator-bin/protoc --nanopb_out=. simple.proto
+
+After that, add the following four files to your project and compile:
+
+  simple.c  simple.pb.c  pb_encode.c  pb_decode.c
+
+

examples/simple/simple.c

@@ -0,0 +1,68 @@
+#include <stdio.h>
+#include <pb_encode.h>
+#include <pb_decode.h>
+#include "simple.pb.h"
+
+int main()
+{
+    /* This is the buffer where we will store our message. */
+    uint8_t buffer[128];
+    size_t message_length;
+    bool status;
+    
+    /* Encode our message */
+    {
+        /* Allocate space on the stack to store the message data.
+         *
+         * Nanopb generates simple struct definitions for all the messages.
+         * - check out the contents of simple.pb.h! */
+        SimpleMessage message;
+        
+        /* Create a stream that will write to our buffer. */
+        pb_ostream_t stream = pb_ostream_from_buffer(buffer, sizeof(buffer));
+        
+        /* Fill in the lucky number */
+        message.lucky_number = 13;
+        
+        /* Now we are ready to encode the message! */
+        status = pb_encode(&stream, SimpleMessage_fields, &message);
+        message_length = stream.bytes_written;
+        
+        /* Then just check for any errors.. */
+        if (!status)
+        {
+            printf("Encoding failed: %s\n", PB_GET_ERROR(&stream));
+            return 1;
+        }
+    }
+    
+    /* Now we could transmit the message over network, store it in a file or
+     * wrap it to a pigeon's leg.
+     */
+
+    /* But because we are lazy, we will just decode it immediately. */
+    
+    {
+        /* Allocate space for the decoded message. */
+        SimpleMessage message;
+        
+        /* Create a stream that reads from the buffer. */
+        pb_istream_t stream = pb_istream_from_buffer(buffer, message_length);
+        
+        /* Now we are ready to decode the message. */
+        status = pb_decode(&stream, SimpleMessage_fields, &message);
+        
+        /* Check for errors... */
+        if (!status)
+        {
+            printf("Decoding failed: %s\n", PB_GET_ERROR(&stream));
+            return 1;
+        }
+        
+        /* Print the data contained in the message. */
+        printf("Your lucky number was %d!\n", message.lucky_number);
+    }
+    
+    return 0;
+}
+

examples/simple/simple.proto

@@ -0,0 +1,7 @@
+// A very simple protocol definition, consisting of only
+// one message.
+
+message SimpleMessage {
+    required int32 lucky_number = 1;
+}
+

examples/using_double_on_avr/Makefile

@@ -0,0 +1,24 @@
+# Include the nanopb provided Makefile rules
+include ../../extra/nanopb.mk
+
+# Compiler flags to enable all warnings & debug info
+CFLAGS = -Wall -Werror -g -O0
+CFLAGS += -I$(NANOPB_DIR)
+
+all: run_tests
+
+.SUFFIXES:
+
+clean:
+	rm -f test_conversions encode_double decode_double doubleproto.pb.c doubleproto.pb.h
+
+test_conversions: test_conversions.c double_conversion.c
+	$(CC) $(CFLAGS) -o $@ $^
+
+%: %.c double_conversion.c doubleproto.pb.c
+	$(CC) $(CFLAGS) -o $@ $^ $(NANOPB_CORE)
+
+run_tests: test_conversions encode_double decode_double
+	./test_conversions
+	./encode_double | ./decode_double
+    

examples/using_double_on_avr/README.txt

@@ -1,3 +1,6 @@
+Nanopb example "using_double_on_avr"
+====================================
+
 Some processors/compilers, such as AVR-GCC, do not support the double
 datatype. Instead, they have sizeof(double) == 4. Because protocol
 binary format uses the double encoding directly, this causes trouble
@@ -9,7 +12,7 @@ platforms. The file double_conversion.c provides functions that
 convert these values to/from floats, without relying on compiler
 support.
 
-To use this method, you need to make two modifications to your code:
+To use this method, you need to make some modifications to your code:
 
 1) Change all 'double' fields into 'fixed64' in the .proto.
 
@@ -17,6 +20,6 @@ To use this method, you need to make two modifications to your code:
 
 3) Whenever reading a 'double' field, use double_to_float().
 
-The conversion routines should be as accurate as the float datatype can
+The conversion routines are as accurate as the float datatype can
 be. Furthermore, they should handle all special values (NaN, inf, denormalized
 numbers) correctly. There are testcases in test_conversions.c.

examples/using_union_messages/Makefile

@@ -0,0 +1,20 @@
+# Include the nanopb provided Makefile rules
+include ../../extra/nanopb.mk
+
+# Compiler flags to enable all warnings & debug info
+CFLAGS = -ansi -Wall -Werror -g -O0
+CFLAGS += -I$(NANOPB_DIR)
+
+all: encode decode
+	./encode 1 | ./decode
+	./encode 2 | ./decode
+	./encode 3 | ./decode
+
+.SUFFIXES:
+
+clean:
+	rm -f encode unionproto.pb.h unionproto.pb.c
+
+%: %.c unionproto.pb.c
+	$(CC) $(CFLAGS) -o $@ $^ $(NANOPB_CORE)
+

examples/using_union_messages/README.txt

@@ -0,0 +1,52 @@
+Nanopb example "using_union_messages"
+=====================================
+
+Union messages is a common technique in Google Protocol Buffers used to
+represent a group of messages, only one of which is passed at a time.
+It is described in Google's documentation:
+https://developers.google.com/protocol-buffers/docs/techniques#union
+
+This directory contains an example on how to encode and decode union messages
+with minimal memory usage. Usually, nanopb would allocate space to store
+all of the possible messages at the same time, even though at most one of
+them will be used at a time.
+
+By using some of the lower level nanopb APIs, we can manually generate the
+top level message, so that we only need to allocate the one submessage that
+we actually want. Similarly when decoding, we can manually read the tag of
+the top level message, and only then allocate the memory for the submessage
+after we already know its type.
+
+
+Example usage
+-------------
+
+Type `make` to run the example. It will build it and run commands like
+following:
+
+./encode 1 | ./decode
+Got MsgType1: 42
+./encode 2 | ./decode
+Got MsgType2: true
+./encode 3 | ./decode
+Got MsgType3: 3 1415
+
+This simply demonstrates that the "decode" program has correctly identified
+the type of the received message, and managed to decode it.
+
+
+Details of implementation
+-------------------------
+
+unionproto.proto contains the protocol used in the example. It consists of
+three messages: MsgType1, MsgType2 and MsgType3, which are collected together
+into UnionMessage.
+
+encode.c takes one command line argument, which should be a number 1-3. It
+then fills in and encodes the corresponding message, and writes it to stdout.
+
+decode.c reads a UnionMessage from stdin. Then it calls the function
+decode_unionmessage_type() to determine the type of the message. After that,
+the corresponding message is decoded and the contents of it printed to the
+screen.
+

extra/FindNanopb.cmake

@@ -0,0 +1,225 @@
+# This is an example script for use with CMake projects for locating and configuring
+# the nanopb library.
+#
+# The following varialbes have to be set:
+#
+#   NANOPB_SRC_ROOT_FOLDER  - Path to nanopb source folder
+#
+# The following variables can be set and are optional:
+#
+#
+#   PROTOBUF_SRC_ROOT_FOLDER - When compiling with MSVC, if this cache variable is set
+#                              the protobuf-default VS project build locations
+#                              (vsprojects/Debug & vsprojects/Release) will be searched
+#                              for libraries and binaries.
+#
+#   NANOPB_IMPORT_DIRS       - List of additional directories to be searched for
+#                              imported .proto files.
+#
+#   NANOPB_GENERATE_CPP_APPEND_PATH - By default -I will be passed to protoc
+#                                     for each directory where a proto file is referenced.
+#                                     Set to FALSE if you want to disable this behaviour.
+#
+# Defines the following variables:
+#
+#   NANOPB_FOUND - Found the nanopb library (source&header files, generator tool, protoc compiler tool)
+#   NANOPB_INCLUDE_DIRS - Include directories for Google Protocol Buffers
+#
+# The following cache variables are also available to set or use:
+#   NANOPB_GENERATOR_EXECUTABLE - The nanopb generator
+#   PROTOBUF_PROTOC_EXECUTABLE - The protoc compiler
+#
+#  ====================================================================
+#
+# NANOPB_GENERATE_CPP (public function)
+#   SRCS = Variable to define with autogenerated
+#          source files
+#   HDRS = Variable to define with autogenerated
+#          header files
+#   ARGN = proto files
+#
+#  ====================================================================
+#  Example:
+#
+#   set(NANOPB_SRC_ROOT_FOLDER "/path/to/nanopb")
+#   set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} ${NANOPB_SRC_ROOT_FOLDER}/cmake)
+#   find_package( Nanopb REQUIRED )
+#   include_directories(${NANOPB_INCLUDE_DIRS})
+#
+#   NANOPB_GENERATE_CPP(PROTO_SRCS PROTO_HDRS foo.proto)
+#
+#   include_directories(${CMAKE_CURRENT_BINARY_DIR})
+#   add_executable(bar bar.cc ${PROTO_SRCS} ${PROTO_HDRS})
+#
+#  ====================================================================
+
+#=============================================================================
+# Copyright 2009 Kitware, Inc.
+# Copyright 2009-2011 Philip Lowman <philip@yhbt.com>
+# Copyright 2008 Esben Mose Hansen, Ange Optimization ApS
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+# * Redistributions of source code must retain the above copyright
+#   notice, this list of conditions and the following disclaimer.
+#
+# * Redistributions in binary form must reproduce the above copyright
+#   notice, this list of conditions and the following disclaimer in the
+#   documentation and/or other materials provided with the distribution.
+#
+# * Neither the names of Kitware, Inc., the Insight Software Consortium,
+#   nor the names of their contributors may be used to endorse or promote
+#   products derived from this software without specific prior written
+#   permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+#
+#=============================================================================
+#
+# Changes
+# 2013.01.31 - Pavlo Ilin - used Modules/FindProtobuf.cmake from cmake 2.8.10 to
+#                           write FindNanopb.cmake
+#
+#=============================================================================
+
+
+function(NANOPB_GENERATE_CPP SRCS HDRS)
+  if(NOT ARGN)
+    return()
+  endif()
+
+  if(NANOPB_GENERATE_CPP_APPEND_PATH)
+    # Create an include path for each file specified
+    foreach(FIL ${ARGN})
+      get_filename_component(ABS_FIL ${FIL} ABSOLUTE)
+      get_filename_component(ABS_PATH ${ABS_FIL} PATH)
+
+      list(FIND _nanobp_include_path ${ABS_PATH} _contains_already)
+      if(${_contains_already} EQUAL -1)
+          list(APPEND _nanobp_include_path -I ${ABS_PATH})
+      endif()
+    endforeach()
+  else()
+    set(_nanobp_include_path -I ${CMAKE_CURRENT_SOURCE_DIR})
+  endif()
+
+  if(DEFINED NANOPB_IMPORT_DIRS)
+    foreach(DIR ${NANOPB_IMPORT_DIRS})
+      get_filename_component(ABS_PATH ${DIR} ABSOLUTE)
+      list(FIND _nanobp_include_path ${ABS_PATH} _contains_already)
+      if(${_contains_already} EQUAL -1)
+          list(APPEND _nanobp_include_path -I ${ABS_PATH})
+      endif()
+    endforeach()
+  endif()
+
+  set(${SRCS})
+  set(${HDRS})
+  get_filename_component(GENERATOR_PATH ${NANOPB_GENERATOR_EXECUTABLE} PATH)
+
+  foreach(FIL ${ARGN})
+    get_filename_component(ABS_FIL ${FIL} ABSOLUTE)
+    get_filename_component(FIL_WE ${FIL} NAME_WE)
+
+    list(APPEND ${SRCS} "${CMAKE_CURRENT_BINARY_DIR}/${FIL_WE}.pb.c")
+    list(APPEND ${HDRS} "${CMAKE_CURRENT_BINARY_DIR}/${FIL_WE}.pb.h")
+
+    add_custom_command(
+      OUTPUT "${CMAKE_CURRENT_BINARY_DIR}/${FIL_WE}.pb"
+      COMMAND  ${PROTOBUF_PROTOC_EXECUTABLE}
+      ARGS -I${GENERATOR_PATH} -I${CMAKE_CURRENT_BINARY_DIR} ${_nanobp_include_path} -o${FIL_WE}.pb ${ABS_FIL}
+      DEPENDS ${ABS_FIL}
+      COMMENT "Running C++ protocol buffer compiler on ${FIL}"
+      VERBATIM )
+
+    add_custom_command(
+      OUTPUT "${CMAKE_CURRENT_BINARY_DIR}/${FIL_WE}.pb.c"
+             "${CMAKE_CURRENT_BINARY_DIR}/${FIL_WE}.pb.h"
+      COMMAND python
+      ARGS ${NANOPB_GENERATOR_EXECUTABLE} ${FIL_WE}.pb
+      DEPENDS "${CMAKE_CURRENT_BINARY_DIR}/${FIL_WE}.pb"
+      COMMENT "Running nanopb generator on ${FIL_WE}.pb"
+      VERBATIM )
+  endforeach()
+
+  set_source_files_properties(${${SRCS}} ${${HDRS}} PROPERTIES GENERATED TRUE)
+  set(${SRCS} ${${SRCS}} ${NANOPB_SRCS} PARENT_SCOPE)
+  set(${HDRS} ${${HDRS}} ${NANOPB_HDRS} PARENT_SCOPE)
+
+endfunction()
+
+
+
+#
+# Main.
+#
+
+# By default have NANOPB_GENERATE_CPP macro pass -I to protoc
+# for each directory where a proto file is referenced.
+if(NOT DEFINED NANOPB_GENERATE_CPP_APPEND_PATH)
+  set(NANOPB_GENERATE_CPP_APPEND_PATH TRUE)
+endif()
+
+# Find the include directory
+find_path(NANOPB_INCLUDE_DIRS
+    pb.h
+    PATHS ${NANOPB_SRC_ROOT_FOLDER}
+)
+mark_as_advanced(NANOPB_INCLUDE_DIRS)
+
+# Find nanopb source files
+set(NANOPB_SRCS)
+set(NANOPB_HDRS)
+list(APPEND _nanopb_srcs pb_decode.c pb_encode.c)
+list(APPEND _nanopb_hdrs pb_decode.h pb_encode.h pb.h)
+
+foreach(FIL ${_nanopb_srcs})
+  find_file(${FIL}__nano_pb_file NAMES ${FIL} PATHS ${NANOPB_SRC_ROOT_FOLDER} ${NANOPB_INCLUDE_DIRS})
+  list(APPEND NANOPB_SRCS "${${FIL}__nano_pb_file}")
+  mark_as_advanced(${FIL}__nano_pb_file)
+endforeach()
+
+foreach(FIL ${_nanopb_hdrs})
+  find_file(${FIL}__nano_pb_file NAMES ${FIL} PATHS ${NANOPB_INCLUDE_DIRS})
+  mark_as_advanced(${FIL}__nano_pb_file)
+  list(APPEND NANOPB_HDRS "${${FIL}__nano_pb_file}")
+endforeach()
+
+# Find the protoc Executable
+find_program(PROTOBUF_PROTOC_EXECUTABLE
+    NAMES protoc
+    DOC "The Google Protocol Buffers Compiler"
+    PATHS
+    ${PROTOBUF_SRC_ROOT_FOLDER}/vsprojects/Release
+    ${PROTOBUF_SRC_ROOT_FOLDER}/vsprojects/Debug
+)
+mark_as_advanced(PROTOBUF_PROTOC_EXECUTABLE)
+
+# Find nanopb generator
+find_file(NANOPB_GENERATOR_EXECUTABLE
+    NAMES nanopb_generator.py
+    DOC "nanopb generator"
+    PATHS
+    ${NANOPB_SRC_ROOT_FOLDER}/generator
+)
+mark_as_advanced(NANOPB_GENERATOR_EXECUTABLE)
+
+include(FindPackageHandleStandardArgs)
+FIND_PACKAGE_HANDLE_STANDARD_ARGS(NANOPB DEFAULT_MSG
+  NANOPB_INCLUDE_DIRS
+  NANOPB_SRCS NANOPB_HDRS
+  NANOPB_GENERATOR_EXECUTABLE
+  PROTOBUF_PROTOC_EXECUTABLE
+  )

extra/nanopb.mk

@@ -0,0 +1,37 @@
+# This is an include file for Makefiles. It provides rules for building
+# .pb.c and .pb.h files out of .proto, as well the path to nanopb core.
+
+# Path to the nanopb root directory
+NANOPB_DIR := $(abspath $(dir $(lastword $(MAKEFILE_LIST)))../)
+
+# Files for the nanopb core
+NANOPB_CORE = $(NANOPB_DIR)/pb_encode.c $(NANOPB_DIR)/pb_decode.c
+
+# Check if we are running on Windows
+ifdef windir
+WINDOWS = 1
+endif
+ifdef WINDIR
+WINDOWS = 1
+endif
+
+# Check whether to use binary version of nanopb_generator or the
+# system-supplied python interpreter.
+ifneq "$(wildcard $(NANOPB_DIR)/generator-bin)" ""
+	# Binary package
+	PROTOC = $(NANOPB_DIR)/generator-bin/protoc
+	PROTOC_OPTS = 
+else
+	# Source only or git checkout
+	PROTOC = protoc
+	ifdef WINDOWS
+		PROTOC_OPTS = --plugin=protoc-gen-nanopb=$(NANOPB_DIR)/generator/protoc-gen-nanopb.bat
+	else
+		PROTOC_OPTS = --plugin=protoc-gen-nanopb=$(NANOPB_DIR)/generator/protoc-gen-nanopb
+	endif
+endif
+
+# Rule for building .pb.c and .pb.h
+%.pb.c %.pb.h: %.proto $(wildcard %.options)
+	$(PROTOC) $(PROTOC_OPTS) --nanopb_out=. $<
+

extra/pb_syshdr.h

@@ -0,0 +1,94 @@
+/* This is an example of a header file for platforms/compilers that do
+ * not come with stdint.h/stddef.h/stdbool.h/string.h. To use it, define
+ * PB_SYSTEM_HEADER as "pb_syshdr.h", including the quotes, and add the
+ * extra folder to your include path.
+ *
+ * It is very likely that you will need to customize this file to suit
+ * your platform. For any compiler that supports C99, this file should
+ * not be necessary.
+ */
+
+#ifndef _PB_SYSHDR_H_
+#define _PB_SYSHDR_H_
+
+/* stdint.h subset */
+#ifdef HAVE_STDINT_H
+#include <stdint.h>
+#else
+/* You will need to modify these to match the word size of your platform. */
+typedef signed char int8_t;
+typedef unsigned char uint8_t;
+typedef signed short int16_t;
+typedef unsigned short uint16_t;
+typedef signed int int32_t;
+typedef unsigned int uint32_t;
+typedef signed long long int64_t;
+typedef unsigned long long uint64_t;
+#endif
+
+/* stddef.h subset */
+#ifdef HAVE_STDDEF_H
+#include <stddef.h>
+#else
+
+typedef uint32_t size_t;
+#define offsetof(st, m) ((size_t)(&((st *)0)->m))
+
+#ifndef NULL
+#define NULL 0
+#endif
+
+#endif
+
+/* stdbool.h subset */
+#ifdef HAVE_STDBOOL_H
+#include <stdbool.h>
+#else
+
+#ifndef __cplusplus
+typedef int bool;
+#define false 0
+#define true 1
+#endif
+
+#endif
+
+/* string.h subset */
+#ifdef HAVE_STRING_H
+#include <string.h>
+#else
+
+/* Implementations are from the Public Domain C Library (PDCLib). */
+static size_t strlen( const char * s )
+{
+    size_t rc = 0;
+    while ( s[rc] )
+    {
+        ++rc;
+    }
+    return rc;
+}
+
+static void * memcpy( void *s1, const void *s2, size_t n )
+{
+    char * dest = (char *) s1;
+    const char * src = (const char *) s2;
+    while ( n-- )
+    {
+        *dest++ = *src++;
+    }
+    return s1;
+}
+
+static void * memset( void * s, int c, size_t n )
+{
+    unsigned char * p = (unsigned char *) s;
+    while ( n-- )
+    {
+        *p++ = (unsigned char) c;
+    }
+    return s;
+}
+#endif
+
+#endif

generator/Makefile

@@ -1,2 +0,0 @@
-nanopb_pb2.py: nanopb.proto
-	protoc --python_out=. -I /usr/include -I . nanopb.proto

generator/nanopb_generator.py

@@ -1,33 +1,42 @@
+#!/usr/bin/python
+
 '''Generate header file for nanopb from a ProtoBuf FileDescriptorSet.'''
-nanopb_version = "0.2.0-dev"
+nanopb_version = "nanopb-0.2.7-dev"
+
+import sys
 
 try:
+    # Add some dummy imports to keep packaging tools happy.
+    import google, distutils.util # bbfreeze seems to need these
+    import pkg_resources # pyinstaller / protobuf 2.5 seem to need these
+except:
+    # Don't care, we will error out later if it is actually important.
+    pass
+
+try:
+    import google.protobuf.text_format as text_format
     import google.protobuf.descriptor_pb2 as descriptor
 except:
-    print
+    sys.stderr.write('''
-    print "*************************************************************"
+         *************************************************************
-    print "*** Could not import the Google protobuf Python libraries ***"
+         *** Could not import the Google protobuf Python libraries ***
-    print "*** Try installing package 'python-protobuf' or similar.  ***"
+         *** Try installing package 'python-protobuf' or similar.  ***
-    print "*************************************************************"
+         *************************************************************
-    print
+    ''' + '\n')
     raise
 
 try:
-    import nanopb_pb2
+    import proto.nanopb_pb2 as nanopb_pb2
+    import proto.plugin_pb2 as plugin_pb2
 except:
-    print
+    sys.stderr.write('''
-    print "***************************************************************"
+         ********************************************************************
-    print "*** Could not import the precompiled nanopb_pb2.py.         ***"
+         *** Failed to import the protocol definitions for generator.     ***
-    print "*** Run 'make' in the 'generator' folder to update the file.***"
+         *** You have to run 'make' in the nanopb/generator/proto folder. ***
-    print "***************************************************************"
+         ********************************************************************
-    print
+    ''' + '\n')
     raise
 
-
-
-
-
-
 # ---------------------------------------------------------------------------
 #                     Generation of single fields
 # ---------------------------------------------------------------------------
@@ -35,28 +44,26 @@ except:
 import time
 import os.path
 
-# Values are tuple (c type, pb type)
+# Values are tuple (c type, pb type, encoded size)
 FieldD = descriptor.FieldDescriptorProto
 datatypes = {
-    FieldD.TYPE_BOOL:       ('bool',     'BOOL'),
+    FieldD.TYPE_BOOL:       ('bool',     'BOOL',        1),
-    FieldD.TYPE_DOUBLE:     ('double',   'DOUBLE'),
+    FieldD.TYPE_DOUBLE:     ('double',   'DOUBLE',      8),
-    FieldD.TYPE_FIXED32:    ('uint32_t', 'FIXED32'),
+    FieldD.TYPE_FIXED32:    ('uint32_t', 'FIXED32',     4),
-    FieldD.TYPE_FIXED64:    ('uint64_t', 'FIXED64'),
+    FieldD.TYPE_FIXED64:    ('uint64_t', 'FIXED64',     8),
-    FieldD.TYPE_FLOAT:      ('float',    'FLOAT'),
+    FieldD.TYPE_FLOAT:      ('float',    'FLOAT',       4),
-    FieldD.TYPE_INT32:      ('int32_t',  'INT32'),
+    FieldD.TYPE_INT32:      ('int32_t',  'INT32',      10),
-    FieldD.TYPE_INT64:      ('int64_t',  'INT64'),
+    FieldD.TYPE_INT64:      ('int64_t',  'INT64',      10),
-    FieldD.TYPE_SFIXED32:   ('int32_t',  'SFIXED32'),
+    FieldD.TYPE_SFIXED32:   ('int32_t',  'SFIXED32',    4),
-    FieldD.TYPE_SFIXED64:   ('int64_t',  'SFIXED64'),
+    FieldD.TYPE_SFIXED64:   ('int64_t',  'SFIXED64',    8),
-    FieldD.TYPE_SINT32:     ('int32_t',  'SINT32'),
+    FieldD.TYPE_SINT32:     ('int32_t',  'SINT32',      5),
-    FieldD.TYPE_SINT64:     ('int64_t',  'SINT64'),
+    FieldD.TYPE_SINT64:     ('int64_t',  'SINT64',     10),
-    FieldD.TYPE_UINT32:     ('uint32_t', 'UINT32'),
+    FieldD.TYPE_UINT32:     ('uint32_t', 'UINT32',      5),
-    FieldD.TYPE_UINT64:     ('uint64_t', 'UINT64')
+    FieldD.TYPE_UINT64:     ('uint64_t', 'UINT64',     10)
 }
 
 class Names:
-    '''Keeps a set of nested names and formats them to C identifier.
+    '''Keeps a set of nested names and formats them to C identifier.'''
-    You can subclass this with your own implementation.
-    '''
     def __init__(self, parts = ()):
         if isinstance(parts, Names):
             parts = parts.parts
@@ -82,6 +89,55 @@ def names_from_type_name(type_name):
         raise NotImplementedError("Lookup of non-absolute type names is not supported")
     return Names(type_name[1:].split('.'))
 
+def varint_max_size(max_value):
+    '''Returns the maximum number of bytes a varint can take when encoded.'''
+    for i in range(1, 11):
+        if (max_value >> (i * 7)) == 0:
+            return i
+    raise ValueError("Value too large for varint: " + str(max_value))
+
+assert varint_max_size(0) == 1
+assert varint_max_size(127) == 1
+assert varint_max_size(128) == 2
+
+class EncodedSize:
+    '''Class used to represent the encoded size of a field or a message.
+    Consists of a combination of symbolic sizes and integer sizes.'''
+    def __init__(self, value = 0, symbols = []):
+        if isinstance(value, (str, Names)):
+            symbols = [str(value)]
+            value = 0
+        self.value = value
+        self.symbols = symbols
+    
+    def __add__(self, other):
+        if isinstance(other, (int, long)):
+            return EncodedSize(self.value + other, self.symbols)
+        elif isinstance(other, (str, Names)):
+            return EncodedSize(self.value, self.symbols + [str(other)])
+        elif isinstance(other, EncodedSize):
+            return EncodedSize(self.value + other.value, self.symbols + other.symbols)
+        else:
+            raise ValueError("Cannot add size: " + repr(other))
+
+    def __mul__(self, other):
+        if isinstance(other, (int, long)):
+            return EncodedSize(self.value * other, [str(other) + '*' + s for s in self.symbols])
+        else:
+            raise ValueError("Cannot multiply size: " + repr(other))
+
+    def __str__(self):
+        if not self.symbols:
+            return str(self.value)
+        else:
+            return '(' + str(self.value) + ' + ' + ' + '.join(self.symbols) + ')'
+
+    def upperlimit(self):
+        if not self.symbols:
+            return self.value
+        else:
+            return 2**32 - 1
+
 class Enum:
     def __init__(self, names, desc, enum_options):
         '''desc is EnumDescriptorProto'''
@@ -112,6 +168,8 @@ class Field:
         self.max_size = None
         self.max_count = None
         self.array_decl = ""
+        self.enc_size = None
+        self.ctype = None
         
         # Parse field options
         if field_options.HasField("max_size"):
@@ -138,33 +196,15 @@ class Field:
         else:
             raise NotImplementedError(desc.label)
         
-        # Decide the C data type to use in the struct.
+        # Check if the field can be implemented with static allocation
-        if datatypes.has_key(desc.type):
+        # i.e. whether the data size is known.
-            self.ctype, self.pbtype = datatypes[desc.type]
+        if desc.type == FieldD.TYPE_STRING and self.max_size is None:
-        elif desc.type == FieldD.TYPE_ENUM:
-            self.pbtype = 'ENUM'
-            self.ctype = names_from_type_name(desc.type_name)
-            if self.default is not None:
-                self.default = self.ctype + self.default
-        elif desc.type == FieldD.TYPE_STRING:
-            self.pbtype = 'STRING'
-            if self.max_size is None:
             can_be_static = False
-            else:
-                self.ctype = 'char'
-                self.array_decl += '[%d]' % self.max_size
-        elif desc.type == FieldD.TYPE_BYTES:
-            self.pbtype = 'BYTES'
-            if self.max_size is None:
-                can_be_static = False
-            else:
-                self.ctype = self.struct_name + self.name + 't'
-        elif desc.type == FieldD.TYPE_MESSAGE:
-            self.pbtype = 'MESSAGE'
-            self.ctype = self.submsgname = names_from_type_name(desc.type_name)
-        else:
-            raise NotImplementedError(desc.type)
         
+        if desc.type == FieldD.TYPE_BYTES and self.max_size is None:
+            can_be_static = False
+        
+        # Decide how the field data will be allocated
         if field_options.type == nanopb_pb2.FT_DEFAULT:
             if can_be_static:
                 field_options.type = nanopb_pb2.FT_STATIC
@@ -172,27 +212,72 @@ class Field:
                 field_options.type = nanopb_pb2.FT_CALLBACK
         
         if field_options.type == nanopb_pb2.FT_STATIC and not can_be_static:
-            raise Exception("Field %s is defined as static, but max_size or max_count is not given." % self.name)
+            raise Exception("Field %s is defined as static, but max_size or "
+                            "max_count is not given." % self.name)
         
         if field_options.type == nanopb_pb2.FT_STATIC:
             self.allocation = 'STATIC'
+        elif field_options.type == nanopb_pb2.FT_POINTER:
+            self.allocation = 'POINTER'
         elif field_options.type == nanopb_pb2.FT_CALLBACK:
             self.allocation = 'CALLBACK'
-            self.ctype = 'pb_callback_t'
-            self.array_decl = ''
         else:
             raise NotImplementedError(field_options.type)
         
+        # Decide the C data type to use in the struct.
+        if datatypes.has_key(desc.type):
+            self.ctype, self.pbtype, self.enc_size = datatypes[desc.type]
+        elif desc.type == FieldD.TYPE_ENUM:
+            self.pbtype = 'ENUM'
+            self.ctype = names_from_type_name(desc.type_name)
+            if self.default is not None:
+                self.default = self.ctype + self.default
+            self.enc_size = 5 # protoc rejects enum values > 32 bits
+        elif desc.type == FieldD.TYPE_STRING:
+            self.pbtype = 'STRING'
+            self.ctype = 'char'
+            if self.allocation == 'STATIC':
+                self.ctype = 'char'
+                self.array_decl += '[%d]' % self.max_size
+                self.enc_size = varint_max_size(self.max_size) + self.max_size
+        elif desc.type == FieldD.TYPE_BYTES:
+            self.pbtype = 'BYTES'
+            if self.allocation == 'STATIC':
+                self.ctype = self.struct_name + self.name + 't'
+                self.enc_size = varint_max_size(self.max_size) + self.max_size
+            elif self.allocation == 'POINTER':
+                self.ctype = 'pb_bytes_array_t'
+        elif desc.type == FieldD.TYPE_MESSAGE:
+            self.pbtype = 'MESSAGE'
+            self.ctype = self.submsgname = names_from_type_name(desc.type_name)
+            self.enc_size = None # Needs to be filled in after the message type is available
+        else:
+            raise NotImplementedError(desc.type)
+        
     def __cmp__(self, other):
         return cmp(self.tag, other.tag)
     
     def __str__(self):
-        if self.rules == 'OPTIONAL':
-            result = '    bool has_' + self.name + ';\n'
-        elif self.rules == 'REPEATED' and self.allocation == 'STATIC':
-            result = '    size_t ' + self.name + '_count;\n'
-        else:
         result = ''
+        if self.allocation == 'POINTER':
+            if self.rules == 'REPEATED':
+                result += '    size_t ' + self.name + '_count;\n'
+            
+            if self.pbtype == 'MESSAGE':
+                # Use struct definition, so recursive submessages are possible
+                result += '    struct _%s *%s;' % (self.ctype, self.name)
+            elif self.rules == 'REPEATED' and self.pbtype in ['STRING', 'BYTES']:
+                # String/bytes arrays need to be defined as pointers to pointers
+                result += '    %s **%s;' % (self.ctype, self.name)
+            else:
+                result += '    %s *%s;' % (self.ctype, self.name)
+        elif self.allocation == 'CALLBACK':
+            result += '    pb_callback_t %s;' % self.name
+        else:
+            if self.rules == 'OPTIONAL' and self.allocation == 'STATIC':
+                result += '    bool has_' + self.name + ';\n'
+            elif self.rules == 'REPEATED' and self.allocation == 'STATIC':
+                result += '    size_t ' + self.name + '_count;\n'
             result += '    %s %s%s;' % (self.ctype, self.name, self.array_decl)
         return result
     
@@ -230,20 +315,32 @@ class Field:
             data = self.default.decode('string_escape')
             data = ['0x%02x' % ord(c) for c in data]
             default = '{%d, {%s}}' % (len(data), ','.join(data))
+        elif self.pbtype in ['FIXED32', 'UINT32']:
+            default += 'u'
+        elif self.pbtype in ['FIXED64', 'UINT64']:
+            default += 'ull'
+        elif self.pbtype in ['SFIXED64', 'INT64']:
+            default += 'll'
         
         if declaration_only:
             return 'extern const %s %s_default%s;' % (ctype, self.struct_name + self.name, array_decl)
         else:
             return 'const %s %s_default%s = %s;' % (ctype, self.struct_name + self.name, array_decl, default)
     
+    def tags(self):
+        '''Return the #define for the tag number of this field.'''
+        identifier = '%s_%s_tag' % (self.struct_name, self.name)
+        return '#define %-40s %d\n' % (identifier, self.tag)
+    
     def pb_field_t(self, prev_field_name):
         '''Return the pb_field_t initializer to use in the constant array.
         prev_field_name is the name of the previous field or None.
         '''
-        result  = '    PB_FIELD(%3d, ' % self.tag
+        result  = '    PB_FIELD2(%3d, ' % self.tag
         result += '%-8s, ' % self.pbtype
         result += '%s, ' % self.rules
-        result += '%s, ' % self.allocation
+        result += '%-8s, ' % self.allocation
+        result += '%s, ' % ("FIRST" if not prev_field_name else "OTHER")
         result += '%s, ' % self.struct_name
         result += '%s, ' % self.name
         result += '%s, ' % (prev_field_name or self.name)
@@ -270,9 +367,129 @@ class Field:
 
         return max(self.tag, self.max_size, self.max_count)        
 
+    def encoded_size(self, allmsgs):
+        '''Return the maximum size that this field can take when encoded,
+        including the field tag. If the size cannot be determined, returns
+        None.'''
+        
+        if self.allocation != 'STATIC':
+            return None
+        
+        if self.pbtype == 'MESSAGE':
+            for msg in allmsgs:
+                if msg.name == self.submsgname:
+                    encsize = msg.encoded_size(allmsgs)
+                    if encsize is None:
+                        return None # Submessage size is indeterminate
+                        
+                    # Include submessage length prefix
+                    encsize += varint_max_size(encsize.upperlimit())
+                    break
+            else:
+                # Submessage cannot be found, this currently occurs when
+                # the submessage type is defined in a different file.
+                # Instead of direct numeric value, reference the size that
+                # has been #defined in the other file.
+                encsize = EncodedSize(self.submsgname + 'size')
+
+                # We will have to make a conservative assumption on the length
+                # prefix size, though.
+                encsize += 5
+
+        elif self.enc_size is None:
+            raise RuntimeError("Could not determine encoded size for %s.%s"
+                               % (self.struct_name, self.name))
+        else:
+            encsize = EncodedSize(self.enc_size)
+        
+        encsize += varint_max_size(self.tag << 3) # Tag + wire type
+
+        if self.rules == 'REPEATED':
+            # Decoders must be always able to handle unpacked arrays.
+            # Therefore we have to reserve space for it, even though
+            # we emit packed arrays ourselves.
+            encsize *= self.max_count
+        
+        return encsize
 
 
+class ExtensionRange(Field):
+    def __init__(self, struct_name, range_start, field_options):
+        '''Implements a special pb_extension_t* field in an extensible message
+        structure. The range_start signifies the index at which the extensions
+        start. Not necessarily all tags above this are extensions, it is merely
+        a speed optimization.
+        '''
+        self.tag = range_start
+        self.struct_name = struct_name
+        self.name = 'extensions'
+        self.pbtype = 'EXTENSION'
+        self.rules = 'OPTIONAL'
+        self.allocation = 'CALLBACK'
+        self.ctype = 'pb_extension_t'
+        self.array_decl = ''
+        self.default = None
+        self.max_size = 0
+        self.max_count = 0
         
+    def __str__(self):
+        return '    pb_extension_t *extensions;'
+    
+    def types(self):
+        return None
+    
+    def tags(self):
+        return ''
+        
+    def encoded_size(self, allmsgs):
+        # We exclude extensions from the count, because they cannot be known
+        # until runtime. Other option would be to return None here, but this
+        # way the value remains useful if extensions are not used.
+        return EncodedSize(0)
+
+class ExtensionField(Field):
+    def __init__(self, struct_name, desc, field_options):
+        self.fullname = struct_name + desc.name
+        self.extendee_name = names_from_type_name(desc.extendee)
+        Field.__init__(self, self.fullname + 'struct', desc, field_options)
+        
+        if self.rules != 'OPTIONAL':
+            self.skip = True
+        else:
+            self.skip = False
+            self.rules = 'OPTEXT'
+
+    def tags(self):
+        '''Return the #define for the tag number of this field.'''
+        identifier = '%s_tag' % self.fullname
+        return '#define %-40s %d\n' % (identifier, self.tag)
+
+    def extension_decl(self):
+        '''Declaration of the extension type in the .pb.h file'''
+        if self.skip:
+            msg = '/* Extension field %s was skipped because only "optional"\n' % self.fullname
+            msg +='   type of extension fields is currently supported. */\n'
+            return msg
+        
+        return 'extern const pb_extension_type_t %s;\n' % self.fullname
+
+    def extension_def(self):
+        '''Definition of the extension type in the .pb.c file'''
+
+        if self.skip:
+            return ''
+
+        result  = 'typedef struct {\n'
+        result += str(self)
+        result += '\n} %s;\n\n' % self.struct_name
+        result += ('static const pb_field_t %s_field = \n  %s;\n\n' %
+                    (self.fullname, self.pb_field_t(None)))
+        result += 'const pb_extension_type_t %s = {\n' % self.fullname
+        result += '    NULL,\n'
+        result += '    NULL,\n'
+        result += '    &%s_field\n' % self.fullname
+        result += '};\n'
+        return result
 
 
 # ---------------------------------------------------------------------------
@@ -286,10 +503,16 @@ class Message:
         self.fields = []
         
         for f in desc.field:
-            field_options = get_nanopb_suboptions(f, message_options)
+            field_options = get_nanopb_suboptions(f, message_options, self.name + f.name)
             if field_options.type != nanopb_pb2.FT_IGNORE:
                 self.fields.append(Field(self.name, f, field_options))
         
+        if len(desc.extension_range) > 0:
+            field_options = get_nanopb_suboptions(desc, message_options, self.name + 'extensions')
+            range_start = min([r.start for r in desc.extension_range])
+            if field_options.type != nanopb_pb2.FT_IGNORE:
+                self.fields.append(ExtensionRange(self.name, range_start, field_options))
+        
         self.packed = message_options.packed_struct
         self.ordered_fields = self.fields[:]
         self.ordered_fields.sort()
@@ -300,6 +523,12 @@ class Message:
     
     def __str__(self):
         result = 'typedef struct _%s {\n' % self.name
+
+        if not self.ordered_fields:
+            # Empty structs are not allowed in C standard.
+            # Therefore add a dummy field if an empty message occurs.
+            result += '    uint8_t dummy_field;'
+
         result += '\n'.join([str(f) for f in self.ordered_fields])
         result += '\n}'
         
@@ -307,6 +536,11 @@ class Message:
             result += ' pb_packed'
         
         result += ' %s;' % self.name
+        
+        if self.packed:
+            result = 'PB_PACKED_STRUCT_START\n' + result
+            result += '\nPB_PACKED_STRUCT_END'
+        
         return result
     
     def types(self):
@@ -341,9 +575,18 @@ class Message:
         result += '    PB_LAST_FIELD\n};'
         return result
 
+    def encoded_size(self, allmsgs):
+        '''Return the maximum size that this message can take when encoded.
+        If the size cannot be determined, returns None.
+        '''
+        size = EncodedSize(0)
+        for field in self.fields:
+            fsize = field.encoded_size(allmsgs)
+            if fsize is None:
+                return None
+            size += fsize
         
-
+        return size
-
 
 
 # ---------------------------------------------------------------------------
@@ -365,11 +608,23 @@ def iterate_messages(desc, names = Names()):
         for x in iterate_messages(submsg, sub_names):
             yield x
 
+def iterate_extensions(desc, names = Names()):
+    '''Recursively find all extensions.
+    For each, yield name, FieldDescriptorProto.
+    '''
+    for extension in desc.extension:
+        yield names, extension
+
+    for subname, subdesc in iterate_messages(desc, names):
+        for extension in subdesc.extension:
+            yield subname, extension
+
 def parse_file(fdesc, file_options):
-    '''Takes a FileDescriptorProto and returns tuple (enum, messages).'''
+    '''Takes a FileDescriptorProto and returns tuple (enums, messages, extensions).'''
     
     enums = []
     messages = []
+    extensions = []
     
     if fdesc.package:
         base_name = Names(fdesc.package.split('.'))
@@ -377,16 +632,21 @@ def parse_file(fdesc, file_options):
         base_name = Names()
     
     for enum in fdesc.enum_type:
-        enum_options = get_nanopb_suboptions(enum, file_options)
+        enum_options = get_nanopb_suboptions(enum, file_options, base_name + enum.name)
         enums.append(Enum(base_name, enum, enum_options))
     
     for names, message in iterate_messages(fdesc, base_name):
-        message_options = get_nanopb_suboptions(message, file_options)
+        message_options = get_nanopb_suboptions(message, file_options, names)
         messages.append(Message(names, message, message_options))
         for enum in message.enum_type:
-            enum_options = get_nanopb_suboptions(enum, message_options)
+            enum_options = get_nanopb_suboptions(enum, message_options, names + enum.name)
             enums.append(Enum(names, enum, enum_options))
     
+    for names, extension in iterate_extensions(fdesc, base_name):
+        field_options = get_nanopb_suboptions(extension, file_options, names)
+        if field_options.type != nanopb_pb2.FT_IGNORE:
+            extensions.append(ExtensionField(names, extension, field_options))
+    
     # Fix field default values where enum short names are used.
     for enum in enums:
         if not enum.options.long_names:
@@ -396,7 +656,7 @@ def parse_file(fdesc, file_options):
                         idx = enum.value_longnames.index(field.default)
                         field.default = enum.values[idx][0]
     
-    return enums, messages
+    return enums, messages, extensions
 
 def toposort2(data):
     '''Topological sort.
@@ -439,7 +699,7 @@ def make_identifier(headername):
             result += '_'
     return result
 
-def generate_header(dependencies, headername, enums, messages):
+def generate_header(dependencies, headername, enums, messages, extensions, options):
     '''Generate content for a header file.
     Generates strings, which should be concatenated and stored to file.
     '''
@@ -450,11 +710,17 @@ def generate_header(dependencies, headername, enums, messages):
     symbol = make_identifier(headername)
     yield '#ifndef _PB_%s_\n' % symbol
     yield '#define _PB_%s_\n' % symbol
-    yield '#include <pb.h>\n\n'
+    try:
+        yield options.libformat % ('pb.h')
+    except TypeError:
+        # no %s specified - use whatever was passed in as options.libformat
+        yield options.libformat
+    yield '\n'
     
     for dependency in dependencies:
         noext = os.path.splitext(dependency)[0]
-        yield '#include "%s.pb.h"\n' % noext
+        yield options.genformat % (noext + '.' + options.extension + '.h')
+        yield '\n'
 
     yield '#ifdef __cplusplus\n'
     yield 'extern "C" {\n'
@@ -469,28 +735,52 @@ def generate_header(dependencies, headername, enums, messages):
         yield msg.types()
         yield str(msg) + '\n\n'
     
+    if extensions:
+        yield '/* Extensions */\n'
+        for extension in extensions:
+            yield extension.extension_decl()
+        yield '\n'
+        
     yield '/* Default values for struct fields */\n'
     for msg in messages:
         yield msg.default_decl(True)
     yield '\n'
     
+    yield '/* Field tags (for use in manual encoding/decoding) */\n'
+    for msg in sort_dependencies(messages):
+        for field in msg.fields:
+            yield field.tags()
+    for extension in extensions:
+        yield extension.tags()
+    yield '\n'
+    
     yield '/* Struct field encoding specification for nanopb */\n'
     for msg in messages:
         yield msg.fields_declaration() + '\n'
+    yield '\n'
     
-    yield '\n#ifdef __cplusplus\n'
+    yield '/* Maximum encoded size of messages (where known) */\n'
+    for msg in messages:
+        msize = msg.encoded_size(messages)
+        if msize is not None:
+            identifier = '%s_size' % msg.name
+            yield '#define %-40s %s\n' % (identifier, msize)
+    yield '\n'
+    
+    yield '#ifdef __cplusplus\n'
     yield '} /* extern "C" */\n'
     yield '#endif\n'
     
     # End of header
     yield '\n#endif\n'
 
-def generate_source(headername, enums, messages):
+def generate_source(headername, enums, messages, extensions, options):
     '''Generate content for a source file.'''
     
     yield '/* Automatically generated nanopb constant definitions */\n'
     yield '/* Generated by %s at %s. */\n\n' % (nanopb_version, time.asctime())
-    yield '#include "%s"\n\n' % headername
+    yield options.genformat % (headername)
+    yield '\n'
     
     for msg in messages:
         yield msg.default_decl(False)
@@ -500,6 +790,10 @@ def generate_source(headername, enums, messages):
     for msg in messages:
         yield msg.fields_definition() + '\n\n'
     
+    for ext in extensions:
+        yield ext.extension_def() + '\n'
+        
+    # Add checks for numeric limits
     if messages:
         count_required_fields = lambda m: len([f for f in msg.fields if f.rules == 'REQUIRED'])
         largest_msg = max(messages, key = count_required_fields)
@@ -511,7 +805,6 @@ def generate_source(headername, enums, messages):
             yield '       setting PB_MAX_REQUIRED_FIELDS to %d or more.\n' % largest_count
             yield '#endif\n'
     
-    # Add checks for numeric limits
     worst = 0
     worst_field = ''
     checks = []
@@ -529,16 +822,6 @@ def generate_source(headername, enums, messages):
     if worst > 255 or checks:
         yield '\n/* Check that field information fits in pb_field_t */\n'
         
-        if worst < 65536:
-            yield '#if !defined(PB_FIELD_16BIT) && !defined(PB_FIELD_32BIT)\n'
-            if worst > 255:
-                yield '#error Field descriptor for %s is too large. Define PB_FIELD_16BIT to fix this.\n' % worst_field
-            else:
-                assertion = ' && '.join(str(c) + ' < 256' for c in checks)
-                msgs = '_'.join(str(n) for n in checks_msgnames)
-                yield 'STATIC_ASSERT((%s), YOU_MUST_DEFINE_PB_FIELD_16BIT_FOR_MESSAGES_%s)\n'%(assertion,msgs)
-            yield '#endif\n\n'
-        
         if worst > 65535 or checks:
             yield '#if !defined(PB_FIELD_32BIT)\n'
             if worst > 65535:
@@ -546,8 +829,32 @@ def generate_source(headername, enums, messages):
             else:
                 assertion = ' && '.join(str(c) + ' < 65536' for c in checks)
                 msgs = '_'.join(str(n) for n in checks_msgnames)
+                yield '/* If you get an error here, it means that you need to define PB_FIELD_32BIT\n'
+                yield ' * compile-time option. You can do that in pb.h or on compiler command line.\n'
+                yield ' * \n'
+                yield ' * The reason you need to do this is that some of your messages contain tag\n'
+                yield ' * numbers or field sizes that are larger than what can fit in 8 or 16 bit\n'
+                yield ' * field descriptors.\n'
+                yield ' */\n'
                 yield 'STATIC_ASSERT((%s), YOU_MUST_DEFINE_PB_FIELD_32BIT_FOR_MESSAGES_%s)\n'%(assertion,msgs)
-            yield '#endif\n'
+            yield '#endif\n\n'
+        
+        if worst < 65536:
+            yield '#if !defined(PB_FIELD_16BIT) && !defined(PB_FIELD_32BIT)\n'
+            if worst > 255:
+                yield '#error Field descriptor for %s is too large. Define PB_FIELD_16BIT to fix this.\n' % worst_field
+            else:
+                assertion = ' && '.join(str(c) + ' < 256' for c in checks)
+                msgs = '_'.join(str(n) for n in checks_msgnames)
+                yield '/* If you get an error here, it means that you need to define PB_FIELD_16BIT\n'
+                yield ' * compile-time option. You can do that in pb.h or on compiler command line.\n'
+                yield ' * \n'
+                yield ' * The reason you need to do this is that some of your messages contain tag\n'
+                yield ' * numbers or field sizes that are larger than what can fit in the default\n'
+                yield ' * 8 bit descriptors.\n'
+                yield ' */\n'
+                yield 'STATIC_ASSERT((%s), YOU_MUST_DEFINE_PB_FIELD_16BIT_FOR_MESSAGES_%s)\n'%(assertion,msgs)
+            yield '#endif\n\n'
     
     # Add check for sizeof(double)
     has_double = False
@@ -566,34 +873,48 @@ def generate_source(headername, enums, messages):
     
     yield '\n'
 
-
 # ---------------------------------------------------------------------------
-#                         Command line interface
+#                    Options parsing for the .proto files
 # ---------------------------------------------------------------------------
 
-import sys
+from fnmatch import fnmatch
-import os.path    
-from optparse import OptionParser
-import google.protobuf.text_format as text_format
 
-optparser = OptionParser(
+def read_options_file(infile):
-    usage = "Usage: nanopb_generator.py [options] file.pb ...",
+    '''Parse a separate options file to list:
-    epilog = "Compile file.pb from file.proto by: 'protoc -ofile.pb file.proto'. " +
+        [(namemask, options), ...]
-             "Output will be written to file.pb.h and file.pb.c.")
+    '''
-optparser.add_option("-x", dest="exclude", metavar="FILE", action="append", default=[],
+    results = []
-    help="Exclude file from generated #include list.")
+    for line in infile:
-optparser.add_option("-q", "--quiet", dest="quiet", action="store_true", default=False,
+        line = line.strip()
-    help="Don't print anything except errors.")
+        if not line or line.startswith('//') or line.startswith('#'):
-optparser.add_option("-v", "--verbose", dest="verbose", action="store_true", default=False,
+            continue
-    help="Print more information.")
-optparser.add_option("-s", dest="settings", metavar="OPTION:VALUE", action="append", default=[],
-    help="Set generator option (max_size, max_count etc.).")
         
-def get_nanopb_suboptions(subdesc, options):
+        parts = line.split(None, 1)
+        opts = nanopb_pb2.NanoPBOptions()
+        text_format.Merge(parts[1], opts)
+        results.append((parts[0], opts))
+
+    return results
+
+class Globals:
+    '''Ugly global variables, should find a good way to pass these.'''
+    verbose_options = False
+    separate_options = []
+    matched_namemasks = set()
+
+def get_nanopb_suboptions(subdesc, options, name):
     '''Get copy of options, and merge information from subdesc.'''
     new_options = nanopb_pb2.NanoPBOptions()
     new_options.CopyFrom(options)
     
+    # Handle options defined in a separate file
+    dotname = '.'.join(name.parts)
+    for namemask, options in Globals.separate_options:
+        if fnmatch(dotname, namemask):
+            Globals.matched_namemasks.add(namemask)
+            new_options.MergeFrom(options)
+    
+    # Handle options defined in .proto
     if isinstance(subdesc.options, descriptor.FieldOptions):
         ext_type = nanopb_pb2.nanopb
     elif isinstance(subdesc.options, descriptor.FileOptions):
@@ -609,61 +930,177 @@ def get_nanopb_suboptions(subdesc, options):
         ext = subdesc.options.Extensions[ext_type]
         new_options.MergeFrom(ext)
     
+    if Globals.verbose_options:
+        sys.stderr.write("Options for " + dotname + ": ")
+        sys.stderr.write(text_format.MessageToString(new_options) + "\n")
+    
     return new_options
 
-def process(filenames, options):
-    '''Process the files given on the command line.'''
 
-    if not filenames:
+# ---------------------------------------------------------------------------
-        optparser.print_help()
+#                         Command line interface
-        return False
+# ---------------------------------------------------------------------------
 
-    if options.quiet:
+import sys
-        options.verbose = False
+import os.path    
+from optparse import OptionParser
 
+optparser = OptionParser(
+    usage = "Usage: nanopb_generator.py [options] file.pb ...",
+    epilog = "Compile file.pb from file.proto by: 'protoc -ofile.pb file.proto'. " +
+             "Output will be written to file.pb.h and file.pb.c.")
+optparser.add_option("-x", dest="exclude", metavar="FILE", action="append", default=[],
+    help="Exclude file from generated #include list.")
+optparser.add_option("-e", "--extension", dest="extension", metavar="EXTENSION", default="pb",
+    help="Set extension to use instead of 'pb' for generated files. [default: %default]")
+optparser.add_option("-f", "--options-file", dest="options_file", metavar="FILE", default="%s.options",
+    help="Set name of a separate generator options file.")
+optparser.add_option("-Q", "--generated-include-format", dest="genformat",
+    metavar="FORMAT", default='#include "%s"\n',
+    help="Set format string to use for including other .pb.h files. [default: %default]")
+optparser.add_option("-L", "--library-include-format", dest="libformat",
+    metavar="FORMAT", default='#include <%s>\n',
+    help="Set format string to use for including the nanopb pb.h header. [default: %default]")
+optparser.add_option("-q", "--quiet", dest="quiet", action="store_true", default=False,
+    help="Don't print anything except errors.")
+optparser.add_option("-v", "--verbose", dest="verbose", action="store_true", default=False,
+    help="Print more information.")
+optparser.add_option("-s", dest="settings", metavar="OPTION:VALUE", action="append", default=[],
+    help="Set generator option (max_size, max_count etc.).")
+
+def process_file(filename, fdesc, options):
+    '''Process a single file.
+    filename: The full path to the .proto or .pb source file, as string.
+    fdesc: The loaded FileDescriptorSet, or None to read from the input file.
+    options: Command line options as they come from OptionsParser.
+    
+    Returns a dict:
+        {'headername': Name of header file,
+         'headerdata': Data for the .h header file,
+         'sourcename': Name of the source code file,
+         'sourcedata': Data for the .c source code file
+        }
+    '''
     toplevel_options = nanopb_pb2.NanoPBOptions()
     for s in options.settings:
         text_format.Merge(s, toplevel_options)
     
-    for filename in filenames:
+    if not fdesc:
         data = open(filename, 'rb').read()
-        fdesc = descriptor.FileDescriptorSet.FromString(data)
+        fdesc = descriptor.FileDescriptorSet.FromString(data).file[0]
     
-        file_options = get_nanopb_suboptions(fdesc.file[0], toplevel_options)
+    # Check if there is a separate .options file
+    try:
+        optfilename = options.options_file % os.path.splitext(filename)[0]
+    except TypeError:
+        # No %s specified, use the filename as-is
+        optfilename = options.options_file
     
+    if os.path.isfile(optfilename):
         if options.verbose:
-            print "Options for " + filename + ":"
+            sys.stderr.write('Reading options from ' + optfilename + '\n')
-            print text_format.MessageToString(file_options)
 
-        enums, messages = parse_file(fdesc.file[0], file_options)
+        Globals.separate_options = read_options_file(open(optfilename, "rU"))
+    else:
+        Globals.separate_options = []
+    Globals.matched_namemasks = set()
     
+    # Parse the file
+    file_options = get_nanopb_suboptions(fdesc, toplevel_options, Names([filename]))
+    enums, messages, extensions = parse_file(fdesc, file_options)
+    
+    # Decide the file names
     noext = os.path.splitext(filename)[0]
-        headername = noext + '.pb.h'
+    headername = noext + '.' + options.extension + '.h'
-        sourcename = noext + '.pb.c'
+    sourcename = noext + '.' + options.extension + '.c'
     headerbasename = os.path.basename(headername)
     
-        if not options.quiet:
-            print "Writing to " + headername + " and " + sourcename
-        
     # List of .proto files that should not be included in the C header file
     # even if they are mentioned in the source .proto.
     excludes = ['nanopb.proto', 'google/protobuf/descriptor.proto'] + options.exclude
-        dependencies = [d for d in fdesc.file[0].dependency if d not in excludes]
+    dependencies = [d for d in fdesc.dependency if d not in excludes]
     
-        header = open(headername, 'w')
+    headerdata = ''.join(generate_header(dependencies, headerbasename, enums,
-        for part in generate_header(dependencies, headerbasename, enums, messages):
+                                         messages, extensions, options))
-            header.write(part)
 
-        source = open(sourcename, 'w')
+    sourcedata = ''.join(generate_source(headerbasename, enums,
-        for part in generate_source(headerbasename, enums, messages):
+                                         messages, extensions, options))
-            source.write(part)
 
-    return True
+    # Check if there were any lines in .options that did not match a member
+    unmatched = [n for n,o in Globals.separate_options if n not in Globals.matched_namemasks]
+    if unmatched and not options.quiet:
+        sys.stderr.write("Following patterns in " + optfilename + " did not match any fields: "
+                         + ', '.join(unmatched) + "\n")
+        if not Globals.verbose_options:
+            sys.stderr.write("Use  protoc --nanopb-out=-v:.   to see a list of the field names.\n")
+
+    return {'headername': headername, 'headerdata': headerdata,
+            'sourcename': sourcename, 'sourcedata': sourcedata}
+    
+def main_cli():
+    '''Main function when invoked directly from the command line.'''
     
-if __name__ == '__main__':
     options, filenames = optparser.parse_args()
-    status = process(filenames, options)
     
-    if not status:
+    if not filenames:
+        optparser.print_help()
         sys.exit(1)
     
+    if options.quiet:
+        options.verbose = False
+
+    Globals.verbose_options = options.verbose
+    
+    for filename in filenames:
+        results = process_file(filename, None, options)
+        
+        if not options.quiet:
+            sys.stderr.write("Writing to " + results['headername'] + " and "
+                             + results['sourcename'] + "\n")
+    
+        open(results['headername'], 'w').write(results['headerdata'])
+        open(results['sourcename'], 'w').write(results['sourcedata'])        
+
+def main_plugin():
+    '''Main function when invoked as a protoc plugin.'''
+
+    import sys
+    if sys.platform == "win32":
+        import os, msvcrt
+        # Set stdin and stdout to binary mode
+        msvcrt.setmode(sys.stdin.fileno(), os.O_BINARY)
+        msvcrt.setmode(sys.stdout.fileno(), os.O_BINARY)
+    
+    data = sys.stdin.read()
+    request = plugin_pb2.CodeGeneratorRequest.FromString(data)
+    
+    import shlex
+    args = shlex.split(request.parameter)
+    options, dummy = optparser.parse_args(args)
+    
+    Globals.verbose_options = options.verbose
+    
+    response = plugin_pb2.CodeGeneratorResponse()
+    
+    for filename in request.file_to_generate:
+        for fdesc in request.proto_file:
+            if fdesc.name == filename:
+                results = process_file(filename, fdesc, options)
+                
+                f = response.file.add()
+                f.name = results['headername']
+                f.content = results['headerdata']
+
+                f = response.file.add()
+                f.name = results['sourcename']
+                f.content = results['sourcedata']    
+    
+    sys.stdout.write(response.SerializeToString())
+
+if __name__ == '__main__':
+    # Check if we are running as a plugin under protoc
+    if 'protoc-gen-' in sys.argv[0] or '--protoc-plugin' in sys.argv:
+        main_plugin()
+    else:
+        main_cli()
+

generator/nanopb_pb2.py

@@ -1,158 +0,0 @@
-# Generated by the protocol buffer compiler.  DO NOT EDIT!
-
-from google.protobuf import descriptor
-from google.protobuf import message
-from google.protobuf import reflection
-from google.protobuf import descriptor_pb2
-# @@protoc_insertion_point(imports)
-
-
-import google.protobuf.descriptor_pb2
-
-DESCRIPTOR = descriptor.FileDescriptor(
-  name='nanopb.proto',
-  package='',
-  serialized_pb='\n\x0cnanopb.proto\x1a google/protobuf/descriptor.proto\"\x92\x01\n\rNanoPBOptions\x12\x10\n\x08max_size\x18\x01 \x01(\x05\x12\x11\n\tmax_count\x18\x02 \x01(\x05\x12$\n\x04type\x18\x03 \x01(\x0e\x32\n.FieldType:\nFT_DEFAULT\x12\x18\n\nlong_names\x18\x04 \x01(\x08:\x04true\x12\x1c\n\rpacked_struct\x18\x05 \x01(\x08:\x05\x66\x61lse*J\n\tFieldType\x12\x0e\n\nFT_DEFAULT\x10\x00\x12\x0f\n\x0b\x46T_CALLBACK\x10\x01\x12\r\n\tFT_STATIC\x10\x02\x12\r\n\tFT_IGNORE\x10\x03:E\n\x0enanopb_fileopt\x12\x1c.google.protobuf.FileOptions\x18\xf2\x07 \x01(\x0b\x32\x0e.NanoPBOptions:G\n\rnanopb_msgopt\x12\x1f.google.protobuf.MessageOptions\x18\xf2\x07 \x01(\x0b\x32\x0e.NanoPBOptions:E\n\x0enanopb_enumopt\x12\x1c.google.protobuf.EnumOptions\x18\xf2\x07 \x01(\x0b\x32\x0e.NanoPBOptions:>\n\x06nanopb\x12\x1d.google.protobuf.FieldOptions\x18\xf2\x07 \x01(\x0b\x32\x0e.NanoPBOptions')
-
-_FIELDTYPE = descriptor.EnumDescriptor(
-  name='FieldType',
-  full_name='FieldType',
-  filename=None,
-  file=DESCRIPTOR,
-  values=[
-    descriptor.EnumValueDescriptor(
-      name='FT_DEFAULT', index=0, number=0,
-      options=None,
-      type=None),
-    descriptor.EnumValueDescriptor(
-      name='FT_CALLBACK', index=1, number=1,
-      options=None,
-      type=None),
-    descriptor.EnumValueDescriptor(
-      name='FT_STATIC', index=2, number=2,
-      options=None,
-      type=None),
-    descriptor.EnumValueDescriptor(
-      name='FT_IGNORE', index=3, number=3,
-      options=None,
-      type=None),
-  ],
-  containing_type=None,
-  options=None,
-  serialized_start=199,
-  serialized_end=273,
-)
-
-
-FT_DEFAULT = 0
-FT_CALLBACK = 1
-FT_STATIC = 2
-FT_IGNORE = 3
-
-NANOPB_FILEOPT_FIELD_NUMBER = 1010
-nanopb_fileopt = descriptor.FieldDescriptor(
-  name='nanopb_fileopt', full_name='nanopb_fileopt', index=0,
-  number=1010, type=11, cpp_type=10, label=1,
-  has_default_value=False, default_value=None,
-  message_type=None, enum_type=None, containing_type=None,
-  is_extension=True, extension_scope=None,
-  options=None)
-NANOPB_MSGOPT_FIELD_NUMBER = 1010
-nanopb_msgopt = descriptor.FieldDescriptor(
-  name='nanopb_msgopt', full_name='nanopb_msgopt', index=1,
-  number=1010, type=11, cpp_type=10, label=1,
-  has_default_value=False, default_value=None,
-  message_type=None, enum_type=None, containing_type=None,
-  is_extension=True, extension_scope=None,
-  options=None)
-NANOPB_ENUMOPT_FIELD_NUMBER = 1010
-nanopb_enumopt = descriptor.FieldDescriptor(
-  name='nanopb_enumopt', full_name='nanopb_enumopt', index=2,
-  number=1010, type=11, cpp_type=10, label=1,
-  has_default_value=False, default_value=None,
-  message_type=None, enum_type=None, containing_type=None,
-  is_extension=True, extension_scope=None,
-  options=None)
-NANOPB_FIELD_NUMBER = 1010
-nanopb = descriptor.FieldDescriptor(
-  name='nanopb', full_name='nanopb', index=3,
-  number=1010, type=11, cpp_type=10, label=1,
-  has_default_value=False, default_value=None,
-  message_type=None, enum_type=None, containing_type=None,
-  is_extension=True, extension_scope=None,
-  options=None)
-
-
-_NANOPBOPTIONS = descriptor.Descriptor(
-  name='NanoPBOptions',
-  full_name='NanoPBOptions',
-  filename=None,
-  file=DESCRIPTOR,
-  containing_type=None,
-  fields=[
-    descriptor.FieldDescriptor(
-      name='max_size', full_name='NanoPBOptions.max_size', index=0,
-      number=1, type=5, cpp_type=1, label=1,
-      has_default_value=False, default_value=0,
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='max_count', full_name='NanoPBOptions.max_count', index=1,
-      number=2, type=5, cpp_type=1, label=1,
-      has_default_value=False, default_value=0,
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='type', full_name='NanoPBOptions.type', index=2,
-      number=3, type=14, cpp_type=8, label=1,
-      has_default_value=True, default_value=0,
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='long_names', full_name='NanoPBOptions.long_names', index=3,
-      number=4, type=8, cpp_type=7, label=1,
-      has_default_value=True, default_value=True,
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-    descriptor.FieldDescriptor(
-      name='packed_struct', full_name='NanoPBOptions.packed_struct', index=4,
-      number=5, type=8, cpp_type=7, label=1,
-      has_default_value=True, default_value=False,
-      message_type=None, enum_type=None, containing_type=None,
-      is_extension=False, extension_scope=None,
-      options=None),
-  ],
-  extensions=[
-  ],
-  nested_types=[],
-  enum_types=[
-  ],
-  options=None,
-  is_extendable=False,
-  extension_ranges=[],
-  serialized_start=51,
-  serialized_end=197,
-)
-
-_NANOPBOPTIONS.fields_by_name['type'].enum_type = _FIELDTYPE
-DESCRIPTOR.message_types_by_name['NanoPBOptions'] = _NANOPBOPTIONS
-
-class NanoPBOptions(message.Message):
-  __metaclass__ = reflection.GeneratedProtocolMessageType
-  DESCRIPTOR = _NANOPBOPTIONS
-  
-  # @@protoc_insertion_point(class_scope:NanoPBOptions)
-
-nanopb_fileopt.message_type = _NANOPBOPTIONS
-google.protobuf.descriptor_pb2.FileOptions.RegisterExtension(nanopb_fileopt)
-nanopb_msgopt.message_type = _NANOPBOPTIONS
-google.protobuf.descriptor_pb2.MessageOptions.RegisterExtension(nanopb_msgopt)
-nanopb_enumopt.message_type = _NANOPBOPTIONS
-google.protobuf.descriptor_pb2.EnumOptions.RegisterExtension(nanopb_enumopt)
-nanopb.message_type = _NANOPBOPTIONS
-google.protobuf.descriptor_pb2.FieldOptions.RegisterExtension(nanopb)
-# @@protoc_insertion_point(module_scope)

generator/proto/Makefile

@@ -0,0 +1,4 @@
+all: nanopb_pb2.py plugin_pb2.py
+
+%_pb2.py: %.proto
+	protoc --python_out=. $<

generator/proto/google/protobuf/descriptor.proto

@@ -0,0 +1,620 @@
+// Protocol Buffers - Google's data interchange format
+// Copyright 2008 Google Inc.  All rights reserved.
+// http://code.google.com/p/protobuf/
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions are
+// met:
+//
+//     * Redistributions of source code must retain the above copyright
+// notice, this list of conditions and the following disclaimer.
+//     * Redistributions in binary form must reproduce the above
+// copyright notice, this list of conditions and the following disclaimer
+// in the documentation and/or other materials provided with the
+// distribution.
+//     * Neither the name of Google Inc. nor the names of its
+// contributors may be used to endorse or promote products derived from
+// this software without specific prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+// Author: kenton@google.com (Kenton Varda)
+//  Based on original Protocol Buffers design by
+//  Sanjay Ghemawat, Jeff Dean, and others.
+//
+// The messages in this file describe the definitions found in .proto files.
+// A valid .proto file can be translated directly to a FileDescriptorProto
+// without any other information (e.g. without reading its imports).
+
+
+
+package google.protobuf;
+option java_package = "com.google.protobuf";
+option java_outer_classname = "DescriptorProtos";
+
+// descriptor.proto must be optimized for speed because reflection-based
+// algorithms don't work during bootstrapping.
+option optimize_for = SPEED;
+
+// The protocol compiler can output a FileDescriptorSet containing the .proto
+// files it parses.
+message FileDescriptorSet {
+  repeated FileDescriptorProto file = 1;
+}
+
+// Describes a complete .proto file.
+message FileDescriptorProto {
+  optional string name = 1;       // file name, relative to root of source tree
+  optional string package = 2;    // e.g. "foo", "foo.bar", etc.
+
+  // Names of files imported by this file.
+  repeated string dependency = 3;
+  // Indexes of the public imported files in the dependency list above.
+  repeated int32 public_dependency = 10;
+  // Indexes of the weak imported files in the dependency list.
+  // For Google-internal migration only. Do not use.
+  repeated int32 weak_dependency = 11;
+
+  // All top-level definitions in this file.
+  repeated DescriptorProto message_type = 4;
+  repeated EnumDescriptorProto enum_type = 5;
+  repeated ServiceDescriptorProto service = 6;
+  repeated FieldDescriptorProto extension = 7;
+
+  optional FileOptions options = 8;
+
+  // This field contains optional information about the original source code.
+  // You may safely remove this entire field whithout harming runtime
+  // functionality of the descriptors -- the information is needed only by
+  // development tools.
+  optional SourceCodeInfo source_code_info = 9;
+}
+
+// Describes a message type.
+message DescriptorProto {
+  optional string name = 1;
+
+  repeated FieldDescriptorProto field = 2;
+  repeated FieldDescriptorProto extension = 6;
+
+  repeated DescriptorProto nested_type = 3;
+  repeated EnumDescriptorProto enum_type = 4;
+
+  message ExtensionRange {
+    optional int32 start = 1;
+    optional int32 end = 2;
+  }
+  repeated ExtensionRange extension_range = 5;
+
+  optional MessageOptions options = 7;
+}
+
+// Describes a field within a message.
+message FieldDescriptorProto {
+  enum Type {
+    // 0 is reserved for errors.
+    // Order is weird for historical reasons.
+    TYPE_DOUBLE         = 1;
+    TYPE_FLOAT          = 2;
+    // Not ZigZag encoded.  Negative numbers take 10 bytes.  Use TYPE_SINT64 if
+    // negative values are likely.
+    TYPE_INT64          = 3;
+    TYPE_UINT64         = 4;
+    // Not ZigZag encoded.  Negative numbers take 10 bytes.  Use TYPE_SINT32 if
+    // negative values are likely.
+    TYPE_INT32          = 5;
+    TYPE_FIXED64        = 6;
+    TYPE_FIXED32        = 7;
+    TYPE_BOOL           = 8;
+    TYPE_STRING         = 9;
+    TYPE_GROUP          = 10;  // Tag-delimited aggregate.
+    TYPE_MESSAGE        = 11;  // Length-delimited aggregate.
+
+    // New in version 2.
+    TYPE_BYTES          = 12;
+    TYPE_UINT32         = 13;
+    TYPE_ENUM           = 14;
+    TYPE_SFIXED32       = 15;
+    TYPE_SFIXED64       = 16;
+    TYPE_SINT32         = 17;  // Uses ZigZag encoding.
+    TYPE_SINT64         = 18;  // Uses ZigZag encoding.
+  };
+
+  enum Label {
+    // 0 is reserved for errors
+    LABEL_OPTIONAL      = 1;
+    LABEL_REQUIRED      = 2;
+    LABEL_REPEATED      = 3;
+    // TODO(sanjay): Should we add LABEL_MAP?
+  };
+
+  optional string name = 1;
+  optional int32 number = 3;
+  optional Label label = 4;
+
+  // If type_name is set, this need not be set.  If both this and type_name
+  // are set, this must be either TYPE_ENUM or TYPE_MESSAGE.
+  optional Type type = 5;
+
+  // For message and enum types, this is the name of the type.  If the name
+  // starts with a '.', it is fully-qualified.  Otherwise, C++-like scoping
+  // rules are used to find the type (i.e. first the nested types within this
+  // message are searched, then within the parent, on up to the root
+  // namespace).
+  optional string type_name = 6;
+
+  // For extensions, this is the name of the type being extended.  It is
+  // resolved in the same manner as type_name.
+  optional string extendee = 2;
+
+  // For numeric types, contains the original text representation of the value.
+  // For booleans, "true" or "false".
+  // For strings, contains the default text contents (not escaped in any way).
+  // For bytes, contains the C escaped value.  All bytes >= 128 are escaped.
+  // TODO(kenton):  Base-64 encode?
+  optional string default_value = 7;
+
+  optional FieldOptions options = 8;
+}
+
+// Describes an enum type.
+message EnumDescriptorProto {
+  optional string name = 1;
+
+  repeated EnumValueDescriptorProto value = 2;
+
+  optional EnumOptions options = 3;
+}
+
+// Describes a value within an enum.
+message EnumValueDescriptorProto {
+  optional string name = 1;
+  optional int32 number = 2;
+
+  optional EnumValueOptions options = 3;
+}
+
+// Describes a service.
+message ServiceDescriptorProto {
+  optional string name = 1;
+  repeated MethodDescriptorProto method = 2;
+
+  optional ServiceOptions options = 3;
+}
+
+// Describes a method of a service.
+message MethodDescriptorProto {
+  optional string name = 1;
+
+  // Input and output type names.  These are resolved in the same way as
+  // FieldDescriptorProto.type_name, but must refer to a message type.
+  optional string input_type = 2;
+  optional string output_type = 3;
+
+  optional MethodOptions options = 4;
+}
+
+
+// ===================================================================
+// Options
+
+// Each of the definitions above may have "options" attached.  These are
+// just annotations which may cause code to be generated slightly differently
+// or may contain hints for code that manipulates protocol messages.
+//
+// Clients may define custom options as extensions of the *Options messages.
+// These extensions may not yet be known at parsing time, so the parser cannot
+// store the values in them.  Instead it stores them in a field in the *Options
+// message called uninterpreted_option. This field must have the same name
+// across all *Options messages. We then use this field to populate the
+// extensions when we build a descriptor, at which point all protos have been
+// parsed and so all extensions are known.
+//
+// Extension numbers for custom options may be chosen as follows:
+// * For options which will only be used within a single application or
+//   organization, or for experimental options, use field numbers 50000
+//   through 99999.  It is up to you to ensure that you do not use the
+//   same number for multiple options.
+// * For options which will be published and used publicly by multiple
+//   independent entities, e-mail protobuf-global-extension-registry@google.com
+//   to reserve extension numbers. Simply provide your project name (e.g.
+//   Object-C plugin) and your porject website (if available) -- there's no need
+//   to explain how you intend to use them. Usually you only need one extension
+//   number. You can declare multiple options with only one extension number by
+//   putting them in a sub-message. See the Custom Options section of the docs
+//   for examples:
+//   http://code.google.com/apis/protocolbuffers/docs/proto.html#options
+//   If this turns out to be popular, a web service will be set up
+//   to automatically assign option numbers.
+
+
+message FileOptions {
+
+  // Sets the Java package where classes generated from this .proto will be
+  // placed.  By default, the proto package is used, but this is often
+  // inappropriate because proto packages do not normally start with backwards
+  // domain names.
+  optional string java_package = 1;
+
+
+  // If set, all the classes from the .proto file are wrapped in a single
+  // outer class with the given name.  This applies to both Proto1
+  // (equivalent to the old "--one_java_file" option) and Proto2 (where
+  // a .proto always translates to a single class, but you may want to
+  // explicitly choose the class name).
+  optional string java_outer_classname = 8;
+
+  // If set true, then the Java code generator will generate a separate .java
+  // file for each top-level message, enum, and service defined in the .proto
+  // file.  Thus, these types will *not* be nested inside the outer class
+  // named by java_outer_classname.  However, the outer class will still be
+  // generated to contain the file's getDescriptor() method as well as any
+  // top-level extensions defined in the file.
+  optional bool java_multiple_files = 10 [default=false];
+
+  // If set true, then the Java code generator will generate equals() and
+  // hashCode() methods for all messages defined in the .proto file. This is
+  // purely a speed optimization, as the AbstractMessage base class includes
+  // reflection-based implementations of these methods.
+  optional bool java_generate_equals_and_hash = 20 [default=false];
+
+  // Generated classes can be optimized for speed or code size.
+  enum OptimizeMode {
+    SPEED = 1;        // Generate complete code for parsing, serialization,
+                      // etc.
+    CODE_SIZE = 2;    // Use ReflectionOps to implement these methods.
+    LITE_RUNTIME = 3; // Generate code using MessageLite and the lite runtime.
+  }
+  optional OptimizeMode optimize_for = 9 [default=SPEED];
+
+  // Sets the Go package where structs generated from this .proto will be
+  // placed.  There is no default.
+  optional string go_package = 11;
+
+
+
+  // Should generic services be generated in each language?  "Generic" services
+  // are not specific to any particular RPC system.  They are generated by the
+  // main code generators in each language (without additional plugins).
+  // Generic services were the only kind of service generation supported by
+  // early versions of proto2.
+  //
+  // Generic services are now considered deprecated in favor of using plugins
+  // that generate code specific to your particular RPC system.  Therefore,
+  // these default to false.  Old code which depends on generic services should
+  // explicitly set them to true.
+  optional bool cc_generic_services = 16 [default=false];
+  optional bool java_generic_services = 17 [default=false];
+  optional bool py_generic_services = 18 [default=false];
+
+  // The parser stores options it doesn't recognize here. See above.
+  repeated UninterpretedOption uninterpreted_option = 999;
+
+  // Clients can define custom options in extensions of this message. See above.
+  extensions 1000 to max;
+}
+
+message MessageOptions {
+  // Set true to use the old proto1 MessageSet wire format for extensions.
+  // This is provided for backwards-compatibility with the MessageSet wire
+  // format.  You should not use this for any other reason:  It's less
+  // efficient, has fewer features, and is more complicated.
+  //
+  // The message must be defined exactly as follows:
+  //   message Foo {
+  //     option message_set_wire_format = true;
+  //     extensions 4 to max;
+  //   }
+  // Note that the message cannot have any defined fields; MessageSets only
+  // have extensions.
+  //
+  // All extensions of your type must be singular messages; e.g. they cannot
+  // be int32s, enums, or repeated messages.
+  //
+  // Because this is an option, the above two restrictions are not enforced by
+  // the protocol compiler.
+  optional bool message_set_wire_format = 1 [default=false];
+
+  // Disables the generation of the standard "descriptor()" accessor, which can
+  // conflict with a field of the same name.  This is meant to make migration
+  // from proto1 easier; new code should avoid fields named "descriptor".
+  optional bool no_standard_descriptor_accessor = 2 [default=false];
+
+  // The parser stores options it doesn't recognize here. See above.
+  repeated UninterpretedOption uninterpreted_option = 999;
+
+  // Clients can define custom options in extensions of this message. See above.
+  extensions 1000 to max;
+}
+
+message FieldOptions {
+  // The ctype option instructs the C++ code generator to use a different
+  // representation of the field than it normally would.  See the specific
+  // options below.  This option is not yet implemented in the open source
+  // release -- sorry, we'll try to include it in a future version!
+  optional CType ctype = 1 [default = STRING];
+  enum CType {
+    // Default mode.
+    STRING = 0;
+
+    CORD = 1;
+
+    STRING_PIECE = 2;
+  }
+  // The packed option can be enabled for repeated primitive fields to enable
+  // a more efficient representation on the wire. Rather than repeatedly
+  // writing the tag and type for each element, the entire array is encoded as
+  // a single length-delimited blob.
+  optional bool packed = 2;
+
+
+
+  // Should this field be parsed lazily?  Lazy applies only to message-type
+  // fields.  It means that when the outer message is initially parsed, the
+  // inner message's contents will not be parsed but instead stored in encoded
+  // form.  The inner message will actually be parsed when it is first accessed.
+  //
+  // This is only a hint.  Implementations are free to choose whether to use
+  // eager or lazy parsing regardless of the value of this option.  However,
+  // setting this option true suggests that the protocol author believes that
+  // using lazy parsing on this field is worth the additional bookkeeping
+  // overhead typically needed to implement it.
+  //
+  // This option does not affect the public interface of any generated code;
+  // all method signatures remain the same.  Furthermore, thread-safety of the
+  // interface is not affected by this option; const methods remain safe to
+  // call from multiple threads concurrently, while non-const methods continue
+  // to require exclusive access.
+  //
+  //
+  // Note that implementations may choose not to check required fields within
+  // a lazy sub-message.  That is, calling IsInitialized() on the outher message
+  // may return true even if the inner message has missing required fields.
+  // This is necessary because otherwise the inner message would have to be
+  // parsed in order to perform the check, defeating the purpose of lazy
+  // parsing.  An implementation which chooses not to check required fields
+  // must be consistent about it.  That is, for any particular sub-message, the
+  // implementation must either *always* check its required fields, or *never*
+  // check its required fields, regardless of whether or not the message has
+  // been parsed.
+  optional bool lazy = 5 [default=false];
+
+  // Is this field deprecated?
+  // Depending on the target platform, this can emit Deprecated annotations
+  // for accessors, or it will be completely ignored; in the very least, this
+  // is a formalization for deprecating fields.
+  optional bool deprecated = 3 [default=false];
+
+  // EXPERIMENTAL.  DO NOT USE.
+  // For "map" fields, the name of the field in the enclosed type that
+  // is the key for this map.  For example, suppose we have:
+  //   message Item {
+  //     required string name = 1;
+  //     required string value = 2;
+  //   }
+  //   message Config {
+  //     repeated Item items = 1 [experimental_map_key="name"];
+  //   }
+  // In this situation, the map key for Item will be set to "name".
+  // TODO: Fully-implement this, then remove the "experimental_" prefix.
+  optional string experimental_map_key = 9;
+
+  // For Google-internal migration only. Do not use.
+  optional bool weak = 10 [default=false];
+
+  // The parser stores options it doesn't recognize here. See above.
+  repeated UninterpretedOption uninterpreted_option = 999;
+
+  // Clients can define custom options in extensions of this message. See above.
+  extensions 1000 to max;
+}
+
+message EnumOptions {
+
+  // Set this option to false to disallow mapping different tag names to a same
+  // value.
+  optional bool allow_alias = 2 [default=true];
+
+  // The parser stores options it doesn't recognize here. See above.
+  repeated UninterpretedOption uninterpreted_option = 999;
+
+  // Clients can define custom options in extensions of this message. See above.
+  extensions 1000 to max;
+}
+
+message EnumValueOptions {
+  // The parser stores options it doesn't recognize here. See above.
+  repeated UninterpretedOption uninterpreted_option = 999;
+
+  // Clients can define custom options in extensions of this message. See above.
+  extensions 1000 to max;
+}
+
+message ServiceOptions {
+
+  // Note:  Field numbers 1 through 32 are reserved for Google's internal RPC
+  //   framework.  We apologize for hoarding these numbers to ourselves, but
+  //   we were already using them long before we decided to release Protocol
+  //   Buffers.
+
+  // The parser stores options it doesn't recognize here. See above.
+  repeated UninterpretedOption uninterpreted_option = 999;
+
+  // Clients can define custom options in extensions of this message. See above.
+  extensions 1000 to max;
+}
+
+message MethodOptions {
+
+  // Note:  Field numbers 1 through 32 are reserved for Google's internal RPC
+  //   framework.  We apologize for hoarding these numbers to ourselves, but
+  //   we were already using them long before we decided to release Protocol
+  //   Buffers.
+
+  // The parser stores options it doesn't recognize here. See above.
+  repeated UninterpretedOption uninterpreted_option = 999;
+
+  // Clients can define custom options in extensions of this message. See above.
+  extensions 1000 to max;
+}
+
+
+// A message representing a option the parser does not recognize. This only
+// appears in options protos created by the compiler::Parser class.
+// DescriptorPool resolves these when building Descriptor objects. Therefore,
+// options protos in descriptor objects (e.g. returned by Descriptor::options(),
+// or produced by Descriptor::CopyTo()) will never have UninterpretedOptions
+// in them.
+message UninterpretedOption {
+  // The name of the uninterpreted option.  Each string represents a segment in
+  // a dot-separated name.  is_extension is true iff a segment represents an
+  // extension (denoted with parentheses in options specs in .proto files).
+  // E.g.,{ ["foo", false], ["bar.baz", true], ["qux", false] } represents
+  // "foo.(bar.baz).qux".
+  message NamePart {
+    required string name_part = 1;
+    required bool is_extension = 2;
+  }
+  repeated NamePart name = 2;
+
+  // The value of the uninterpreted option, in whatever type the tokenizer
+  // identified it as during parsing. Exactly one of these should be set.
+  optional string identifier_value = 3;
+  optional uint64 positive_int_value = 4;
+  optional int64 negative_int_value = 5;
+  optional double double_value = 6;
+  optional bytes string_value = 7;
+  optional string aggregate_value = 8;
+}
+
+// ===================================================================
+// Optional source code info
+
+// Encapsulates information about the original source file from which a
+// FileDescriptorProto was generated.
+message SourceCodeInfo {
+  // A Location identifies a piece of source code in a .proto file which
+  // corresponds to a particular definition.  This information is intended
+  // to be useful to IDEs, code indexers, documentation generators, and similar
+  // tools.
+  //
+  // For example, say we have a file like:
+  //   message Foo {
+  //     optional string foo = 1;
+  //   }
+  // Let's look at just the field definition:
+  //   optional string foo = 1;
+  //   ^       ^^     ^^  ^  ^^^
+  //   a       bc     de  f  ghi
+  // We have the following locations:
+  //   span   path               represents
+  //   [a,i)  [ 4, 0, 2, 0 ]     The whole field definition.
+  //   [a,b)  [ 4, 0, 2, 0, 4 ]  The label (optional).
+  //   [c,d)  [ 4, 0, 2, 0, 5 ]  The type (string).
+  //   [e,f)  [ 4, 0, 2, 0, 1 ]  The name (foo).
+  //   [g,h)  [ 4, 0, 2, 0, 3 ]  The number (1).
+  //
+  // Notes:
+  // - A location may refer to a repeated field itself (i.e. not to any
+  //   particular index within it).  This is used whenever a set of elements are
+  //   logically enclosed in a single code segment.  For example, an entire
+  //   extend block (possibly containing multiple extension definitions) will
+  //   have an outer location whose path refers to the "extensions" repeated
+  //   field without an index.
+  // - Multiple locations may have the same path.  This happens when a single
+  //   logical declaration is spread out across multiple places.  The most
+  //   obvious example is the "extend" block again -- there may be multiple
+  //   extend blocks in the same scope, each of which will have the same path.
+  // - A location's span is not always a subset of its parent's span.  For
+  //   example, the "extendee" of an extension declaration appears at the
+  //   beginning of the "extend" block and is shared by all extensions within
+  //   the block.
+  // - Just because a location's span is a subset of some other location's span
+  //   does not mean that it is a descendent.  For example, a "group" defines
+  //   both a type and a field in a single declaration.  Thus, the locations
+  //   corresponding to the type and field and their components will overlap.
+  // - Code which tries to interpret locations should probably be designed to
+  //   ignore those that it doesn't understand, as more types of locations could
+  //   be recorded in the future.
+  repeated Location location = 1;
+  message Location {
+    // Identifies which part of the FileDescriptorProto was defined at this
+    // location.
+    //
+    // Each element is a field number or an index.  They form a path from
+    // the root FileDescriptorProto to the place where the definition.  For
+    // example, this path:
+    //   [ 4, 3, 2, 7, 1 ]
+    // refers to:
+    //   file.message_type(3)  // 4, 3
+    //       .field(7)         // 2, 7
+    //       .name()           // 1
+    // This is because FileDescriptorProto.message_type has field number 4:
+    //   repeated DescriptorProto message_type = 4;
+    // and DescriptorProto.field has field number 2:
+    //   repeated FieldDescriptorProto field = 2;
+    // and FieldDescriptorProto.name has field number 1:
+    //   optional string name = 1;
+    //
+    // Thus, the above path gives the location of a field name.  If we removed
+    // the last element:
+    //   [ 4, 3, 2, 7 ]
+    // this path refers to the whole field declaration (from the beginning
+    // of the label to the terminating semicolon).
+    repeated int32 path = 1 [packed=true];
+
+    // Always has exactly three or four elements: start line, start column,
+    // end line (optional, otherwise assumed same as start line), end column.
+    // These are packed into a single field for efficiency.  Note that line
+    // and column numbers are zero-based -- typically you will want to add
+    // 1 to each before displaying to a user.
+    repeated int32 span = 2 [packed=true];
+
+    // If this SourceCodeInfo represents a complete declaration, these are any
+    // comments appearing before and after the declaration which appear to be
+    // attached to the declaration.
+    //
+    // A series of line comments appearing on consecutive lines, with no other
+    // tokens appearing on those lines, will be treated as a single comment.
+    //
+    // Only the comment content is provided; comment markers (e.g. //) are
+    // stripped out.  For block comments, leading whitespace and an asterisk
+    // will be stripped from the beginning of each line other than the first.
+    // Newlines are included in the output.
+    //
+    // Examples:
+    //
+    //   optional int32 foo = 1;  // Comment attached to foo.
+    //   // Comment attached to bar.
+    //   optional int32 bar = 2;
+    //
+    //   optional string baz = 3;
+    //   // Comment attached to baz.
+    //   // Another line attached to baz.
+    //
+    //   // Comment attached to qux.
+    //   //
+    //   // Another line attached to qux.
+    //   optional double qux = 4;
+    //
+    //   optional string corge = 5;
+    //   /* Block comment attached
+    //    * to corge.  Leading asterisks
+    //    * will be removed. */
+    //   /* Block comment attached to
+    //    * grault. */
+    //   optional int32 grault = 6;
+    optional string leading_comments = 3;
+    optional string trailing_comments = 4;
+  }
+}

generator/proto/nanopb.proto

@@ -12,10 +12,14 @@ option java_package = "fi.kapsi.koti.jpa.nanopb";
 enum FieldType {
     FT_DEFAULT = 0; // Automatically decide field type, generate static field if possible.
     FT_CALLBACK = 1; // Always generate a callback field.
+    FT_POINTER = 4; // Always generate a dynamically allocated field.
     FT_STATIC = 2; // Generate a static field or raise an exception if not possible.
     FT_IGNORE = 3; // Ignore the field completely.
 }
 
+// This is the inner options message, which basically defines options for
+// a field. When it is used in message or file scope, it applies to all
+// fields.
 message NanoPBOptions {
   // Allocated size for 'bytes' and 'string' fields.
   optional int32 max_size = 1;
@@ -30,9 +34,14 @@ message NanoPBOptions {
   optional bool long_names = 4 [default = true];
   
   // Add 'packed' attribute to generated structs.
+  // Note: this cannot be used on CPUs that break on unaligned
+  // accesses to variables.
   optional bool packed_struct = 5 [default = false];
 }
 
+// Extensions to protoc 'Descriptor' type in order to define options
+// inside a .proto file.
+//
 // Protocol Buffers extension number registry
 // --------------------------------
 // Project:  Nanopb

generator/proto/plugin.proto

@@ -0,0 +1,145 @@
+// Protocol Buffers - Google's data interchange format
+// Copyright 2008 Google Inc.  All rights reserved.
+// http://code.google.com/p/protobuf/
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions are
+// met:
+//
+//     * Redistributions of source code must retain the above copyright
+// notice, this list of conditions and the following disclaimer.
+//     * Redistributions in binary form must reproduce the above
+// copyright notice, this list of conditions and the following disclaimer
+// in the documentation and/or other materials provided with the
+// distribution.
+//     * Neither the name of Google Inc. nor the names of its
+// contributors may be used to endorse or promote products derived from
+// this software without specific prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+// Author: kenton@google.com (Kenton Varda)
+//
+// WARNING:  The plugin interface is currently EXPERIMENTAL and is subject to
+//   change.
+//
+// protoc (aka the Protocol Compiler) can be extended via plugins.  A plugin is
+// just a program that reads a CodeGeneratorRequest from stdin and writes a
+// CodeGeneratorResponse to stdout.
+//
+// Plugins written using C++ can use google/protobuf/compiler/plugin.h instead
+// of dealing with the raw protocol defined here.
+//
+// A plugin executable needs only to be placed somewhere in the path.  The
+// plugin should be named "protoc-gen-$NAME", and will then be used when the
+// flag "--${NAME}_out" is passed to protoc.
+
+package google.protobuf.compiler;
+
+import "google/protobuf/descriptor.proto";
+
+// An encoded CodeGeneratorRequest is written to the plugin's stdin.
+message CodeGeneratorRequest {
+  // The .proto files that were explicitly listed on the command-line.  The
+  // code generator should generate code only for these files.  Each file's
+  // descriptor will be included in proto_file, below.
+  repeated string file_to_generate = 1;
+
+  // The generator parameter passed on the command-line.
+  optional string parameter = 2;
+
+  // FileDescriptorProtos for all files in files_to_generate and everything
+  // they import.  The files will appear in topological order, so each file
+  // appears before any file that imports it.
+  //
+  // protoc guarantees that all proto_files will be written after
+  // the fields above, even though this is not technically guaranteed by the
+  // protobuf wire format.  This theoretically could allow a plugin to stream
+  // in the FileDescriptorProtos and handle them one by one rather than read
+  // the entire set into memory at once.  However, as of this writing, this
+  // is not similarly optimized on protoc's end -- it will store all fields in
+  // memory at once before sending them to the plugin.
+  repeated FileDescriptorProto proto_file = 15;
+}
+
+// The plugin writes an encoded CodeGeneratorResponse to stdout.
+message CodeGeneratorResponse {
+  // Error message.  If non-empty, code generation failed.  The plugin process
+  // should exit with status code zero even if it reports an error in this way.
+  //
+  // This should be used to indicate errors in .proto files which prevent the
+  // code generator from generating correct code.  Errors which indicate a
+  // problem in protoc itself -- such as the input CodeGeneratorRequest being
+  // unparseable -- should be reported by writing a message to stderr and
+  // exiting with a non-zero status code.
+  optional string error = 1;
+
+  // Represents a single generated file.
+  message File {
+    // The file name, relative to the output directory.  The name must not
+    // contain "." or ".." components and must be relative, not be absolute (so,
+    // the file cannot lie outside the output directory).  "/" must be used as
+    // the path separator, not "\".
+    //
+    // If the name is omitted, the content will be appended to the previous
+    // file.  This allows the generator to break large files into small chunks,
+    // and allows the generated text to be streamed back to protoc so that large
+    // files need not reside completely in memory at one time.  Note that as of
+    // this writing protoc does not optimize for this -- it will read the entire
+    // CodeGeneratorResponse before writing files to disk.
+    optional string name = 1;
+
+    // If non-empty, indicates that the named file should already exist, and the
+    // content here is to be inserted into that file at a defined insertion
+    // point.  This feature allows a code generator to extend the output
+    // produced by another code generator.  The original generator may provide
+    // insertion points by placing special annotations in the file that look
+    // like:
+    //   @@protoc_insertion_point(NAME)
+    // The annotation can have arbitrary text before and after it on the line,
+    // which allows it to be placed in a comment.  NAME should be replaced with
+    // an identifier naming the point -- this is what other generators will use
+    // as the insertion_point.  Code inserted at this point will be placed
+    // immediately above the line containing the insertion point (thus multiple
+    // insertions to the same point will come out in the order they were added).
+    // The double-@ is intended to make it unlikely that the generated code
+    // could contain things that look like insertion points by accident.
+    //
+    // For example, the C++ code generator places the following line in the
+    // .pb.h files that it generates:
+    //   // @@protoc_insertion_point(namespace_scope)
+    // This line appears within the scope of the file's package namespace, but
+    // outside of any particular class.  Another plugin can then specify the
+    // insertion_point "namespace_scope" to generate additional classes or
+    // other declarations that should be placed in this scope.
+    //
+    // Note that if the line containing the insertion point begins with
+    // whitespace, the same whitespace will be added to every line of the
+    // inserted text.  This is useful for languages like Python, where
+    // indentation matters.  In these languages, the insertion point comment
+    // should be indented the same amount as any inserted code will need to be
+    // in order to work correctly in that context.
+    //
+    // The code generator that generates the initial file and the one which
+    // inserts into it must both run as part of a single invocation of protoc.
+    // Code generators are executed in the order in which they appear on the
+    // command line.
+    //
+    // If |insertion_point| is present, |name| must also be present.
+    optional string insertion_point = 2;
+
+    // The file contents.
+    optional string content = 15;
+  }
+  repeated File file = 15;
+}

generator/protoc-gen-nanopb

@@ -0,0 +1,13 @@
+#!/bin/sh
+
+# This file is used to invoke nanopb_generator.py as a plugin
+# to protoc on Linux and other *nix-style systems.
+# Use it like this:
+# protoc --plugin=nanopb=..../protoc-gen-nanopb --nanopb_out=dir foo.proto
+#
+# Note that if you use the binary package of nanopb, the protoc
+# path is already set up properly and there is no need to give
+# --plugin= on the command line.
+
+MYPATH=$(dirname "$0")
+exec python "$MYPATH/nanopb_generator.py" --protoc-plugin

generator/protoc-gen-nanopb.bat

@@ -0,0 +1,12 @@
+@echo off
+:: This file is used to invoke nanopb_generator.py as a plugin
+:: to protoc on Windows.
+:: Use it like this:
+:: protoc --plugin=nanopb=..../protoc-gen-nanopb.bat --nanopb_out=dir foo.proto
+::
+:: Note that if you use the binary package of nanopb, the protoc
+:: path is already set up properly and there is no need to give
+:: --plugin= on the command line.
+
+set mydir=%~dp0
+python "%mydir%\nanopb_generator.py" --protoc-plugin

pb.h

@@ -1,21 +1,99 @@
+/* Common parts of the nanopb library. Most of these are quite low-level
+ * stuff. For the high-level interface, see pb_encode.h and pb_decode.h.
+ */
+
 #ifndef _PB_H_
 #define _PB_H_
 
-/* pb.h: Common parts for nanopb library.
+/*****************************************************************
- * Most of these are quite low-level stuff. For the high-level interface,
+ * Nanopb compilation time options. You can change these here by *
- * see pb_encode.h or pb_decode.h
+ * uncommenting the lines, or on the compiler command line.      *
+ *****************************************************************/
+
+/* Enable support for dynamically allocated fields */
+/* #define PB_ENABLE_MALLOC 1 */
+
+/* Define this if your CPU architecture is big endian, i.e. it
+ * stores the most-significant byte first. */
+/* #define __BIG_ENDIAN__ 1 */
+
+/* Increase the number of required fields that are tracked.
+ * A compiler warning will tell if you need this. */
+/* #define PB_MAX_REQUIRED_FIELDS 256 */
+
+/* Add support for tag numbers > 255 and fields larger than 255 bytes. */
+/* #define PB_FIELD_16BIT 1 */
+
+/* Add support for tag numbers > 65536 and fields larger than 65536 bytes. */
+/* #define PB_FIELD_32BIT 1 */
+
+/* Disable support for error messages in order to save some code space. */
+/* #define PB_NO_ERRMSG 1 */
+
+/* Disable support for custom streams (support only memory buffers). */
+/* #define PB_BUFFER_ONLY 1 */
+
+/* Switch back to the old-style callback function signature.
+ * This was the default until nanopb-0.2.1. */
+/* #define PB_OLD_CALLBACK_STYLE */
+
+
+/******************************************************************
+ * You usually don't need to change anything below this line.     *
+ * Feel free to look around and use the defined macros, though.   *
+ ******************************************************************/
+
+
+/* Version of the nanopb library. Just in case you want to check it in
+ * your own program. */
+#define NANOPB_VERSION nanopb-0.2.7-dev
+
+/* Include all the system headers needed by nanopb. You will need the
+ * definitions of the following:
+ * - strlen, memcpy, memset functions
+ * - [u]int8_t, [u]int16_t, [u]int32_t, [u]int64_t
+ * - size_t
+ * - bool
+ *
+ * If you don't have the standard header files, you can instead provide
+ * a custom header that defines or includes all this. In that case,
+ * define PB_SYSTEM_HEADER to the path of this file.
  */
-
+#ifdef PB_SYSTEM_HEADER
-#define NANOPB_VERSION nanopb-0.2.0-dev
+#include PB_SYSTEM_HEADER
-
+#else
 #include <stdint.h>
 #include <stddef.h>
 #include <stdbool.h>
+#include <string.h>
 
-#ifdef __GNUC__
+#ifdef PB_ENABLE_MALLOC
-/* This just reduces memory requirements, but is not required. */
+#include <stdlib.h>
+#endif
+#endif
+
+/* Macro for defining packed structures (compiler dependent).
+ * This just reduces memory requirements, but is not required.
+ */
+#if defined(__GNUC__) || defined(__clang__)
+    /* For GCC and clang */
+#   define PB_PACKED_STRUCT_START
+#   define PB_PACKED_STRUCT_END
 #   define pb_packed __attribute__((packed))
+#elif defined(__ICCARM__)
+    /* For IAR ARM compiler */
+#   define PB_PACKED_STRUCT_START _Pragma("pack(push, 1)")
+#   define PB_PACKED_STRUCT_END _Pragma("pack(pop)")
+#   define pb_packed
+#elif defined(_MSC_VER) && (_MSC_VER >= 1500)
+    /* For Microsoft Visual C++ */
+#   define PB_PACKED_STRUCT_START __pragma(pack(push, 1))
+#   define PB_PACKED_STRUCT_END __pragma(pack(pop))
+#   define pb_packed
 #else
+    /* Unknown compiler */
+#   define PB_PACKED_STRUCT_START
+#   define PB_PACKED_STRUCT_END
 #   define pb_packed
 #endif
 
@@ -25,16 +103,21 @@
 #endif
 
 /* Compile-time assertion, used for checking compatible compilation options.
- * If this fails on your compiler for some reason, use #define STATIC_ASSERT
+ * If this does not work properly on your compiler, use #define STATIC_ASSERT
- * to disable it. */
+ * to disable it.
+ *
+ * But before doing that, check carefully the error message / place where it
+ * comes from to see if the error has a real cause. Unfortunately the error
+ * message is not always very clear to read, but you can see the reason better
+ * in the place where the STATIC_ASSERT macro was called.
+ */
 #ifndef STATIC_ASSERT
 #define STATIC_ASSERT(COND,MSG) typedef char STATIC_ASSERT_MSG(MSG, __LINE__, __COUNTER__)[(COND)?1:-1];
 #define STATIC_ASSERT_MSG(MSG, LINE, COUNTER) STATIC_ASSERT_MSG_(MSG, LINE, COUNTER)
 #define STATIC_ASSERT_MSG_(MSG, LINE, COUNTER) static_assertion_##MSG##LINE##COUNTER
 #endif
 
-/* Number of required fields to keep track of
+/* Number of required fields to keep track of. */
- * (change here or on compiler command line). */
 #ifndef PB_MAX_REQUIRED_FIELDS
 #define PB_MAX_REQUIRED_FIELDS 64
 #endif
@@ -46,58 +129,53 @@
 /* List of possible field types. These are used in the autogenerated code.
  * Least-significant 4 bits tell the scalar type
  * Most-significant 4 bits specify repeated/required/packed etc.
- * 
- * INT32 and UINT32 are treated the same, as are (U)INT64 and (S)FIXED*
- * These types are simply casted to correct field type when they are
- * assigned to the memory pointer.
- * SINT* is different, though, because it is zig-zag coded.
  */
 
 typedef uint8_t pb_type_t;
 
-/************************
+/**** Field data types ****/
- * Field contents types *
- ************************/
 
 /* Numeric types */
-#define PB_LTYPE_VARINT 0x00 /* int32, uint32, int64, uint64, bool, enum */
+#define PB_LTYPE_VARINT  0x00 /* int32, int64, enum, bool */
-#define PB_LTYPE_SVARINT 0x01 /* sint32, sint64 */
+#define PB_LTYPE_UVARINT 0x01 /* uint32, uint64 */
-#define PB_LTYPE_FIXED32 0x02 /* fixed32, sfixed32, float */
+#define PB_LTYPE_SVARINT 0x02 /* sint32, sint64 */
-#define PB_LTYPE_FIXED64 0x03 /* fixed64, sfixed64, double */
+#define PB_LTYPE_FIXED32 0x03 /* fixed32, sfixed32, float */
+#define PB_LTYPE_FIXED64 0x04 /* fixed64, sfixed64, double */
 
 /* Marker for last packable field type. */
-#define PB_LTYPE_LAST_PACKABLE 0x03
+#define PB_LTYPE_LAST_PACKABLE 0x04
 
 /* Byte array with pre-allocated buffer.
  * data_size is the length of the allocated PB_BYTES_ARRAY structure. */
-#define PB_LTYPE_BYTES 0x04
+#define PB_LTYPE_BYTES 0x05
 
 /* String with pre-allocated buffer.
  * data_size is the maximum length. */
-#define PB_LTYPE_STRING 0x05
+#define PB_LTYPE_STRING 0x06
 
 /* Submessage
  * submsg_fields is pointer to field descriptions */
-#define PB_LTYPE_SUBMESSAGE 0x06
+#define PB_LTYPE_SUBMESSAGE 0x07
+
+/* Extension pseudo-field
+ * The field contains a pointer to pb_extension_t */
+#define PB_LTYPE_EXTENSION 0x08
 
 /* Number of declared LTYPES */
-#define PB_LTYPES_COUNT 7
+#define PB_LTYPES_COUNT 9
 #define PB_LTYPE_MASK 0x0F
 
-/**************************
+/**** Field repetition rules ****/
- * Field repetition rules *
- **************************/
 
 #define PB_HTYPE_REQUIRED 0x00
 #define PB_HTYPE_OPTIONAL 0x10
 #define PB_HTYPE_REPEATED 0x20
 #define PB_HTYPE_MASK     0x30
 
-/********************
+/**** Field allocation types ****/
- * Allocation types *
- ********************/
  
 #define PB_ATYPE_STATIC   0x00
+#define PB_ATYPE_POINTER  0x80
 #define PB_ATYPE_CALLBACK 0x40
 #define PB_ATYPE_MASK     0xC0
 
@@ -105,6 +183,20 @@ typedef uint8_t pb_type_t;
 #define PB_HTYPE(x) ((x) & PB_HTYPE_MASK)
 #define PB_LTYPE(x) ((x) & PB_LTYPE_MASK)
 
+/* Data type used for storing sizes of struct fields
+ * and array counts.
+ */
+#if defined(PB_FIELD_32BIT)
+    typedef uint32_t pb_size_t;
+    typedef int32_t pb_ssize_t;
+#elif defined(PB_FIELD_16BIT)
+    typedef uint16_t pb_size_t;
+    typedef int16_t pb_ssize_t;
+#else
+    typedef uint8_t pb_size_t;
+    typedef int8_t pb_ssize_t;
+#endif
+
 /* This structure is used in auto-generated constants
  * to specify struct fields.
  * You can change field sizes if you need structures
@@ -113,47 +205,49 @@ typedef uint8_t pb_type_t;
  * structures. Fix that by defining PB_FIELD_16BIT or
  * PB_FIELD_32BIT.
  */
+PB_PACKED_STRUCT_START
 typedef struct _pb_field_t pb_field_t;
 struct _pb_field_t {
-
+    pb_size_t tag;
-#if !defined(PB_FIELD_16BIT) && !defined(PB_FIELD_32BIT)
-    uint8_t tag;
     pb_type_t type;
-    uint8_t data_offset; /* Offset of field data, relative to previous field. */
+    pb_size_t data_offset; /* Offset of field data, relative to previous field. */
-    int8_t size_offset; /* Offset of array size or has-boolean, relative to data */
+    pb_ssize_t size_offset; /* Offset of array size or has-boolean, relative to data */
-    uint8_t data_size; /* Data size in bytes for a single item */
+    pb_size_t data_size; /* Data size in bytes for a single item */
-    uint8_t array_size; /* Maximum number of entries in array */
+    pb_size_t array_size; /* Maximum number of entries in array */
-#elif defined(PB_FIELD_16BIT) && !defined(PB_FIELD_32BIT)
-    uint16_t tag;
-    pb_type_t type;
-    uint8_t data_offset;
-    int8_t size_offset;
-    uint16_t data_size;
-    uint16_t array_size;
-#else
-    uint32_t tag;
-    pb_type_t type;
-    uint8_t data_offset;
-    int8_t size_offset;
-    uint32_t data_size;
-    uint32_t array_size;
-#endif
     
     /* Field definitions for submessage
      * OR default value for all other non-array, non-callback types
      * If null, then field will zeroed. */
     const void *ptr;
 } pb_packed;
+PB_PACKED_STRUCT_END
+
+/* Make sure that the standard integer types are of the expected sizes.
+ * All kinds of things may break otherwise.. atleast all fixed* types.
+ *
+ * If you get errors here, it probably means that your stdint.h is not
+ * correct for your platform.
+ */
+STATIC_ASSERT(sizeof(int8_t) == 1, INT8_T_WRONG_SIZE)
+STATIC_ASSERT(sizeof(uint8_t) == 1, UINT8_T_WRONG_SIZE)
+STATIC_ASSERT(sizeof(int16_t) == 2, INT16_T_WRONG_SIZE)
+STATIC_ASSERT(sizeof(uint16_t) == 2, UINT16_T_WRONG_SIZE)
+STATIC_ASSERT(sizeof(int32_t) == 4, INT32_T_WRONG_SIZE)
+STATIC_ASSERT(sizeof(uint32_t) == 4, UINT32_T_WRONG_SIZE)
+STATIC_ASSERT(sizeof(int64_t) == 8, INT64_T_WRONG_SIZE)
+STATIC_ASSERT(sizeof(uint64_t) == 8, UINT64_T_WRONG_SIZE)
 
 /* This structure is used for 'bytes' arrays.
  * It has the number of bytes in the beginning, and after that an array.
  * Note that actual structs used will have a different length of bytes array.
  */
+#define PB_BYTES_ARRAY_T(n) struct { size_t size; uint8_t bytes[n]; }
+#define PB_BYTES_ARRAY_T_ALLOCSIZE(n) ((size_t)n + offsetof(pb_bytes_array_t, bytes))
+
 struct _pb_bytes_array_t {
     size_t size;
     uint8_t bytes[1];
 };
-
 typedef struct _pb_bytes_array_t pb_bytes_array_t;
 
 /* This structure is used for giving the callback function.
@@ -178,10 +272,19 @@ typedef struct _pb_istream_t pb_istream_t;
 typedef struct _pb_ostream_t pb_ostream_t;
 typedef struct _pb_callback_t pb_callback_t;
 struct _pb_callback_t {
+#ifdef PB_OLD_CALLBACK_STYLE
+    /* Deprecated since nanopb-0.2.1 */
     union {
         bool (*decode)(pb_istream_t *stream, const pb_field_t *field, void *arg);
         bool (*encode)(pb_ostream_t *stream, const pb_field_t *field, const void *arg);
     } funcs;
+#else
+    /* New function signature, which allows modifying arg contents in callback. */
+    union {
+        bool (*decode)(pb_istream_t *stream, const pb_field_t *field, void **arg);
+        bool (*encode)(pb_ostream_t *stream, const pb_field_t *field, void * const *arg);
+    } funcs;
+#endif    
     
     /* Free arg for use by callback */
     void *arg;
@@ -195,50 +298,144 @@ typedef enum {
     PB_WT_32BIT  = 5
 } pb_wire_type_t;
 
+/* Structure for defining the handling of unknown/extension fields.
+ * Usually the pb_extension_type_t structure is automatically generated,
+ * while the pb_extension_t structure is created by the user. However,
+ * if you want to catch all unknown fields, you can also create a custom
+ * pb_extension_type_t with your own callback.
+ */
+typedef struct _pb_extension_type_t pb_extension_type_t;
+typedef struct _pb_extension_t pb_extension_t;
+struct _pb_extension_type_t {
+    /* Called for each unknown field in the message.
+     * If you handle the field, read off all of its data and return true.
+     * If you do not handle the field, do not read anything and return true.
+     * If you run into an error, return false.
+     * Set to NULL for default handler.
+     */
+    bool (*decode)(pb_istream_t *stream, pb_extension_t *extension,
+                   uint32_t tag, pb_wire_type_t wire_type);
+    
+    /* Called once after all regular fields have been encoded.
+     * If you have something to write, do so and return true.
+     * If you do not have anything to write, just return true.
+     * If you run into an error, return false.
+     * Set to NULL for default handler.
+     */
+    bool (*encode)(pb_ostream_t *stream, const pb_extension_t *extension);
+    
+    /* Free field for use by the callback. */
+    const void *arg;
+};
+
+struct _pb_extension_t {
+    /* Type describing the extension field. Usually you'll initialize
+     * this to a pointer to the automatically generated structure. */
+    const pb_extension_type_t *type;
+    
+    /* Destination for the decoded data. This must match the datatype
+     * of the extension field. */
+    void *dest;
+    
+    /* Pointer to the next extension handler, or NULL.
+     * If this extension does not match a field, the next handler is
+     * automatically called. */
+    pb_extension_t *next;
+};
+
+/* Memory allocation functions to use. You can define pb_realloc and
+ * pb_free to custom functions if you want. */
+#ifdef PB_ENABLE_MALLOC
+#   ifndef pb_realloc
+#       define pb_realloc(ptr, size) realloc(ptr, size)
+#   endif
+#   ifndef pb_free
+#       define pb_free(ptr) free(ptr)
+#   endif
+#endif
+
 /* These macros are used to declare pb_field_t's in the constant array. */
+/* Size of a structure member, in bytes. */
 #define pb_membersize(st, m) (sizeof ((st*)0)->m)
+/* Number of entries in an array. */
 #define pb_arraysize(st, m) (pb_membersize(st, m) / pb_membersize(st, m[0]))
+/* Delta from start of one member to the start of another member. */
 #define pb_delta(st, m1, m2) ((int)offsetof(st, m1) - (int)offsetof(st, m2))
-#define pb_delta_end(st, m1, m2) (offsetof(st, m1) == offsetof(st, m2) \
+/* Marks the end of the field list */
-                                  ? offsetof(st, m1) \
-                                  : offsetof(st, m1) - offsetof(st, m2) - pb_membersize(st, m2))
 #define PB_LAST_FIELD {0,(pb_type_t) 0,0,0,0,0,0}
 
+/* Macros for filling in the data_offset field */
+/* data_offset for first field in a message */
+#define PB_DATAOFFSET_FIRST(st, m1, m2) (offsetof(st, m1))
+/* data_offset for subsequent fields */
+#define PB_DATAOFFSET_OTHER(st, m1, m2) (offsetof(st, m1) - offsetof(st, m2) - pb_membersize(st, m2))
+/* Choose first/other based on m1 == m2 (deprecated, remains for backwards compatibility) */
+#define PB_DATAOFFSET_CHOOSE(st, m1, m2) (int)(offsetof(st, m1) == offsetof(st, m2) \
+                                  ? PB_DATAOFFSET_FIRST(st, m1, m2) \
+                                  : PB_DATAOFFSET_OTHER(st, m1, m2))
+
 /* Required fields are the simplest. They just have delta (padding) from
  * previous field end, and the size of the field. Pointer is used for
  * submessages and default values.
  */
-#define PB_REQUIRED_STATIC(tag, st, m, pm, ltype, ptr) \
+#define PB_REQUIRED_STATIC(tag, st, m, fd, ltype, ptr) \
     {tag, PB_ATYPE_STATIC | PB_HTYPE_REQUIRED | ltype, \
-    pb_delta_end(st, m, pm), 0, pb_membersize(st, m), 0, ptr}
+    fd, 0, pb_membersize(st, m), 0, ptr}
 
 /* Optional fields add the delta to the has_ variable. */
-#define PB_OPTIONAL_STATIC(tag, st, m, pm, ltype, ptr) \
+#define PB_OPTIONAL_STATIC(tag, st, m, fd, ltype, ptr) \
     {tag, PB_ATYPE_STATIC | PB_HTYPE_OPTIONAL | ltype, \
-    pb_delta_end(st, m, pm), \
+    fd, \
     pb_delta(st, has_ ## m, m), \
     pb_membersize(st, m), 0, ptr}
 
 /* Repeated fields have a _count field and also the maximum number of entries. */
-#define PB_REPEATED_STATIC(tag, st, m, pm, ltype, ptr) \
+#define PB_REPEATED_STATIC(tag, st, m, fd, ltype, ptr) \
     {tag, PB_ATYPE_STATIC | PB_HTYPE_REPEATED | ltype, \
-    pb_delta_end(st, m, pm), \
+    fd, \
     pb_delta(st, m ## _count, m), \
     pb_membersize(st, m[0]), \
     pb_arraysize(st, m), ptr}
 
+/* Allocated fields carry the size of the actual data, not the pointer */
+#define PB_REQUIRED_POINTER(tag, st, m, fd, ltype, ptr) \
+    {tag, PB_ATYPE_POINTER | PB_HTYPE_REQUIRED | ltype, \
+    fd, 0, pb_membersize(st, m[0]), 0, ptr}
+
+/* Optional fields don't need a has_ variable, as information would be redundant */
+#define PB_OPTIONAL_POINTER(tag, st, m, fd, ltype, ptr) \
+    {tag, PB_ATYPE_POINTER | PB_HTYPE_OPTIONAL | ltype, \
+    fd, 0, pb_membersize(st, m[0]), 0, ptr}
+
+/* Repeated fields have a _count field and a pointer to array of pointers */
+#define PB_REPEATED_POINTER(tag, st, m, fd, ltype, ptr) \
+    {tag, PB_ATYPE_POINTER | PB_HTYPE_REPEATED | ltype, \
+    fd, pb_delta(st, m ## _count, m), \
+    pb_membersize(st, m[0]), 0, ptr}
+
 /* Callbacks are much like required fields except with special datatype. */
-#define PB_REQUIRED_CALLBACK(tag, st, m, pm, ltype, ptr) \
+#define PB_REQUIRED_CALLBACK(tag, st, m, fd, ltype, ptr) \
     {tag, PB_ATYPE_CALLBACK | PB_HTYPE_REQUIRED | ltype, \
-    pb_delta_end(st, m, pm), 0, pb_membersize(st, m), 0, ptr}
+    fd, 0, pb_membersize(st, m), 0, ptr}
 
-#define PB_OPTIONAL_CALLBACK(tag, st, m, pm, ltype, ptr) \
+#define PB_OPTIONAL_CALLBACK(tag, st, m, fd, ltype, ptr) \
     {tag, PB_ATYPE_CALLBACK | PB_HTYPE_OPTIONAL | ltype, \
-    pb_delta_end(st, m, pm), 0, pb_membersize(st, m), 0, ptr}
+    fd, 0, pb_membersize(st, m), 0, ptr}
     
-#define PB_REPEATED_CALLBACK(tag, st, m, pm, ltype, ptr) \
+#define PB_REPEATED_CALLBACK(tag, st, m, fd, ltype, ptr) \
     {tag, PB_ATYPE_CALLBACK | PB_HTYPE_REPEATED | ltype, \
-    pb_delta_end(st, m, pm), 0, pb_membersize(st, m), 0, ptr}
+    fd, 0, pb_membersize(st, m), 0, ptr}
+
+/* Optional extensions don't have the has_ field, as that would be redundant. */
+#define PB_OPTEXT_STATIC(tag, st, m, fd, ltype, ptr) \
+    {tag, PB_ATYPE_STATIC | PB_HTYPE_OPTIONAL | ltype, \
+    0, \
+    0, \
+    pb_membersize(st, m), 0, ptr}
+
+#define PB_OPTEXT_CALLBACK(tag, st, m, fd, ltype, ptr) \
+    {tag, PB_ATYPE_CALLBACK | PB_HTYPE_OPTIONAL | ltype, \
+    0, 0, pb_membersize(st, m), 0, ptr}
 
 /* The mapping from protobuf types to LTYPEs is done using these macros. */
 #define PB_LTYPE_MAP_BOOL       PB_LTYPE_VARINT
@@ -256,15 +453,16 @@ typedef enum {
 #define PB_LTYPE_MAP_SINT32     PB_LTYPE_SVARINT
 #define PB_LTYPE_MAP_SINT64     PB_LTYPE_SVARINT
 #define PB_LTYPE_MAP_STRING     PB_LTYPE_STRING
-#define PB_LTYPE_MAP_UINT32     PB_LTYPE_VARINT
+#define PB_LTYPE_MAP_UINT32     PB_LTYPE_UVARINT
-#define PB_LTYPE_MAP_UINT64     PB_LTYPE_VARINT
+#define PB_LTYPE_MAP_UINT64     PB_LTYPE_UVARINT
+#define PB_LTYPE_MAP_EXTENSION  PB_LTYPE_EXTENSION
 
 /* This is the actual macro used in field descriptions.
  * It takes these arguments:
  * - Field tag number
  * - Field type:   BOOL, BYTES, DOUBLE, ENUM, FIXED32, FIXED64,
  *                 FLOAT, INT32, INT64, MESSAGE, SFIXED32, SFIXED64
- *                 SINT32, SINT64, STRING, UINT32 or UINT64
+ *                 SINT32, SINT64, STRING, UINT32, UINT64 or EXTENSION
  * - Field rules:  REQUIRED, OPTIONAL or REPEATED
  * - Allocation:   STATIC or CALLBACK
  * - Message name
@@ -274,9 +472,23 @@ typedef enum {
  */
 
 #define PB_FIELD(tag, type, rules, allocation, message, field, prevfield, ptr) \
-    PB_ ## rules ## _ ## allocation(tag, message, field, prevfield, \
+    PB_ ## rules ## _ ## allocation(tag, message, field, \
+        PB_DATAOFFSET_CHOOSE(message, field, prevfield), \
         PB_LTYPE_MAP_ ## type, ptr)
 
+/* This is a new version of the macro used by nanopb generator from
+ * version 0.2.3 onwards. It avoids the use of a ternary expression in
+ * the initialization, which confused some compilers.
+ *
+ * - Placement: FIRST or OTHER, depending on if this is the first field in structure.
+ *
+ */
+#define PB_FIELD2(tag, type, rules, allocation, placement, message, field, prevfield, ptr) \
+    PB_ ## rules ## _ ## allocation(tag, message, field, \
+        PB_DATAOFFSET_ ## placement(message, field, prevfield), \
+        PB_LTYPE_MAP_ ## type, ptr)
+
+
 /* These macros are used for giving out error messages.
  * They are mostly a debugging aid; the main error information
  * is the true/false return value from functions.

pb_decode.c

@@ -3,38 +3,80 @@
  * 2011 Petteri Aimonen <jpa@kapsi.fi>
  */
 
-/* The warn_unused_result attribute appeared first in gcc-3.4.0 */
+/* Use the GCC warn_unused_result attribute to check that all return values
+ * are propagated correctly. On other compilers and gcc before 3.4.0 just
+ * ignore the annotation.
+ */
 #if !defined(__GNUC__) || ( __GNUC__ < 3) || (__GNUC__ == 3 && __GNUC_MINOR__ < 4)
     #define checkreturn
 #else
-    /* Verify that we remember to check all return values for proper error propagation */
     #define checkreturn __attribute__((warn_unused_result))
 #endif
 
 #define NANOPB_INTERNALS
 #include "pb.h"
 #include "pb_decode.h"
-#include <string.h>
+
+/**************************************
+ * Declarations internal to this file *
+ **************************************/
+
+/* Iterator for pb_field_t list */
+typedef struct {
+    const pb_field_t *start; /* Start of the pb_field_t array */
+    const pb_field_t *pos; /* Current position of the iterator */
+    unsigned field_index; /* Zero-based index of the field. */
+    unsigned required_field_index; /* Zero-based index that counts only the required fields */
+    void *dest_struct; /* Pointer to the destination structure to decode to */
+    void *pData; /* Pointer where to store current field value */
+    void *pSize; /* Pointer where to store the size of current array field */
+} pb_field_iterator_t;
 
 typedef bool (*pb_decoder_t)(pb_istream_t *stream, const pb_field_t *field, void *dest) checkreturn;
 
+static bool checkreturn buf_read(pb_istream_t *stream, uint8_t *buf, size_t count);
+static bool checkreturn pb_decode_varint32(pb_istream_t *stream, uint32_t *dest);
+static bool checkreturn read_raw_value(pb_istream_t *stream, pb_wire_type_t wire_type, uint8_t *buf, size_t *size);
+static void pb_field_init(pb_field_iterator_t *iter, const pb_field_t *fields, void *dest_struct);
+static bool pb_field_next(pb_field_iterator_t *iter);
+static bool checkreturn pb_field_find(pb_field_iterator_t *iter, uint32_t tag);
+static bool checkreturn decode_static_field(pb_istream_t *stream, pb_wire_type_t wire_type, pb_field_iterator_t *iter);
+static bool checkreturn decode_callback_field(pb_istream_t *stream, pb_wire_type_t wire_type, pb_field_iterator_t *iter);
+static bool checkreturn decode_field(pb_istream_t *stream, pb_wire_type_t wire_type, pb_field_iterator_t *iter);
+static bool checkreturn default_extension_decoder(pb_istream_t *stream, pb_extension_t *extension, uint32_t tag, pb_wire_type_t wire_type);
+static bool checkreturn decode_extension(pb_istream_t *stream, uint32_t tag, pb_wire_type_t wire_type, pb_field_iterator_t *iter);
+static bool checkreturn find_extension_field(pb_field_iterator_t *iter);
+static void pb_message_set_to_defaults(const pb_field_t fields[], void *dest_struct);
+static bool checkreturn pb_dec_varint(pb_istream_t *stream, const pb_field_t *field, void *dest);
+static bool checkreturn pb_dec_uvarint(pb_istream_t *stream, const pb_field_t *field, void *dest);
+static bool checkreturn pb_dec_svarint(pb_istream_t *stream, const pb_field_t *field, void *dest);
+static bool checkreturn pb_dec_fixed32(pb_istream_t *stream, const pb_field_t *field, void *dest);
+static bool checkreturn pb_dec_fixed64(pb_istream_t *stream, const pb_field_t *field, void *dest);
+static bool checkreturn pb_dec_bytes(pb_istream_t *stream, const pb_field_t *field, void *dest);
+static bool checkreturn pb_dec_string(pb_istream_t *stream, const pb_field_t *field, void *dest);
+static bool checkreturn pb_dec_submessage(pb_istream_t *stream, const pb_field_t *field, void *dest);
+static bool checkreturn pb_skip_varint(pb_istream_t *stream);
+static bool checkreturn pb_skip_string(pb_istream_t *stream);
+
 /* --- Function pointers to field decoders ---
  * Order in the array must match pb_action_t LTYPE numbering.
  */
 static const pb_decoder_t PB_DECODERS[PB_LTYPES_COUNT] = {
     &pb_dec_varint,
+    &pb_dec_uvarint,
     &pb_dec_svarint,
     &pb_dec_fixed32,
     &pb_dec_fixed64,
     
     &pb_dec_bytes,
     &pb_dec_string,
-    &pb_dec_submessage
+    &pb_dec_submessage,
+    NULL /* extensions */
 };
 
-/**************
+/*******************************
- * pb_istream *
+ * pb_istream_t implementation *
- **************/
+ *******************************/
 
 static bool checkreturn buf_read(pb_istream_t *stream, uint8_t *buf, size_t count)
 {
@@ -84,6 +126,26 @@ bool checkreturn pb_read(pb_istream_t *stream, uint8_t *buf, size_t count)
     return true;
 }
 
+/* Read a single byte from input stream. buf may not be NULL.
+ * This is an optimization for the varint decoding. */
+static bool checkreturn pb_readbyte(pb_istream_t *stream, uint8_t *buf)
+{
+    if (!stream->bytes_left)
+        PB_RETURN_ERROR(stream, "end-of-stream");
+
+#ifndef PB_BUFFER_ONLY
+    if (!stream->callback(stream, buf, 1))
+        PB_RETURN_ERROR(stream, "io error");
+#else
+    *buf = *(uint8_t*)stream->state;
+    stream->state = (uint8_t*)stream->state + 1;
+#endif
+
+    stream->bytes_left--;
+    
+    return true;    
+}
+
 pb_istream_t pb_istream_from_buffer(uint8_t *buf, size_t bufsize)
 {
     pb_istream_t stream;
@@ -109,7 +171,7 @@ static bool checkreturn pb_decode_varint32(pb_istream_t *stream, uint32_t *dest)
     uint8_t byte;
     uint32_t result;
     
-    if (!pb_read(stream, &byte, 1))
+    if (!pb_readbyte(stream, &byte))
         return false;
     
     if (!(byte & 0x80))
@@ -128,7 +190,7 @@ static bool checkreturn pb_decode_varint32(pb_istream_t *stream, uint32_t *dest)
             if (bitpos >= 32)
                 PB_RETURN_ERROR(stream, "varint overflow");
             
-            if (!pb_read(stream, &byte, 1))
+            if (!pb_readbyte(stream, &byte))
                 return false;
             
             result |= (uint32_t)(byte & 0x7F) << bitpos;
@@ -151,7 +213,7 @@ bool checkreturn pb_decode_varint(pb_istream_t *stream, uint64_t *dest)
         if (bitpos >= 64)
             PB_RETURN_ERROR(stream, "varint overflow");
         
-        if (!pb_read(stream, &byte, 1))
+        if (!pb_readbyte(stream, &byte))
             return false;
 
         result |= (uint64_t)(byte & 0x7F) << bitpos;
@@ -277,46 +339,42 @@ void pb_close_string_substream(pb_istream_t *stream, pb_istream_t *substream)
 #endif
 }
 
-/* Iterator for pb_field_t list */
-typedef struct {
-    const pb_field_t *start; /* Start of the pb_field_t array */
-    const pb_field_t *current; /* Current position of the iterator */
-    unsigned field_index; /* Zero-based index of the field. */
-    unsigned required_field_index; /* Zero-based index that counts only the required fields */
-    void *dest_struct; /* Pointer to the destination structure to decode to */
-    void *pData; /* Pointer where to store current field value */
-    void *pSize; /* Pointer where to store the size of current array field */
-} pb_field_iterator_t;
-
 static void pb_field_init(pb_field_iterator_t *iter, const pb_field_t *fields, void *dest_struct)
 {
-    iter->start = iter->current = fields;
+    iter->start = iter->pos = fields;
     iter->field_index = 0;
     iter->required_field_index = 0;
-    iter->pData = (char*)dest_struct + iter->current->data_offset;
+    iter->pData = (char*)dest_struct + iter->pos->data_offset;
-    iter->pSize = (char*)iter->pData + iter->current->size_offset;
+    iter->pSize = (char*)iter->pData + iter->pos->size_offset;
     iter->dest_struct = dest_struct;
 }
 
 static bool pb_field_next(pb_field_iterator_t *iter)
 {
     bool notwrapped = true;
-    size_t prev_size = iter->current->data_size;
+    size_t prev_size = iter->pos->data_size;
     
-    if (PB_ATYPE(iter->current->type) == PB_ATYPE_STATIC &&
+    if (PB_ATYPE(iter->pos->type) == PB_ATYPE_STATIC &&
-        PB_HTYPE(iter->current->type) == PB_HTYPE_REPEATED)
+        PB_HTYPE(iter->pos->type) == PB_HTYPE_REPEATED)
     {
-        prev_size *= iter->current->array_size;
+        prev_size *= iter->pos->array_size;
+    }
+    else if (PB_ATYPE(iter->pos->type) == PB_ATYPE_POINTER)
+    {
+        prev_size = sizeof(void*);
     }
     
-    if (PB_HTYPE(iter->current->type) == PB_HTYPE_REQUIRED)
+    if (iter->pos->tag == 0)
+        return false; /* Only happens with empty message types */
+    
+    if (PB_HTYPE(iter->pos->type) == PB_HTYPE_REQUIRED)
         iter->required_field_index++;
     
-    iter->current++;
+    iter->pos++;
     iter->field_index++;
-    if (iter->current->tag == 0)
+    if (iter->pos->tag == 0)
     {
-        iter->current = iter->start;
+        iter->pos = iter->start;
         iter->field_index = 0;
         iter->required_field_index = 0;
         iter->pData = iter->dest_struct;
@@ -324,8 +382,8 @@ static bool pb_field_next(pb_field_iterator_t *iter)
         notwrapped = false;
     }
     
-    iter->pData = (char*)iter->pData + prev_size + iter->current->data_offset;
+    iter->pData = (char*)iter->pData + prev_size + iter->pos->data_offset;
-    iter->pSize = (char*)iter->pData + iter->current->size_offset;
+    iter->pSize = (char*)iter->pData + iter->pos->size_offset;
     return notwrapped;
 }
 
@@ -334,8 +392,11 @@ static bool checkreturn pb_field_find(pb_field_iterator_t *iter, uint32_t tag)
     unsigned start = iter->field_index;
     
     do {
-        if (iter->current->tag == tag)
+        if (iter->pos->tag == tag &&
+            PB_LTYPE(iter->pos->type) != PB_LTYPE_EXTENSION)
+        {
             return true;
+        }
         pb_field_next(iter);
     } while (iter->field_index != start);
     
@@ -351,17 +412,17 @@ static bool checkreturn decode_static_field(pb_istream_t *stream, pb_wire_type_t
     pb_type_t type;
     pb_decoder_t func;
     
-    type = iter->current->type;
+    type = iter->pos->type;
     func = PB_DECODERS[PB_LTYPE(type)];
 
     switch (PB_HTYPE(type))
     {
         case PB_HTYPE_REQUIRED:
-            return func(stream, iter->current, iter->pData);
+            return func(stream, iter->pos, iter->pData);
             
         case PB_HTYPE_OPTIONAL:
             *(bool*)iter->pSize = true;
-            return func(stream, iter->current, iter->pData);
+            return func(stream, iter->pos, iter->pData);
     
         case PB_HTYPE_REPEATED:
             if (wire_type == PB_WT_STRING
@@ -374,10 +435,10 @@ static bool checkreturn decode_static_field(pb_istream_t *stream, pb_wire_type_t
                 if (!pb_make_string_substream(stream, &substream))
                     return false;
                 
-                while (substream.bytes_left && *size < iter->current->array_size)
+                while (substream.bytes_left && *size < iter->pos->array_size)
                 {
-                    void *pItem = (uint8_t*)iter->pData + iter->current->data_size * (*size);
+                    void *pItem = (uint8_t*)iter->pData + iter->pos->data_size * (*size);
-                    if (!func(&substream, iter->current, pItem))
+                    if (!func(&substream, iter->pos, pItem))
                     {
                         status = false;
                         break;
@@ -395,12 +456,12 @@ static bool checkreturn decode_static_field(pb_istream_t *stream, pb_wire_type_t
             {
                 /* Repeated field */
                 size_t *size = (size_t*)iter->pSize;
-                void *pItem = (uint8_t*)iter->pData + iter->current->data_size * (*size);
+                void *pItem = (uint8_t*)iter->pData + iter->pos->data_size * (*size);
-                if (*size >= iter->current->array_size)
+                if (*size >= iter->pos->array_size)
                     PB_RETURN_ERROR(stream, "array overflow");
                 
                 (*size)++;
-                return func(stream, iter->current, pItem);
+                return func(stream, iter->pos, pItem);
             }
 
         default:
@@ -408,10 +469,146 @@ static bool checkreturn decode_static_field(pb_istream_t *stream, pb_wire_type_t
     }
 }
 
+#ifdef PB_ENABLE_MALLOC
+/* Allocate storage for the field and store the pointer at iter->pData.
+ * array_size is the number of entries to reserve in an array. */
+static bool checkreturn allocate_field(pb_istream_t *stream, void *pData, size_t data_size, size_t array_size)
+{    
+    void *ptr = *(void**)pData;
+    size_t size = array_size * data_size;
+    
+    /* Allocate new or expand previous allocation */
+    /* Note: on failure the old pointer will remain in the structure,
+     * the message must be freed by caller also on error return. */
+    ptr = pb_realloc(ptr, size);
+    if (ptr == NULL)
+        PB_RETURN_ERROR(stream, "realloc failed");
+    
+    *(void**)pData = ptr;
+    return true;
+}
+
+/* Clear a newly allocated item in case it contains a pointer, or is a submessage. */
+static void initialize_pointer_field(void *pItem, pb_field_iterator_t *iter)
+{
+    if (PB_LTYPE(iter->pos->type) == PB_LTYPE_STRING ||
+        PB_LTYPE(iter->pos->type) == PB_LTYPE_BYTES)
+    {
+        *(void**)pItem = NULL;
+    }
+    else if (PB_LTYPE(iter->pos->type) == PB_LTYPE_SUBMESSAGE)
+    {
+        pb_message_set_to_defaults((const pb_field_t *) iter->pos->ptr, pItem);
+    }
+}
+#endif
+
+static bool checkreturn decode_pointer_field(pb_istream_t *stream, pb_wire_type_t wire_type, pb_field_iterator_t *iter)
+{
+#ifndef PB_ENABLE_MALLOC
+    UNUSED(wire_type);
+    UNUSED(iter);
+    PB_RETURN_ERROR(stream, "no malloc support");
+#else
+    pb_type_t type;
+    pb_decoder_t func;
+    
+    type = iter->pos->type;
+    func = PB_DECODERS[PB_LTYPE(type)];
+    
+    switch (PB_HTYPE(type))
+    {
+        case PB_HTYPE_REQUIRED:
+        case PB_HTYPE_OPTIONAL:
+            if (PB_LTYPE(type) == PB_LTYPE_STRING ||
+                PB_LTYPE(type) == PB_LTYPE_BYTES)
+            {
+                return func(stream, iter->pos, iter->pData);
+            }
+            else
+            {
+                if (!allocate_field(stream, iter->pData, iter->pos->data_size, 1))
+                    return false;
+                
+                initialize_pointer_field(*(void**)iter->pData, iter);
+                return func(stream, iter->pos, *(void**)iter->pData);
+            }
+    
+        case PB_HTYPE_REPEATED:
+            if (wire_type == PB_WT_STRING
+                && PB_LTYPE(type) <= PB_LTYPE_LAST_PACKABLE)
+            {
+                /* Packed array, multiple items come in at once. */
+                bool status = true;
+                size_t *size = (size_t*)iter->pSize;
+                size_t allocated_size = *size;
+                void *pItem;
+                pb_istream_t substream;
+                
+                if (!pb_make_string_substream(stream, &substream))
+                    return false;
+                
+                while (substream.bytes_left)
+                {
+                    if (*size + 1 > allocated_size)
+                    {
+                        /* Allocate more storage. This tries to guess the
+                         * number of remaining entries. Round the division
+                         * upwards. */
+                        allocated_size += (substream.bytes_left - 1) / iter->pos->data_size + 1;
+                        
+                        if (!allocate_field(&substream, iter->pData, iter->pos->data_size, allocated_size))
+                        {
+                            status = false;
+                            break;
+                        }
+                    }
+
+                    /* Decode the array entry */
+                    pItem = *(uint8_t**)iter->pData + iter->pos->data_size * (*size);
+                    initialize_pointer_field(pItem, iter);
+                    if (!func(&substream, iter->pos, pItem))
+                    {
+                        status = false;
+                        break;
+                    }
+                    (*size)++;
+                }
+                pb_close_string_substream(stream, &substream);
+                
+                return status;
+            }
+            else
+            {
+                /* Normal repeated field, i.e. only one item at a time. */
+                size_t *size = (size_t*)iter->pSize;
+                void *pItem;
+                
+                (*size)++;
+                if (!allocate_field(stream, iter->pData, iter->pos->data_size, *size))
+                    return false;
+            
+                pItem = *(uint8_t**)iter->pData + iter->pos->data_size * (*size - 1);
+                initialize_pointer_field(pItem, iter);
+                return func(stream, iter->pos, pItem);
+            }
+            
+        default:
+            PB_RETURN_ERROR(stream, "invalid field type");
+    }
+#endif
+}
+
 static bool checkreturn decode_callback_field(pb_istream_t *stream, pb_wire_type_t wire_type, pb_field_iterator_t *iter)
 {
     pb_callback_t *pCallback = (pb_callback_t*)iter->pData;
     
+#ifdef PB_OLD_CALLBACK_STYLE
+    void *arg = pCallback->arg;
+#else
+    void **arg = &(pCallback->arg);
+#endif
+    
     if (pCallback->funcs.decode == NULL)
         return pb_skip_field(stream, wire_type);
     
@@ -422,11 +619,11 @@ static bool checkreturn decode_callback_field(pb_istream_t *stream, pb_wire_type
         if (!pb_make_string_substream(stream, &substream))
             return false;
         
-        while (substream.bytes_left)
+        do
         {
-            if (!pCallback->funcs.decode(&substream, iter->current, pCallback->arg))
+            if (!pCallback->funcs.decode(&substream, iter->pos, arg))
                 PB_RETURN_ERROR(stream, "callback failed");
-        }
+        } while (substream.bytes_left);
         
         pb_close_string_substream(stream, &substream);
         return true;
@@ -445,17 +642,20 @@ static bool checkreturn decode_callback_field(pb_istream_t *stream, pb_wire_type
             return false;
         substream = pb_istream_from_buffer(buffer, size);
         
-        return pCallback->funcs.decode(&substream, iter->current, pCallback->arg);
+        return pCallback->funcs.decode(&substream, iter->pos, arg);
     }
 }
 
 static bool checkreturn decode_field(pb_istream_t *stream, pb_wire_type_t wire_type, pb_field_iterator_t *iter)
 {
-    switch (PB_ATYPE(iter->current->type))
+    switch (PB_ATYPE(iter->pos->type))
     {
         case PB_ATYPE_STATIC:
             return decode_static_field(stream, wire_type, iter);
         
+        case PB_ATYPE_POINTER:
+            return decode_pointer_field(stream, wire_type, iter);
+        
         case PB_ATYPE_CALLBACK:
             return decode_callback_field(stream, wire_type, iter);
         
@@ -464,51 +664,130 @@ static bool checkreturn decode_field(pb_istream_t *stream, pb_wire_type_t wire_t
     }
 }
 
+/* Default handler for extension fields. Expects a pb_field_t structure
+ * in extension->type->arg. */
+static bool checkreturn default_extension_decoder(pb_istream_t *stream,
+    pb_extension_t *extension, uint32_t tag, pb_wire_type_t wire_type)
+{
+    const pb_field_t *field = (const pb_field_t*)extension->type->arg;
+    pb_field_iterator_t iter;
+    bool dummy;
+    
+    if (field->tag != tag)
+        return true;
+    
+    iter.start = field;
+    iter.pos = field;
+    iter.field_index = 0;
+    iter.required_field_index = 0;
+    iter.dest_struct = extension->dest;
+    iter.pData = extension->dest;
+    iter.pSize = &dummy;
+    
+    return decode_field(stream, wire_type, &iter);
+}
+
+/* Try to decode an unknown field as an extension field. Tries each extension
+ * decoder in turn, until one of them handles the field or loop ends. */
+static bool checkreturn decode_extension(pb_istream_t *stream,
+    uint32_t tag, pb_wire_type_t wire_type, pb_field_iterator_t *iter)
+{
+    pb_extension_t *extension = *(pb_extension_t* const *)iter->pData;
+    size_t pos = stream->bytes_left;
+    
+    while (extension && pos == stream->bytes_left)
+    {
+        bool status;
+        if (extension->type->decode)
+            status = extension->type->decode(stream, extension, tag, wire_type);
+        else
+            status = default_extension_decoder(stream, extension, tag, wire_type);
+
+        if (!status)
+            return false;
+        
+        extension = extension->next;
+    }
+    
+    return true;
+}
+
+/* Step through the iterator until an extension field is found or until all
+ * entries have been checked. There can be only one extension field per
+ * message. Returns false if no extension field is found. */
+static bool checkreturn find_extension_field(pb_field_iterator_t *iter)
+{
+    unsigned start = iter->field_index;
+    
+    do {
+        if (PB_LTYPE(iter->pos->type) == PB_LTYPE_EXTENSION)
+            return true;
+        pb_field_next(iter);
+    } while (iter->field_index != start);
+    
+    return false;
+}
+
 /* Initialize message fields to default values, recursively */
 static void pb_message_set_to_defaults(const pb_field_t fields[], void *dest_struct)
 {
     pb_field_iterator_t iter;
     pb_field_init(&iter, fields, dest_struct);
     
-    /* Initialize size/has fields and apply default values */
     do
     {
         pb_type_t type;
-        type = iter.current->type;
+        type = iter.pos->type;
     
-        if (iter.current->tag == 0)
+        /* Avoid crash on empty message types (zero fields) */
+        if (iter.pos->tag == 0)
             continue;
         
         if (PB_ATYPE(type) == PB_ATYPE_STATIC)
         {
-            /* Initialize the size field for optional/repeated fields to 0. */
             if (PB_HTYPE(type) == PB_HTYPE_OPTIONAL)
             {
+                /* Set has_field to false. Still initialize the optional field
+                 * itself also. */
                 *(bool*)iter.pSize = false;
             }
             else if (PB_HTYPE(type) == PB_HTYPE_REPEATED)
             {
+                /* Set array count to 0, no need to initialize contents. */
                 *(size_t*)iter.pSize = 0;
-                continue; /* Array is empty, no need to initialize contents */
+                continue;
             }
             
-            /* Initialize field contents to default value */
+            if (PB_LTYPE(iter.pos->type) == PB_LTYPE_SUBMESSAGE)
-            if (PB_LTYPE(iter.current->type) == PB_LTYPE_SUBMESSAGE)
             {
-                pb_message_set_to_defaults((const pb_field_t *) iter.current->ptr, iter.pData);
+                /* Initialize submessage to defaults */
+                pb_message_set_to_defaults((const pb_field_t *) iter.pos->ptr, iter.pData);
             }
-            else if (iter.current->ptr != NULL)
+            else if (iter.pos->ptr != NULL)
             {
-                memcpy(iter.pData, iter.current->ptr, iter.current->data_size);
+                /* Initialize to default value */
+                memcpy(iter.pData, iter.pos->ptr, iter.pos->data_size);
             }
             else
             {
-                memset(iter.pData, 0, iter.current->data_size);
+                /* Initialize to zeros */
+                memset(iter.pData, 0, iter.pos->data_size);
+            }
+        }
+        else if (PB_ATYPE(type) == PB_ATYPE_POINTER)
+        {
+            /* Initialize the pointer to NULL. */
+            *(void**)iter.pData = NULL;
+            
+            /* Initialize array count to 0. */
+            if (PB_HTYPE(type) == PB_HTYPE_REPEATED)
+            {
+                *(size_t*)iter.pSize = 0;
             }
         }
         else if (PB_ATYPE(type) == PB_ATYPE_CALLBACK)
         {
-            continue; /* Don't overwrite callback */
+            /* Don't overwrite callback */
         }
     } while (pb_field_next(&iter));
 }
@@ -520,6 +799,7 @@ static void pb_message_set_to_defaults(const pb_field_t fields[], void *dest_str
 bool checkreturn pb_decode_noinit(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct)
 {
     uint8_t fields_seen[(PB_MAX_REQUIRED_FIELDS + 7) / 8] = {0}; /* Used to check for required fields */
+    uint32_t extension_range_start = 0;
     pb_field_iterator_t iter;
     
     pb_field_init(&iter, fields, dest_struct);
@@ -540,13 +820,36 @@ bool checkreturn pb_decode_noinit(pb_istream_t *stream, const pb_field_t fields[
         
         if (!pb_field_find(&iter, tag))
         {
+            /* No match found, check if it matches an extension. */
+            if (tag >= extension_range_start)
+            {
+                if (!find_extension_field(&iter))
+                    extension_range_start = (uint32_t)-1;
+                else
+                    extension_range_start = iter.pos->tag;
+                
+                if (tag >= extension_range_start)
+                {
+                    size_t pos = stream->bytes_left;
+                
+                    if (!decode_extension(stream, tag, wire_type, &iter))
+                        return false;
+                    
+                    if (pos != stream->bytes_left)
+                    {
+                        /* The field was handled */
+                        continue;                    
+                    }
+                }
+            }
+        
             /* No match found, skip data */
             if (!pb_skip_field(stream, wire_type))
                 return false;
             continue;
         }
         
-        if (PB_HTYPE(iter.current->type) == PB_HTYPE_REQUIRED
+        if (PB_HTYPE(iter.pos->type) == PB_HTYPE_REQUIRED
             && iter.required_field_index < PB_MAX_REQUIRED_FIELDS)
         {
             fields_seen[iter.required_field_index >> 3] |= (uint8_t)(1 << (iter.required_field_index & 7));
@@ -567,11 +870,11 @@ bool checkreturn pb_decode_noinit(pb_istream_t *stream, const pb_field_t fields[
         unsigned i;
         do {
             req_field_count = iter.required_field_index;
-            last_type = iter.current->type;
+            last_type = iter.pos->type;
         } while (pb_field_next(&iter));
         
         /* Fixup if last field was also required. */
-        if (PB_HTYPE(last_type) == PB_HTYPE_REQUIRED)
+        if (PB_HTYPE(last_type) == PB_HTYPE_REQUIRED && iter.pos->tag)
             req_field_count++;
         
         /* Check the whole bytes */
@@ -591,10 +894,87 @@ bool checkreturn pb_decode_noinit(pb_istream_t *stream, const pb_field_t fields[
 
 bool checkreturn pb_decode(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct)
 {
+    bool status;
     pb_message_set_to_defaults(fields, dest_struct);
-    return pb_decode_noinit(stream, fields, dest_struct);
+    status = pb_decode_noinit(stream, fields, dest_struct);
+    
+#ifdef PB_ENABLE_MALLOC
+    if (!status)
+        pb_release(fields, dest_struct);
+#endif
+    
+    return status;
 }
 
+bool pb_decode_delimited(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct)
+{
+    pb_istream_t substream;
+    bool status;
+    
+    if (!pb_make_string_substream(stream, &substream))
+        return false;
+    
+    status = pb_decode(&substream, fields, dest_struct);
+    pb_close_string_substream(stream, &substream);
+    return status;
+}
+
+#ifdef PB_ENABLE_MALLOC
+void pb_release(const pb_field_t fields[], void *dest_struct)
+{
+    pb_field_iterator_t iter;
+    pb_field_init(&iter, fields, dest_struct);
+    
+    do
+    {
+        pb_type_t type;
+        type = iter.pos->type;
+    
+        /* Avoid crash on empty message types (zero fields) */
+        if (iter.pos->tag == 0)
+            continue;
+        
+        if (PB_ATYPE(type) == PB_ATYPE_POINTER)
+        {
+            if (PB_HTYPE(type) == PB_HTYPE_REPEATED &&
+                (PB_LTYPE(type) == PB_LTYPE_STRING ||
+                 PB_LTYPE(type) == PB_LTYPE_BYTES))
+            {
+                /* Release entries in repeated string or bytes array */
+                void **pItem = *(void***)iter.pData;
+                size_t count = *(size_t*)iter.pSize;
+                while (count--)
+                {
+                    pb_free(*pItem);
+                    *pItem++ = NULL;
+                }
+            }
+            else if (PB_LTYPE(type) == PB_LTYPE_SUBMESSAGE)
+            {
+                /* Release fields in submessages */
+                void *pItem = *(void**)iter.pData;
+                size_t count = (pItem ? 1 : 0);
+                
+                if (PB_HTYPE(type) == PB_HTYPE_REPEATED)
+                {
+                    count = *(size_t*)iter.pSize;   
+                }
+                
+                while (count--)
+                {
+                    pb_release((const pb_field_t*)iter.pos->ptr, pItem);
+                    pItem = (uint8_t*)pItem + iter.pos->data_size;
+                }
+            }
+            
+            /* Release main item */
+            pb_free(*(void**)iter.pData);
+            *(void**)iter.pData = NULL;
+        }
+    } while (pb_field_next(&iter));
+}
+#endif
+
 /* Field decoders */
 
 bool pb_decode_svarint(pb_istream_t *stream, int64_t *dest)
@@ -656,24 +1036,42 @@ bool pb_decode_fixed64(pb_istream_t *stream, void *dest)
 bool checkreturn pb_dec_varint(pb_istream_t *stream, const pb_field_t *field, void *dest)
 {
     uint64_t value;
-    bool status = pb_decode_varint(stream, &value);
+    if (!pb_decode_varint(stream, &value))
+        return false;
+    
+    switch (field->data_size)
+    {
+        case 1: *(int8_t*)dest = (int8_t)value; break;
+        case 2: *(int16_t*)dest = (int16_t)value; break;
+        case 4: *(int32_t*)dest = (int32_t)value; break;
+        case 8: *(int64_t*)dest = (int64_t)value; break;
+        default: PB_RETURN_ERROR(stream, "invalid data_size");
+    }
+    
+    return true;
+}
+
+bool checkreturn pb_dec_uvarint(pb_istream_t *stream, const pb_field_t *field, void *dest)
+{
+    uint64_t value;
+    if (!pb_decode_varint(stream, &value))
+        return false;
     
     switch (field->data_size)
     {
-        case 1: *(uint8_t*)dest = (uint8_t)value; break;
-        case 2: *(uint16_t*)dest = (uint16_t)value; break;
         case 4: *(uint32_t*)dest = (uint32_t)value; break;
         case 8: *(uint64_t*)dest = value; break;
         default: PB_RETURN_ERROR(stream, "invalid data_size");
     }
     
-    return status;
+    return true;
 }
 
 bool checkreturn pb_dec_svarint(pb_istream_t *stream, const pb_field_t *field, void *dest)
 {
     int64_t value;
-    bool status = pb_decode_svarint(stream, &value);
+    if (!pb_decode_svarint(stream, &value))
+        return false;
     
     switch (field->data_size)
     {
@@ -682,7 +1080,7 @@ bool checkreturn pb_dec_svarint(pb_istream_t *stream, const pb_field_t *field, v
         default: PB_RETURN_ERROR(stream, "invalid data_size");
     }
     
-    return status;
+    return true;
 }
 
 bool checkreturn pb_dec_fixed32(pb_istream_t *stream, const pb_field_t *field, void *dest)
@@ -699,30 +1097,59 @@ bool checkreturn pb_dec_fixed64(pb_istream_t *stream, const pb_field_t *field, v
 
 bool checkreturn pb_dec_bytes(pb_istream_t *stream, const pb_field_t *field, void *dest)
 {
-    pb_bytes_array_t *x = (pb_bytes_array_t*)dest;
+    uint32_t size;
+    pb_bytes_array_t *bdest;
     
-    uint32_t temp;
+    if (!pb_decode_varint32(stream, &size))
-    if (!pb_decode_varint32(stream, &temp))
         return false;
-    x->size = temp;
     
-    /* Check length, noting the space taken by the size_t header. */
+    if (PB_ATYPE(field->type) == PB_ATYPE_POINTER)
-    if (x->size > field->data_size - offsetof(pb_bytes_array_t, bytes))
+    {
+#ifndef PB_ENABLE_MALLOC
+        PB_RETURN_ERROR(stream, "no malloc support");
+#else
+        if (!allocate_field(stream, dest, PB_BYTES_ARRAY_T_ALLOCSIZE(size), 1))
+            return false;
+        bdest = *(pb_bytes_array_t**)dest;
+#endif
+    }
+    else
+    {
+        if (PB_BYTES_ARRAY_T_ALLOCSIZE(size) > field->data_size)
             PB_RETURN_ERROR(stream, "bytes overflow");
+        bdest = (pb_bytes_array_t*)dest;
+    }
     
-    return pb_read(stream, x->bytes, x->size);
+    bdest->size = size;
+    return pb_read(stream, bdest->bytes, size);
 }
 
 bool checkreturn pb_dec_string(pb_istream_t *stream, const pb_field_t *field, void *dest)
 {
     uint32_t size;
+    size_t alloc_size;
     bool status;
     if (!pb_decode_varint32(stream, &size))
         return false;
     
-    /* Check length, noting the null terminator */
+    /* Space for null terminator */
-    if (size + 1 > field->data_size)
+    alloc_size = size + 1;
+    
+    if (PB_ATYPE(field->type) == PB_ATYPE_POINTER)
+    {
+#ifndef PB_ENABLE_MALLOC
+        PB_RETURN_ERROR(stream, "no malloc support");
+#else
+        if (!allocate_field(stream, dest, alloc_size, 1))
+            return false;
+        dest = *(void**)dest;
+#endif
+    }
+    else
+    {
+        if (alloc_size > field->data_size)
             PB_RETURN_ERROR(stream, "string overflow");
+    }
     
     status = pb_read(stream, (uint8_t*)dest, size);
     *((uint8_t*)dest + size) = 0;

pb_decode.h

@@ -1,31 +1,26 @@
+/* pb_decode.h: Functions to decode protocol buffers. Depends on pb_decode.c.
+ * The main function is pb_decode. You also need an input stream, and the
+ * field descriptions created by nanopb_generator.py.
+ */
+
 #ifndef _PB_DECODE_H_
 #define _PB_DECODE_H_
 
-/* pb_decode.h: Functions to decode protocol buffers. Depends on pb_decode.c.
- * The main function is pb_decode. You will also need to create an input
- * stream, which is easiest to do with pb_istream_from_buffer().
- * 
- * You also need structures and their corresponding pb_field_t descriptions.
- * These are usually generated from .proto-files with a script.
- */
-
-#include <stdbool.h>
 #include "pb.h"
 
 #ifdef __cplusplus
 extern "C" {
 #endif
 
-/* Lightweight input stream.
+/* Structure for defining custom input streams. You will need to provide
- * You can provide a callback function for reading or use
+ * a callback function to read the bytes from your storage, which can be
- * pb_istream_from_buffer.
+ * for example a file or a network socket.
+ * 
+ * The callback must conform to these rules:
  *
- * Rules for callback:
  * 1) Return false on IO errors. This will cause decoding to abort.
- * 
  * 2) You can use state to store your own data (e.g. buffer pointer),
  *    and rely on pb_read to verify that no-body reads past bytes_left.
- * 
  * 3) Your callback may be used with substreams, in which case bytes_left
  *    is different than from the main stream. Don't use bytes_left to compute
  *    any pointers.
@@ -50,24 +45,75 @@ struct _pb_istream_t
 #endif
 };
 
-pb_istream_t pb_istream_from_buffer(uint8_t *buf, size_t bufsize);
+/***************************
-bool pb_read(pb_istream_t *stream, uint8_t *buf, size_t count);
+ * Main decoding functions *
+ ***************************/
  
-/* Decode from stream to destination struct.
+/* Decode a single protocol buffers message from input stream into a C structure.
  * Returns true on success, false on any failure.
  * The actual struct pointed to by dest must match the description in fields.
+ * Callback fields of the destination structure must be initialized by caller.
+ * All other fields will be initialized by this function.
+ *
+ * Example usage:
+ *    MyMessage msg = {};
+ *    uint8_t buffer[64];
+ *    pb_istream_t stream;
+ *    
+ *    // ... read some data into buffer ...
+ *
+ *    stream = pb_istream_from_buffer(buffer, count);
+ *    pb_decode(&stream, MyMessage_fields, &msg);
  */
 bool pb_decode(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct);
 
 /* Same as pb_decode, except does not initialize the destination structure
  * to default values. This is slightly faster if you need no default values
  * and just do memset(struct, 0, sizeof(struct)) yourself.
+ *
+ * This can also be used for 'merging' two messages, i.e. update only the
+ * fields that exist in the new message.
+ *
+ * Note: If this function returns with an error, it will not release any
+ * dynamically allocated fields. You will need to call pb_release() yourself.
  */
 bool pb_decode_noinit(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct);
 
-/* --- Helper functions ---
+/* Same as pb_decode, except expects the stream to start with the message size
- * You may want to use these from your caller or callbacks.
+ * encoded as varint. Corresponds to parseDelimitedFrom() in Google's
+ * protobuf API.
  */
+bool pb_decode_delimited(pb_istream_t *stream, const pb_field_t fields[], void *dest_struct);
+
+#ifdef PB_ENABLE_MALLOC
+/* Release any allocated pointer fields. If you use dynamic allocation, you should
+ * call this for any successfully decoded message when you are done with it. If
+ * pb_decode() returns with an error, the message is already released.
+ */
+void pb_release(const pb_field_t fields[], void *dest_struct);
+#endif
+
+
+/**************************************
+ * Functions for manipulating streams *
+ **************************************/
+
+/* Create an input stream for reading from a memory buffer.
+ *
+ * Alternatively, you can use a custom stream that reads directly from e.g.
+ * a file or a network socket.
+ */
+pb_istream_t pb_istream_from_buffer(uint8_t *buf, size_t bufsize);
+
+/* Function to read from a pb_istream_t. You can use this if you need to
+ * read some custom header data, or to read data in field callbacks.
+ */
+bool pb_read(pb_istream_t *stream, uint8_t *buf, size_t count);
+
+
+/************************************************
+ * Helper functions for writing field callbacks *
+ ************************************************/
 
 /* Decode the tag for the next field in the stream. Gives the wire type and
  * field tag. At end of the message, returns false and sets eof to true. */
@@ -96,25 +142,6 @@ bool pb_decode_fixed64(pb_istream_t *stream, void *dest);
 bool pb_make_string_substream(pb_istream_t *stream, pb_istream_t *substream);
 void pb_close_string_substream(pb_istream_t *stream, pb_istream_t *substream);
 
-/* --- Internal functions ---
- * These functions are not terribly useful for the average library user, but
- * are exported to make the unit testing and extending nanopb easier.
- */
-
-#ifdef NANOPB_INTERNALS
-bool pb_dec_varint(pb_istream_t *stream, const pb_field_t *field, void *dest);
-bool pb_dec_svarint(pb_istream_t *stream, const pb_field_t *field, void *dest);
-bool pb_dec_fixed32(pb_istream_t *stream, const pb_field_t *field, void *dest);
-bool pb_dec_fixed64(pb_istream_t *stream, const pb_field_t *field, void *dest);
-
-bool pb_dec_bytes(pb_istream_t *stream, const pb_field_t *field, void *dest);
-bool pb_dec_string(pb_istream_t *stream, const pb_field_t *field, void *dest);
-bool pb_dec_submessage(pb_istream_t *stream, const pb_field_t *field, void *dest);
-
-bool pb_skip_varint(pb_istream_t *stream);
-bool pb_skip_string(pb_istream_t *stream);
-#endif
-
 #ifdef __cplusplus
 } /* extern "C" */
 #endif

pb_encode.c

@@ -6,33 +6,55 @@
 #define NANOPB_INTERNALS
 #include "pb.h"
 #include "pb_encode.h"
-#include <string.h>
 
-/* The warn_unused_result attribute appeared first in gcc-3.4.0 */
+/* Use the GCC warn_unused_result attribute to check that all return values
+ * are propagated correctly. On other compilers and gcc before 3.4.0 just
+ * ignore the annotation.
+ */
 #if !defined(__GNUC__) || ( __GNUC__ < 3) || (__GNUC__ == 3 && __GNUC_MINOR__ < 4)
     #define checkreturn
 #else
-    /* Verify that we remember to check all return values for proper error propagation */
     #define checkreturn __attribute__((warn_unused_result))
 #endif
 
+/**************************************
+ * Declarations internal to this file *
+ **************************************/
 typedef bool (*pb_encoder_t)(pb_ostream_t *stream, const pb_field_t *field, const void *src) checkreturn;
 
+static bool checkreturn buf_write(pb_ostream_t *stream, const uint8_t *buf, size_t count);
+static bool checkreturn encode_array(pb_ostream_t *stream, const pb_field_t *field, const void *pData, size_t count, pb_encoder_t func);
+static bool checkreturn encode_field(pb_ostream_t *stream, const pb_field_t *field, const void *pData);
+static bool checkreturn default_extension_encoder(pb_ostream_t *stream, const pb_extension_t *extension);
+static bool checkreturn encode_extension_field(pb_ostream_t *stream, const pb_field_t *field, const void *pData);
+static bool checkreturn pb_enc_varint(pb_ostream_t *stream, const pb_field_t *field, const void *src);
+static bool checkreturn pb_enc_uvarint(pb_ostream_t *stream, const pb_field_t *field, const void *src);
+static bool checkreturn pb_enc_svarint(pb_ostream_t *stream, const pb_field_t *field, const void *src);
+static bool checkreturn pb_enc_fixed32(pb_ostream_t *stream, const pb_field_t *field, const void *src);
+static bool checkreturn pb_enc_fixed64(pb_ostream_t *stream, const pb_field_t *field, const void *src);
+static bool checkreturn pb_enc_bytes(pb_ostream_t *stream, const pb_field_t *field, const void *src);
+static bool checkreturn pb_enc_string(pb_ostream_t *stream, const pb_field_t *field, const void *src);
+static bool checkreturn pb_enc_submessage(pb_ostream_t *stream, const pb_field_t *field, const void *src);
+
 /* --- Function pointers to field encoders ---
  * Order in the array must match pb_action_t LTYPE numbering.
  */
 static const pb_encoder_t PB_ENCODERS[PB_LTYPES_COUNT] = {
     &pb_enc_varint,
+    &pb_enc_uvarint,
     &pb_enc_svarint,
     &pb_enc_fixed32,
     &pb_enc_fixed64,
     
     &pb_enc_bytes,
     &pb_enc_string,
-    &pb_enc_submessage
+    &pb_enc_submessage,
+    NULL /* extensions */
 };
 
-/* pb_ostream_t implementation */
+/*******************************
+ * pb_ostream_t implementation *
+ *******************************/
 
 static bool checkreturn buf_write(pb_ostream_t *stream, const uint8_t *buf, size_t count)
 {
@@ -49,13 +71,16 @@ pb_ostream_t pb_ostream_from_buffer(uint8_t *buf, size_t bufsize)
 {
     pb_ostream_t stream;
 #ifdef PB_BUFFER_ONLY
-    stream.callback = (void*)1; /* Just some marker value */
+    stream.callback = (void*)1; /* Just a marker value */
 #else
     stream.callback = &buf_write;
 #endif
     stream.state = buf;
     stream.max_size = bufsize;
     stream.bytes_written = 0;
+#ifndef PB_NO_ERRMSG
+    stream.errmsg = NULL;
+#endif
     return stream;
 }
 
@@ -64,14 +89,14 @@ bool checkreturn pb_write(pb_ostream_t *stream, const uint8_t *buf, size_t count
     if (stream->callback != NULL)
     {
         if (stream->bytes_written + count > stream->max_size)
-            return false;
+            PB_RETURN_ERROR(stream, "stream full");
 
 #ifdef PB_BUFFER_ONLY
         if (!buf_write(stream, buf, count))
-            return false;
+            PB_RETURN_ERROR(stream, "io error");
 #else        
         if (!stream->callback(stream, buf, count))
-            return false;
+            PB_RETURN_ERROR(stream, "io error");
 #endif
     }
     
@@ -79,12 +104,11 @@ bool checkreturn pb_write(pb_ostream_t *stream, const uint8_t *buf, size_t count
     return true;
 }
 
-/* Main encoding stuff */
+/*************************
+ * Encode a single field *
+ *************************/
 
-/* Callbacks don't need this function because they usually know the data type
+/* Encode a static array. Handles the size calculations and possible packing. */
- * without examining the field structure.
- * Therefore it is static for now.
- */
 static bool checkreturn encode_array(pb_ostream_t *stream, const pb_field_t *field,
                          const void *pData, size_t count, pb_encoder_t func)
 {
@@ -95,6 +119,10 @@ static bool checkreturn encode_array(pb_ostream_t *stream, const pb_field_t *fie
     if (count == 0)
         return true;
 
+    if (PB_ATYPE(field->type) != PB_ATYPE_POINTER && count > field->array_size)
+        PB_RETURN_ERROR(stream, "array max size exceeded");
+    
+    /* We always pack arrays if the datatype allows it. */
     if (PB_LTYPE(field->type) <= PB_LTYPE_LAST_PACKABLE)
     {
         if (!pb_encode_tag(stream, PB_WT_STRING, field->tag))
@@ -111,7 +139,7 @@ static bool checkreturn encode_array(pb_ostream_t *stream, const pb_field_t *fie
         }
         else
         { 
-            pb_ostream_t sizestream = {0,0,0,0};
+            pb_ostream_t sizestream = PB_OSTREAM_SIZING;
             p = pData;
             for (i = 0; i < count; i++)
             {
@@ -144,8 +172,23 @@ static bool checkreturn encode_array(pb_ostream_t *stream, const pb_field_t *fie
         {
             if (!pb_encode_tag_for_field(stream, field))
                 return false;
+
+            /* Normally the data is stored directly in the array entries, but
+             * for pointer-type string and bytes fields, the array entries are
+             * actually pointers themselves also. So we have to dereference once
+             * more to get to the actual data. */
+            if (PB_ATYPE(field->type) == PB_ATYPE_POINTER &&
+                (PB_LTYPE(field->type) == PB_LTYPE_STRING ||
+                 PB_LTYPE(field->type) == PB_LTYPE_BYTES))
+            {
+                if (!func(stream, field, *(const void* const*)p))
+                    return false;      
+            }
+            else
+            {
                 if (!func(stream, field, p))
                     return false;
+            }
             p = (const char*)p + field->data_size;
         }
     }
@@ -153,17 +196,38 @@ static bool checkreturn encode_array(pb_ostream_t *stream, const pb_field_t *fie
     return true;
 }
 
-bool checkreturn encode_static_field(pb_ostream_t *stream, const pb_field_t *field, const void *pData)
+/* Encode a field with static or pointer allocation, i.e. one whose data
+ * is available to the encoder directly. */
+static bool checkreturn encode_basic_field(pb_ostream_t *stream,
+    const pb_field_t *field, const void *pData)
 {
     pb_encoder_t func;
     const void *pSize;
+    bool implicit_has = true;
     
     func = PB_ENCODERS[PB_LTYPE(field->type)];
+    
+    if (field->size_offset)
         pSize = (const char*)pData + field->size_offset;
+    else
+        pSize = &implicit_has;
+
+    if (PB_ATYPE(field->type) == PB_ATYPE_POINTER)
+    {
+        /* pData is a pointer to the field, which contains pointer to
+         * the data. If the 2nd pointer is NULL, it is interpreted as if
+         * the has_field was false.
+         */
+        
+        pData = *(const void* const*)pData;
+        implicit_has = (pData != NULL);
+    }
 
     switch (PB_HTYPE(field->type))
     {
         case PB_HTYPE_REQUIRED:
+            if (!pData)
+                PB_RETURN_ERROR(stream, "missing required field");
             if (!pb_encode_tag_for_field(stream, field))
                 return false;
             if (!func(stream, field, pData))
@@ -187,23 +251,89 @@ bool checkreturn encode_static_field(pb_ostream_t *stream, const pb_field_t *fie
             break;
         
         default:
-            return false;
+            PB_RETURN_ERROR(stream, "invalid field type");
     }
     
     return true;
 }
 
-bool checkreturn encode_callback_field(pb_ostream_t *stream, const pb_field_t *field, const void *pData)
+/* Encode a field with callback semantics. This means that a user function is
+ * called to provide and encode the actual data. */
+static bool checkreturn encode_callback_field(pb_ostream_t *stream,
+    const pb_field_t *field, const void *pData)
 {
     const pb_callback_t *callback = (const pb_callback_t*)pData;
+    
+#ifdef PB_OLD_CALLBACK_STYLE
+    const void *arg = callback->arg;
+#else
+    void * const *arg = &(callback->arg);
+#endif    
+    
     if (callback->funcs.encode != NULL)
     {
-        if (!callback->funcs.encode(stream, field, callback->arg))
+        if (!callback->funcs.encode(stream, field, arg))
-            return false;
+            PB_RETURN_ERROR(stream, "callback error");
     }
     return true;
 }
 
+/* Encode a single field of any callback or static type. */
+static bool checkreturn encode_field(pb_ostream_t *stream,
+    const pb_field_t *field, const void *pData)
+{
+    switch (PB_ATYPE(field->type))
+    {
+        case PB_ATYPE_STATIC:
+        case PB_ATYPE_POINTER:
+            return encode_basic_field(stream, field, pData);
+        
+        case PB_ATYPE_CALLBACK:
+            return encode_callback_field(stream, field, pData);
+        
+        default:
+            PB_RETURN_ERROR(stream, "invalid field type");
+    }
+}
+
+/* Default handler for extension fields. Expects to have a pb_field_t
+ * pointer in the extension->type->arg field. */
+static bool checkreturn default_extension_encoder(pb_ostream_t *stream,
+    const pb_extension_t *extension)
+{
+    const pb_field_t *field = (const pb_field_t*)extension->type->arg;
+    return encode_field(stream, field, extension->dest);
+}
+
+/* Walk through all the registered extensions and give them a chance
+ * to encode themselves. */
+static bool checkreturn encode_extension_field(pb_ostream_t *stream,
+    const pb_field_t *field, const void *pData)
+{
+    const pb_extension_t *extension = *(const pb_extension_t* const *)pData;
+    UNUSED(field);
+    
+    while (extension)
+    {
+        bool status;
+        if (extension->type->encode)
+            status = extension->type->encode(stream, extension);
+        else
+            status = default_extension_encoder(stream, extension);
+
+        if (!status)
+            return false;
+        
+        extension = extension->next;
+    }
+    
+    return true;
+}
+
+/*********************
+ * Encode all fields *
+ *********************/
+
 bool checkreturn pb_encode(pb_ostream_t *stream, const pb_field_t fields[], const void *src_struct)
 {
     const pb_field_t *field = fields;
@@ -213,6 +343,9 @@ bool checkreturn pb_encode(pb_ostream_t *stream, const pb_field_t fields[], cons
     while (field->tag != 0)
     {
         pData = (const char*)pData + prev_size + field->data_offset;
+        if (PB_ATYPE(field->type) == PB_ATYPE_POINTER)
+            prev_size = sizeof(const void*);
+        else
             prev_size = field->data_size;
         
         /* Special case for static arrays */
@@ -222,19 +355,16 @@ bool checkreturn pb_encode(pb_ostream_t *stream, const pb_field_t fields[], cons
             prev_size *= field->array_size;
         }
         
-        switch (PB_ATYPE(field->type))
+        if (PB_LTYPE(field->type) == PB_LTYPE_EXTENSION)
         {
-            case PB_ATYPE_STATIC:
+            /* Special case for the extension field placeholder */
-                if (!encode_static_field(stream, field, pData))
+            if (!encode_extension_field(stream, field, pData))
                 return false;
-                break;
+        }
-            
+        else
-            case PB_ATYPE_CALLBACK:
+        {
-                if (!encode_callback_field(stream, field, pData))
+            /* Regular field */
-                    return false;
+            if (!encode_field(stream, field, pData))
-                break;
-            
-            default:
                 return false;
         }
     
@@ -244,7 +374,14 @@ bool checkreturn pb_encode(pb_ostream_t *stream, const pb_field_t fields[], cons
     return true;
 }
 
-/* Helper functions */
+bool pb_encode_delimited(pb_ostream_t *stream, const pb_field_t fields[], const void *src_struct)
+{
+    return pb_encode_submessage(stream, fields, src_struct);
+}
+
+/********************
+ * Helper functions *
+ ********************/
 bool checkreturn pb_encode_varint(pb_ostream_t *stream, uint64_t value)
 {
     uint8_t buffer[10];
@@ -321,6 +458,7 @@ bool checkreturn pb_encode_tag_for_field(pb_ostream_t *stream, const pb_field_t
     switch (PB_LTYPE(field->type))
     {
         case PB_LTYPE_VARINT:
+        case PB_LTYPE_UVARINT:
         case PB_LTYPE_SVARINT:
             wiretype = PB_WT_VARINT;
             break;
@@ -340,7 +478,7 @@ bool checkreturn pb_encode_tag_for_field(pb_ostream_t *stream, const pb_field_t
             break;
         
         default:
-            return false;
+            PB_RETURN_ERROR(stream, "invalid field type");
     }
     
     return pb_encode_tag(stream, wiretype, field->tag);
@@ -357,12 +495,17 @@ bool checkreturn pb_encode_string(pb_ostream_t *stream, const uint8_t *buffer, s
 bool checkreturn pb_encode_submessage(pb_ostream_t *stream, const pb_field_t fields[], const void *src_struct)
 {
     /* First calculate the message size using a non-writing substream. */
-    pb_ostream_t substream = {0,0,0,0};
+    pb_ostream_t substream = PB_OSTREAM_SIZING;
     size_t size;
     bool status;
     
     if (!pb_encode(&substream, fields, src_struct))
+    {
+#ifndef PB_NO_ERRMSG
+        stream->errmsg = substream.errmsg;
+#endif
         return false;
+    }
     
     size = substream.bytes_written;
     
@@ -373,7 +516,7 @@ bool checkreturn pb_encode_submessage(pb_ostream_t *stream, const pb_field_t fie
         return pb_write(stream, NULL, size); /* Just sizing */
     
     if (stream->bytes_written + size > stream->max_size)
-        return false;
+        PB_RETURN_ERROR(stream, "stream full");
         
     /* Use a substream to verify that a callback doesn't write more than
      * what it did the first time. */
@@ -381,14 +524,20 @@ bool checkreturn pb_encode_submessage(pb_ostream_t *stream, const pb_field_t fie
     substream.state = stream->state;
     substream.max_size = size;
     substream.bytes_written = 0;
+#ifndef PB_NO_ERRMSG
+    substream.errmsg = NULL;
+#endif
     
     status = pb_encode(&substream, fields, src_struct);
     
     stream->bytes_written += substream.bytes_written;
     stream->state = substream.state;
+#ifndef PB_NO_ERRMSG
+    stream->errmsg = substream.errmsg;
+#endif
     
     if (substream.bytes_written != size)
-        return false;
+        PB_RETURN_ERROR(stream, "submsg size changed");
     
     return status;
 }
@@ -396,16 +545,32 @@ bool checkreturn pb_encode_submessage(pb_ostream_t *stream, const pb_field_t fie
 /* Field encoders */
 
 bool checkreturn pb_enc_varint(pb_ostream_t *stream, const pb_field_t *field, const void *src)
+{
+    int64_t value = 0;
+    
+    /* Cases 1 and 2 are for compilers that have smaller types for bool
+     * or enums. */
+    switch (field->data_size)
+    {
+        case 1: value = *(const int8_t*)src; break;
+        case 2: value = *(const int16_t*)src; break;
+        case 4: value = *(const int32_t*)src; break;
+        case 8: value = *(const int64_t*)src; break;
+        default: PB_RETURN_ERROR(stream, "invalid data_size");
+    }
+    
+    return pb_encode_varint(stream, (uint64_t)value);
+}
+
+bool checkreturn pb_enc_uvarint(pb_ostream_t *stream, const pb_field_t *field, const void *src)
 {
     uint64_t value = 0;
     
     switch (field->data_size)
     {
-        case 1: value = *(const uint8_t*)src; break;
-        case 2: value = *(const uint16_t*)src; break;
         case 4: value = *(const uint32_t*)src; break;
         case 8: value = *(const uint64_t*)src; break;
-        default: return false;
+        default: PB_RETURN_ERROR(stream, "invalid data_size");
     }
     
     return pb_encode_varint(stream, value);
@@ -419,7 +584,7 @@ bool checkreturn pb_enc_svarint(pb_ostream_t *stream, const pb_field_t *field, c
     {
         case 4: value = *(const int32_t*)src; break;
         case 8: value = *(const int64_t*)src; break;
-        default: return false;
+        default: PB_RETURN_ERROR(stream, "invalid data_size");
     }
     
     return pb_encode_svarint(stream, value);
@@ -440,20 +605,52 @@ bool checkreturn pb_enc_fixed32(pb_ostream_t *stream, const pb_field_t *field, c
 bool checkreturn pb_enc_bytes(pb_ostream_t *stream, const pb_field_t *field, const void *src)
 {
     const pb_bytes_array_t *bytes = (const pb_bytes_array_t*)src;
-    UNUSED(field);
+    
+    if (src == NULL)
+    {
+        /* Threat null pointer as an empty bytes field */
+        return pb_encode_string(stream, NULL, 0);
+    }
+    
+    if (PB_ATYPE(field->type) == PB_ATYPE_STATIC &&
+        PB_BYTES_ARRAY_T_ALLOCSIZE(bytes->size) > field->data_size)
+    {
+        PB_RETURN_ERROR(stream, "bytes size exceeded");
+    }
+    
     return pb_encode_string(stream, bytes->bytes, bytes->size);
 }
 
 bool checkreturn pb_enc_string(pb_ostream_t *stream, const pb_field_t *field, const void *src)
 {
-    UNUSED(field);
+    /* strnlen() is not always available, so just use a loop */
-    return pb_encode_string(stream, (const uint8_t*)src, strlen((const char*)src));
+    size_t size = 0;
+    size_t max_size = field->data_size;
+    const char *p = (const char*)src;
+    
+    if (PB_ATYPE(field->type) == PB_ATYPE_POINTER)
+        max_size = (size_t)-1;
+
+    if (src == NULL)
+    {
+        size = 0; /* Threat null pointer as an empty string */
+    }
+    else
+    {
+        while (size < max_size && *p != '\0')
+        {
+            size++;
+            p++;
+        }
+    }
+
+    return pb_encode_string(stream, (const uint8_t*)src, size);
 }
 
 bool checkreturn pb_enc_submessage(pb_ostream_t *stream, const pb_field_t *field, const void *src)
 {
     if (field->ptr == NULL)
-        return false;
+        PB_RETURN_ERROR(stream, "invalid field descriptor");
     
     return pb_encode_submessage(stream, (const pb_field_t*)field->ptr, src);
 }

pb_encode.h

@@ -1,34 +1,28 @@
+/* pb_encode.h: Functions to encode protocol buffers. Depends on pb_encode.c.
+ * The main function is pb_encode. You also need an output stream, and the
+ * field descriptions created by nanopb_generator.py.
+ */
+
 #ifndef _PB_ENCODE_H_
 #define _PB_ENCODE_H_
 
-/* pb_encode.h: Functions to encode protocol buffers. Depends on pb_encode.c.
- * The main function is pb_encode. You also need an output stream, structures
- * and their field descriptions (just like with pb_decode).
- */
-
-#include <stdbool.h>
 #include "pb.h"
 
 #ifdef __cplusplus
 extern "C" {
 #endif
 
-/* Lightweight output stream.
+/* Structure for defining custom output streams. You will need to provide
- * You can provide callback for writing or use pb_ostream_from_buffer.
+ * a callback function to write the bytes to your storage, which can be
+ * for example a file or a network socket.
  *
- * Alternatively, callback can be NULL in which case the stream will just
+ * The callback must conform to these rules:
- * count the number of bytes that would have been written. In this case
- * max_size is not checked.
  *
- * Rules for callback:
  * 1) Return false on IO errors. This will cause encoding to abort.
- * 
  * 2) You can use state to store your own data (e.g. buffer pointer).
- * 
  * 3) pb_write will update bytes_written after your callback runs.
- * 
+ * 4) Substreams will modify max_size and bytes_written. Don't use them
- * 4) Substreams will modify max_size and bytes_written. Don't use them to
+ *    to calculate any pointers.
- * calculate any pointers.
  */
 struct _pb_ostream_t
 {
@@ -43,31 +37,84 @@ struct _pb_ostream_t
 #else
     bool (*callback)(pb_ostream_t *stream, const uint8_t *buf, size_t count);
 #endif
-    void *state; /* Free field for use by callback implementation */
+    void *state;          /* Free field for use by callback implementation. */
     size_t max_size;      /* Limit number of output bytes written (or use SIZE_MAX). */
-    size_t bytes_written;
+    size_t bytes_written; /* Number of bytes written so far. */
+    
+#ifndef PB_NO_ERRMSG
+    const char *errmsg;
+#endif
 };
 
-pb_ostream_t pb_ostream_from_buffer(uint8_t *buf, size_t bufsize);
+/***************************
-bool pb_write(pb_ostream_t *stream, const uint8_t *buf, size_t count);
+ * Main encoding functions *
+ ***************************/
 
-/* Encode struct to given output stream.
+/* Encode a single protocol buffers message from C structure into a stream.
  * Returns true on success, false on any failure.
  * The actual struct pointed to by src_struct must match the description in fields.
  * All required fields in the struct are assumed to have been filled in.
+ *
+ * Example usage:
+ *    MyMessage msg = {};
+ *    uint8_t buffer[64];
+ *    pb_ostream_t stream;
+ *
+ *    msg.field1 = 42;
+ *    stream = pb_ostream_from_buffer(buffer, sizeof(buffer));
+ *    pb_encode(&stream, MyMessage_fields, &msg);
  */
 bool pb_encode(pb_ostream_t *stream, const pb_field_t fields[], const void *src_struct);
 
-/* --- Helper functions ---
+/* Same as pb_encode, but prepends the length of the message as a varint.
- * You may want to use these from your caller or callbacks.
+ * Corresponds to writeDelimitedTo() in Google's protobuf API.
  */
+bool pb_encode_delimited(pb_ostream_t *stream, const pb_field_t fields[], const void *src_struct);
 
-/* Encode field header based on LTYPE and field number defined in the field structure.
+/**************************************
- * Call this from the callback before writing out field contents. */
+ * Functions for manipulating streams *
+ **************************************/
+
+/* Create an output stream for writing into a memory buffer.
+ * The number of bytes written can be found in stream.bytes_written after
+ * encoding the message.
+ *
+ * Alternatively, you can use a custom stream that writes directly to e.g.
+ * a file or a network socket.
+ */
+pb_ostream_t pb_ostream_from_buffer(uint8_t *buf, size_t bufsize);
+
+/* Pseudo-stream for measuring the size of a message without actually storing
+ * the encoded data.
+ * 
+ * Example usage:
+ *    MyMessage msg = {};
+ *    pb_ostream_t stream = PB_OSTREAM_SIZING;
+ *    pb_encode(&stream, MyMessage_fields, &msg);
+ *    printf("Message size is %d\n", stream.bytes_written);
+ */
+#ifndef PB_NO_ERRMSG
+#define PB_OSTREAM_SIZING {0,0,0,0,0}
+#else
+#define PB_OSTREAM_SIZING {0,0,0,0}
+#endif
+
+/* Function to write into a pb_ostream_t stream. You can use this if you need
+ * to append or prepend some custom headers to the message.
+ */
+bool pb_write(pb_ostream_t *stream, const uint8_t *buf, size_t count);
+
+
+/************************************************
+ * Helper functions for writing field callbacks *
+ ************************************************/
+
+/* Encode field header based on type and field number defined in the field
+ * structure. Call this from the callback before writing out field contents. */
 bool pb_encode_tag_for_field(pb_ostream_t *stream, const pb_field_t *field);
 
-/* Encode field header by manually specifing wire type. You need to use this if
+/* Encode field header by manually specifing wire type. You need to use this
- * you want to write out packed arrays from a callback field. */
+ * if you want to write out packed arrays from a callback field. */
 bool pb_encode_tag(pb_ostream_t *stream, pb_wire_type_t wiretype, uint32_t field_number);
 
 /* Encode an integer in the varint format.
@@ -90,29 +137,12 @@ bool pb_encode_fixed32(pb_ostream_t *stream, const void *value);
 bool pb_encode_fixed64(pb_ostream_t *stream, const void *value);
 
 /* Encode a submessage field.
- * You need to pass the pb_field_t array and pointer to struct, just like with pb_encode().
+ * You need to pass the pb_field_t array and pointer to struct, just like
- * This internally encodes the submessage twice, first to calculate message size and then to actually write it out.
+ * with pb_encode(). This internally encodes the submessage twice, first to
+ * calculate message size and then to actually write it out.
  */
 bool pb_encode_submessage(pb_ostream_t *stream, const pb_field_t fields[], const void *src_struct);
 
-/* --- Internal functions ---
- * These functions are not terribly useful for the average library user, but
- * are exported to make the unit testing and extending nanopb easier.
- */
-
-#ifdef NANOPB_INTERNALS
-bool pb_enc_varint(pb_ostream_t *stream, const pb_field_t *field, const void *src);
-bool pb_enc_svarint(pb_ostream_t *stream, const pb_field_t *field, const void *src);
-bool pb_enc_fixed32(pb_ostream_t *stream, const pb_field_t *field, const void *src);
-bool pb_enc_fixed64(pb_ostream_t *stream, const pb_field_t *field, const void *src);
-bool pb_enc_bytes(pb_ostream_t *stream, const pb_field_t *field, const void *src);
-bool pb_enc_string(pb_ostream_t *stream, const pb_field_t *field, const void *src);
-#endif
-
-/* This function is not recommended for new programs. Use pb_encode_submessage()
- * instead, it has the same functionality with a less confusing interface. */
-bool pb_enc_submessage(pb_ostream_t *stream, const pb_field_t *field, const void *src);
-
 #ifdef __cplusplus
 } /* extern "C" */
 #endif

tests/Makefile

@@ -1,131 +1,6 @@
-CFLAGS=-ansi -Wall -Werror -I .. -g -O0
+all:
-DEPS=../pb_decode.h ../pb_encode.h ../pb.h person.pb.h \
+	scons
-	callbacks2.pb.h callbacks.pb.h unittests.h unittestproto.pb.h \
-	alltypes.pb.h missing_fields.pb.h
-TESTS=  decode_unittests encode_unittests \
-	test_decode1 test_decode2 test_decode3 test_decode3_buf \
-	test_encode1 test_encode2 test_encode3 test_encode3_buf \
-	test_decode_callbacks test_encode_callbacks \
-	test_missing_fields test_no_messages test_funny_name \
-	test_multiple_files test_cxxcompile test_options \
-	bc_encode bc_decode
-
-# More strict checks for the core part of nanopb
-CC_VERSION=$(shell $(CC) -v 2>&1)
-CFLAGS_CORE=
-ifneq (,$(findstring gcc,$(CC_VERSION)))
-	CFLAGS_CORE=-pedantic -Wextra -Wcast-qual -Wlogical-op -Wconversion
-        CFLAGS+=--coverage
-        LDFLAGS+=--coverage
-endif
-ifneq (,$(findstring clang,$(CC_VERSION)))
-	CFLAGS_CORE=-pedantic -Wextra -Wcast-qual -Wconversion
-endif
-
-all: breakpoints $(TESTS) run_unittests
 
 clean:
-	rm -f $(TESTS) person.pb* alltypes.pb* *.o *.gcda *.gcno *.pb.h *.pb.c
+	scons -c
 
-%.pb.o: %.pb.c %.pb.h
-	$(CC) $(CFLAGS) $(CFLAGS_CORE) -c -o $@ $<
-
-%.o: %.c
-%.o: %.c $(DEPS)
-	$(CC) $(CFLAGS) -c -o $@ $<
-
-pb_encode.o: ../pb_encode.c $(DEPS)
-	$(CC) $(CFLAGS) $(CFLAGS_CORE) -c -o $@ $<
-pb_decode.o: ../pb_decode.c $(DEPS)
-	$(CC) $(CFLAGS) $(CFLAGS_CORE) -c -o $@ $<
-
-# Test for compilability with c++ compiler
-
-pb_encode.cxx.o: ../pb_encode.c $(DEPS)
-	$(CXX) $(CFLAGS) $(CFLAGS_CORE) -c -o $@ $<
-pb_decode.cxx.o: ../pb_decode.c $(DEPS)
-	$(CXX) $(CFLAGS) $(CFLAGS_CORE) -c -o $@ $<
-
-# Test for PB_BUF_ONLY compilation option
-
-pb_encode.buf.o: ../pb_encode.c $(DEPS)
-	$(CC) -DPB_BUFFER_ONLY $(CFLAGS) $(CFLAGS_CORE) -c -o $@ $<
-pb_decode.buf.o: ../pb_decode.c $(DEPS)
-	$(CC) -DPB_BUFFER_ONLY $(CFLAGS) $(CFLAGS_CORE) -c -o $@ $<
-%.buf.o: %.c $(DEPS)
-	$(CC) -DPB_BUFFER_ONLY $(CFLAGS) -c -o $@ $<	
-test_encode3_buf: test_encode3.buf.o pb_encode.buf.o alltypes.pb.o
-	$(CC) $(LDFLAGS) $^ -o $@
-test_decode3_buf: test_decode3.buf.o pb_decode.buf.o alltypes.pb.o
-	$(CC) $(LDFLAGS) $^ -o $@
-
-test_cxxcompile: pb_encode.cxx.o pb_decode.cxx.o
-test_decode1: test_decode1.o pb_decode.o person.pb.o
-test_decode2: test_decode2.o pb_decode.o person.pb.o
-test_decode3: test_decode3.o pb_decode.o alltypes.pb.o
-test_encode1: test_encode1.o pb_encode.o person.pb.o
-test_encode2: test_encode2.o pb_encode.o person.pb.o
-test_encode3: test_encode3.o pb_encode.o alltypes.pb.o
-test_multiple_files: test_multiple_files.o pb_encode.o callbacks2.pb.o callbacks.pb.o
-test_decode_callbacks: test_decode_callbacks.o pb_decode.o callbacks.pb.o
-test_encode_callbacks: test_encode_callbacks.o pb_encode.o callbacks.pb.o
-test_missing_fields: test_missing_fields.o pb_encode.o pb_decode.o missing_fields.pb.o
-decode_unittests: decode_unittests.o pb_decode.o unittestproto.pb.o
-encode_unittests: encode_unittests.o pb_encode.o unittestproto.pb.o
-test_no_messages: no_messages.pb.h no_messages.pb.c no_messages.pb.o
-test_funny_name: funny-proto+name.pb.h funny-proto+name.pb.o
-bc_encode: bc_alltypes.pb.o pb_encode.o bc_encode.o
-bc_decode: bc_alltypes.pb.o pb_decode.o bc_decode.o
-
-%.pb: %.proto
-	protoc -I. -I../generator -I/usr/include -o$@ $<
-
-%.pb.c %.pb.h: %.pb ../generator/nanopb_generator.py
-	python ../generator/nanopb_generator.py $<
-
-breakpoints: ../*.c *.c
-	grep -n 'return false;' $^ | cut -d: -f-2 | xargs -n 1 echo b > $@
-
-coverage: run_unittests
-	gcov pb_encode.gcda
-	gcov pb_decode.gcda
-
-run_unittests: $(TESTS)
-	rm -f *.gcda
-	
-	./decode_unittests > /dev/null
-	./encode_unittests > /dev/null
-	
-	[ "`./test_encode1 | ./test_decode1`" = \
-	"`./test_encode1 | protoc --decode=Person -I. -I../generator -I/usr/include person.proto`" ]
-
-	[ "`./test_encode2 | ./test_decode1`" = \
-	"`./test_encode2 | protoc --decode=Person -I. -I../generator -I/usr/include person.proto`" ]
-
-	[ "`./test_encode2 | ./test_decode2`" = \
-	"`./test_encode2 | protoc --decode=Person -I. -I../generator -I/usr/include person.proto`" ]
-	
-	[ "`./test_decode2 < person_with_extra_field.pb`" = \
-	"`./test_encode2 | ./test_decode2`" ]
-	
-	[ "`./test_encode_callbacks | ./test_decode_callbacks`" = \
-	"`./test_encode_callbacks | protoc --decode=TestMessage callbacks.proto`" ]
-
-	./test_encode3 | ./test_decode3
-	./test_encode3 1 | ./test_decode3 1
-	./test_encode3 1 | protoc --decode=AllTypes -I. -I../generator -I/usr/include alltypes.proto >/dev/null
-	./test_encode3_buf 1 | ./test_decode3_buf 1
-	./bc_encode | ./bc_decode
-	
-	./test_missing_fields
-
-test_options: options.pb.h options.expected options.pb.o
-	cat options.expected | while read -r p; do \
-	    if ! grep -q "$$p" $<; then \
-	        echo Expected: "$$p"; \
-	        exit 1; \
-	    fi \
-	done
-
-run_fuzztest: test_decode2
-	bash -c 'I=1; while true; do cat /dev/urandom | ./test_decode2 > /dev/null; I=$$(($$I+1)); echo -en "\r$$I"; done'

tests/SConstruct

@@ -0,0 +1,134 @@
+Help('''
+Type 'scons' to build and run all the available test cases.
+It will automatically detect your platform and C compiler and
+build appropriately.
+
+You can modify the behavious using following options:
+CC          Name of C compiler
+CXX         Name of C++ compiler
+CCFLAGS     Flags to pass to the C compiler
+CXXFLAGS    Flags to pass to the C++ compiler
+
+For example, for a clang build, use:
+scons CC=clang CXX=clang++
+''')
+
+import os
+env = Environment(ENV = os.environ, tools = ['default', 'nanopb'])
+
+# Allow overriding the compiler with scons CC=???
+if 'CC' in ARGUMENTS: env.Replace(CC = ARGUMENTS['CC'])
+if 'CXX' in ARGUMENTS: env.Replace(CXX = ARGUMENTS['CXX'])
+if 'CFLAGS' in ARGUMENTS: env.Append(CCFLAGS = ARGUMENTS['CFLAGS'])
+if 'CXXFLAGS' in ARGUMENTS: env.Append(CCFLAGS = ARGUMENTS['CXXFLAGS'])
+
+# Add the builders defined in site_init.py
+add_nanopb_builders(env)
+
+# Path to the files shared by tests, and to the nanopb core.
+env.Append(CPPPATH = ["#../", "$COMMON"])
+
+# Path for finding nanopb.proto
+env.Append(PROTOCPATH = '#../generator')
+
+# Check the compilation environment, unless we are just cleaning up.
+if not env.GetOption('clean'):
+    def check_ccflags(context, flags):
+        '''Check if given CCFLAGS are supported'''
+        context.Message('Checking support for CCFLAGS="%s"... ' % flags)
+        oldflags = context.env['CCFLAGS']
+        context.env.Append(CCFLAGS = flags)
+        result = context.TryCompile("int main() {return 0;}", '.c')
+        context.env.Replace(CCFLAGS = oldflags)
+        context.Result(result)
+        return result
+    
+    conf = Configure(env, custom_tests = {'CheckCCFLAGS': check_ccflags})
+
+    # If the platform doesn't support C99, use our own header file instead.
+    stdbool = conf.CheckCHeader('stdbool.h')
+    stdint = conf.CheckCHeader('stdint.h')
+    stddef = conf.CheckCHeader('stddef.h')
+    string = conf.CheckCHeader('string.h')
+    if not stdbool or not stdint or not stddef or not string:
+        conf.env.Append(CPPDEFINES = {'PB_SYSTEM_HEADER': '\\"pb_syshdr.h\\"'})
+        conf.env.Append(CPPPATH = "#../extra")
+        
+        if stdbool: conf.env.Append(CPPDEFINES = {'HAVE_STDBOOL_H': 1})
+        if stdint: conf.env.Append(CPPDEFINES = {'HAVE_STDINT_H': 1})
+        if stddef: conf.env.Append(CPPDEFINES = {'HAVE_STDDEF_H': 1})
+        if string: conf.env.Append(CPPDEFINES = {'HAVE_STRING_H': 1})
+    
+    # Check if we can use pkg-config to find protobuf include path
+    status, output = conf.TryAction('pkg-config protobuf --variable=includedir > $TARGET')
+    if status:
+        conf.env.Append(PROTOCPATH = output.strip())
+    else:
+        conf.env.Append(PROTOCPATH = '/usr/include')
+    
+    # Check if libmudflap is available (only with GCC)
+    if 'gcc' in env['CC']:
+        if conf.CheckLib('mudflap'):
+            conf.env.Append(CCFLAGS = '-fmudflap')
+            conf.env.Append(LINKFLAGS = '-fmudflap')
+    
+    # Check if we can use extra strict warning flags (only with GCC)
+    extra = '-Wcast-qual -Wlogical-op -Wconversion'
+    extra += ' -fstrict-aliasing -Wstrict-aliasing=1'
+    extra += ' -Wmissing-prototypes -Wmissing-declarations -Wredundant-decls'
+    extra += ' -Wstack-protector '
+    if 'gcc' in env['CC']:
+        if conf.CheckCCFLAGS(extra):
+            conf.env.Append(CORECFLAGS = extra)
+    
+    # End the config stuff
+    env = conf.Finish()
+
+# Initialize the CCFLAGS according to the compiler
+if 'gcc' in env['CC']:
+    # GNU Compiler Collection
+    
+    # Debug info, warnings as errors
+    env.Append(CFLAGS = '-ansi -pedantic -g -O0 -Wall -Werror --coverage -fstack-protector-all')
+    env.Append(CORECFLAGS = '-Wextra')
+    env.Append(LINKFLAGS = '--coverage')
+    
+    # We currently need uint64_t anyway, even though ANSI C90 otherwise..
+    env.Append(CFLAGS = '-Wno-long-long')
+elif 'clang' in env['CC']:
+    # CLang
+    env.Append(CFLAGS = '-ansi -g -O0 -Wall -Werror')
+    env.Append(CORECFLAGS = ' -Wextra -Wcast-qual -Wconversion')
+elif 'cl' in env['CC']:
+    # Microsoft Visual C++
+    
+    # Debug info on, warning level 2 for tests, warnings as errors
+    env.Append(CFLAGS = '/Zi /W2 /WX')
+    env.Append(LINKFLAGS = '/DEBUG')
+    
+    # More strict checks on the nanopb core
+    env.Append(CORECFLAGS = '/W4')
+    
+    # PB_RETURN_ERROR triggers C4127 because of while(0)
+    env.Append(CFLAGS = '/wd4127')
+elif 'tcc' in env['CC']:
+    # Tiny C Compiler
+    env.Append(CFLAGS = '-Wall -Werror -g')
+
+env.SetDefault(CORECFLAGS = '')
+
+if 'clang++' in env['CXX']:
+    env.Append(CXXFLAGS = '-g -O0 -Wall -Werror -Wextra -Wno-missing-field-initializers')
+elif 'g++' in env['CXX']:
+    env.Append(CXXFLAGS = '-g -O0 -Wall -Werror -Wextra -Wno-missing-field-initializers')
+elif 'cl' in env['CXX']:
+    env.Append(CXXFLAGS = '/Zi /W2 /WX')
+    
+# Now include the SConscript files from all subdirectories
+import os.path
+env['VARIANT_DIR'] = 'build'
+env['BUILD'] = '#' + env['VARIANT_DIR']
+env['COMMON'] = '#' + env['VARIANT_DIR'] + '/common'
+for subdir in Glob('*/SConscript'):
+    SConscript(subdir, exports = 'env', variant_dir = env['VARIANT_DIR'] + '/' + os.path.dirname(str(subdir)))
+

tests/alltypes.proto

@@ -1,85 +0,0 @@
-import "nanopb.proto";
-
-message SubMessage {
-    required string substuff1 = 1 [(nanopb).max_size = 16, default = "1"];
-    required int32 substuff2 = 2 [default = 2];
-    optional fixed32 substuff3 = 3 [default = 3];
-}
-
-enum MyEnum {
-    Zero = 0;
-    First = 1;
-    Second = 2;
-    Truth = 42;
-}
-
-message AllTypes {
-    required int32      req_int32   = 1;
-    required int64      req_int64   = 2;
-    required uint32     req_uint32  = 3;
-    required uint64     req_uint64  = 4;
-    required sint32     req_sint32  = 5;
-    required sint64     req_sint64  = 6;
-    required bool       req_bool    = 7;
-    
-    required fixed32    req_fixed32 = 8;
-    required sfixed32   req_sfixed32= 9;
-    required float      req_float   = 10;
-    
-    required fixed64    req_fixed64 = 11;
-    required sfixed64   req_sfixed64= 12;
-    required double     req_double  = 13;
-    
-    required string     req_string  = 14 [(nanopb).max_size = 16];
-    required bytes      req_bytes   = 15 [(nanopb).max_size = 16];
-    required SubMessage req_submsg  = 16;
-    required MyEnum     req_enum    = 17;
-    
-    
-    repeated int32      rep_int32   = 21 [(nanopb).max_count = 5];
-    repeated int64      rep_int64   = 22 [(nanopb).max_count = 5];
-    repeated uint32     rep_uint32  = 23 [(nanopb).max_count = 5];
-    repeated uint64     rep_uint64  = 24 [(nanopb).max_count = 5];
-    repeated sint32     rep_sint32  = 25 [(nanopb).max_count = 5];
-    repeated sint64     rep_sint64  = 26 [(nanopb).max_count = 5];
-    repeated bool       rep_bool    = 27 [(nanopb).max_count = 5];
-    
-    repeated fixed32    rep_fixed32 = 28 [(nanopb).max_count = 5];
-    repeated sfixed32   rep_sfixed32= 29 [(nanopb).max_count = 5];
-    repeated float      rep_float   = 30 [(nanopb).max_count = 5];
-    
-    repeated fixed64    rep_fixed64 = 31 [(nanopb).max_count = 5];
-    repeated sfixed64   rep_sfixed64= 32 [(nanopb).max_count = 5];
-    repeated double     rep_double  = 33 [(nanopb).max_count = 5];
-    
-    repeated string     rep_string  = 34 [(nanopb).max_size = 16, (nanopb).max_count = 5];
-    repeated bytes      rep_bytes   = 35 [(nanopb).max_size = 16, (nanopb).max_count = 5];
-    repeated SubMessage rep_submsg  = 36 [(nanopb).max_count = 5];
-    repeated MyEnum     rep_enum    = 37 [(nanopb).max_count = 5];
-    
-    optional int32      opt_int32   = 41 [default = 4041];
-    optional int64      opt_int64   = 42 [default = 4042];
-    optional uint32     opt_uint32  = 43 [default = 4043];
-    optional uint64     opt_uint64  = 44 [default = 4044];
-    optional sint32     opt_sint32  = 45 [default = 4045];
-    optional sint64     opt_sint64  = 46 [default = 4046];
-    optional bool       opt_bool    = 47 [default = false];
-    
-    optional fixed32    opt_fixed32 = 48 [default = 4048];
-    optional sfixed32   opt_sfixed32= 49 [default = 4049];
-    optional float      opt_float   = 50 [default = 4050];
-    
-    optional fixed64    opt_fixed64 = 51 [default = 4051];
-    optional sfixed64   opt_sfixed64= 52 [default = 4052];
-    optional double     opt_double  = 53 [default = 4053];
-    
-    optional string     opt_string  = 54 [(nanopb).max_size = 16, default = "4054"];
-    optional bytes      opt_bytes   = 55 [(nanopb).max_size = 16, default = "4055"];
-    optional SubMessage opt_submsg  = 56;
-    optional MyEnum     opt_enum    = 57 [default = Second];
-
-    // Just to make sure that the size of the fields has been calculated
-    // properly, i.e. otherwise a bug in last field might not be detected.
-    required int32      end = 99;
-}
-

tests/alltypes/SConscript

@@ -0,0 +1,35 @@
+# Build and run a test that encodes and decodes a message that contains
+# all of the Protocol Buffers data types.
+
+Import("env")
+
+env.NanopbProto(["alltypes", "alltypes.options"])
+enc = env.Program(["encode_alltypes.c", "alltypes.pb.c", "$COMMON/pb_encode.o"])
+dec = env.Program(["decode_alltypes.c", "alltypes.pb.c", "$COMMON/pb_decode.o"])
+
+# Test the round-trip from nanopb encoder to nanopb decoder
+env.RunTest(enc)
+env.RunTest([dec, "encode_alltypes.output"])
+
+# Re-encode the data using protoc, and check that the results from nanopb
+# match byte-per-byte to the protoc output.
+env.Decode("encode_alltypes.output.decoded",
+           ["encode_alltypes.output", "alltypes.proto"],
+           MESSAGE='AllTypes')
+env.Encode("encode_alltypes.output.recoded",
+           ["encode_alltypes.output.decoded", "alltypes.proto"],
+           MESSAGE='AllTypes')
+env.Compare(["encode_alltypes.output", "encode_alltypes.output.recoded"])
+
+# Do the same checks with the optional fields present.
+env.RunTest("optionals.output", enc, ARGS = ['1'])
+env.RunTest("optionals.decout", [dec, "optionals.output"], ARGS = ['1'])
+env.Decode("optionals.output.decoded",
+           ["optionals.output", "alltypes.proto"],
+           MESSAGE='AllTypes')
+env.Encode("optionals.output.recoded",
+           ["optionals.output.decoded", "alltypes.proto"],
+           MESSAGE='AllTypes')
+env.Compare(["optionals.output", "optionals.output.recoded"])
+
+

tests/alltypes/alltypes.options

@@ -0,0 +1,3 @@
+* max_size:16
+* max_count:5
+

tests/alltypes/alltypes.proto

@@ -0,0 +1,114 @@
+message SubMessage {
+    required string substuff1 = 1 [default = "1"];
+    required int32 substuff2 = 2 [default = 2];
+    optional fixed32 substuff3 = 3 [default = 3];
+}
+
+message EmptyMessage {
+
+}
+
+enum HugeEnum {
+    Negative = -2147483647; /* protoc doesn't accept -2147483648 here */
+    Positive =  2147483647;
+}
+
+message Limits {
+    required int32      int32_min  =  1 [default = 2147483647];
+    required int32      int32_max  =  2 [default = -2147483647];
+    required uint32     uint32_min =  3 [default = 4294967295];
+    required uint32     uint32_max =  4 [default = 0];
+    required int64      int64_min  =  5 [default = 9223372036854775807];
+    required int64      int64_max  =  6 [default = -9223372036854775807];
+    required uint64     uint64_min =  7 [default = 18446744073709551615];
+    required uint64     uint64_max =  8 [default = 0];
+    required HugeEnum   enum_min   =  9 [default = Positive];
+    required HugeEnum   enum_max   = 10 [default = Negative];
+}
+
+enum MyEnum {
+    Zero = 0;
+    First = 1;
+    Second = 2;
+    Truth = 42;
+}
+
+message AllTypes {
+    required int32      req_int32   = 1;
+    required int64      req_int64   = 2;
+    required uint32     req_uint32  = 3;
+    required uint64     req_uint64  = 4;
+    required sint32     req_sint32  = 5;
+    required sint64     req_sint64  = 6;
+    required bool       req_bool    = 7;
+    
+    required fixed32    req_fixed32 = 8;
+    required sfixed32   req_sfixed32= 9;
+    required float      req_float   = 10;
+    
+    required fixed64    req_fixed64 = 11;
+    required sfixed64   req_sfixed64= 12;
+    required double     req_double  = 13;
+    
+    required string     req_string  = 14;
+    required bytes      req_bytes   = 15;
+    required SubMessage req_submsg  = 16;
+    required MyEnum     req_enum    = 17;
+    required EmptyMessage req_emptymsg = 18;
+    
+    
+    repeated int32      rep_int32   = 21 [packed = true];
+    repeated int64      rep_int64   = 22 [packed = true];
+    repeated uint32     rep_uint32  = 23 [packed = true];
+    repeated uint64     rep_uint64  = 24 [packed = true];
+    repeated sint32     rep_sint32  = 25 [packed = true];
+    repeated sint64     rep_sint64  = 26 [packed = true];
+    repeated bool       rep_bool    = 27 [packed = true];
+    
+    repeated fixed32    rep_fixed32 = 28 [packed = true];
+    repeated sfixed32   rep_sfixed32= 29 [packed = true];
+    repeated float      rep_float   = 30 [packed = true];
+    
+    repeated fixed64    rep_fixed64 = 31 [packed = true];
+    repeated sfixed64   rep_sfixed64= 32 [packed = true];
+    repeated double     rep_double  = 33 [packed = true];
+    
+    repeated string     rep_string  = 34;
+    repeated bytes      rep_bytes   = 35;
+    repeated SubMessage rep_submsg  = 36;
+    repeated MyEnum     rep_enum    = 37 [packed = true];
+    repeated EmptyMessage rep_emptymsg = 38;
+    
+    optional int32      opt_int32   = 41 [default = 4041];
+    optional int64      opt_int64   = 42 [default = 4042];
+    optional uint32     opt_uint32  = 43 [default = 4043];
+    optional uint64     opt_uint64  = 44 [default = 4044];
+    optional sint32     opt_sint32  = 45 [default = 4045];
+    optional sint64     opt_sint64  = 46 [default = 4046];
+    optional bool       opt_bool    = 47 [default = false];
+    
+    optional fixed32    opt_fixed32 = 48 [default = 4048];
+    optional sfixed32   opt_sfixed32= 49 [default = 4049];
+    optional float      opt_float   = 50 [default = 4050];
+    
+    optional fixed64    opt_fixed64 = 51 [default = 4051];
+    optional sfixed64   opt_sfixed64= 52 [default = 4052];
+    optional double     opt_double  = 53 [default = 4053];
+    
+    optional string     opt_string  = 54 [default = "4054"];
+    optional bytes      opt_bytes   = 55 [default = "4055"];
+    optional SubMessage opt_submsg  = 56;
+    optional MyEnum     opt_enum    = 57 [default = Second];
+    optional EmptyMessage opt_emptymsg = 58;
+
+    // Check that extreme integer values are handled correctly
+    required Limits     req_limits = 98;
+
+    // Just to make sure that the size of the fields has been calculated
+    // properly, i.e. otherwise a bug in last field might not be detected.
+    required int32      end = 99;
+
+
+    extensions 200 to 255;
+}
+

tests/alltypes/decode_alltypes.c

@@ -8,6 +8,7 @@
 #include <stdlib.h>
 #include <pb_decode.h>
 #include "alltypes.pb.h"
+#include "test_helpers.h"
 
 #define TEST(x) if (!(x)) { \
     printf("Test " #x " failed.\n"); \
@@ -76,6 +77,7 @@ bool check_alltypes(pb_istream_t *stream, int mode)
     TEST(alltypes.rep_submsg[4].substuff3 == 2016 && alltypes.rep_submsg[0].substuff3 == 3);
     
     TEST(alltypes.rep_enum_count == 5 && alltypes.rep_enum[4] == MyEnum_Truth && alltypes.rep_enum[0] == MyEnum_Zero);
+    TEST(alltypes.rep_emptymsg_count == 5);
     
     if (mode == 0)
     {
@@ -120,6 +122,7 @@ bool check_alltypes(pb_istream_t *stream, int mode)
         TEST(alltypes.opt_submsg.substuff3 == 3);
         TEST(alltypes.has_opt_enum     == false);
         TEST(alltypes.opt_enum == MyEnum_Second);
+        TEST(alltypes.has_opt_emptymsg == false);
     }
     else
     {
@@ -164,8 +167,20 @@ bool check_alltypes(pb_istream_t *stream, int mode)
         TEST(alltypes.opt_submsg.substuff3 == 3);
         TEST(alltypes.has_opt_enum      == true);
         TEST(alltypes.opt_enum == MyEnum_Truth);
+        TEST(alltypes.has_opt_emptymsg  == true);
     }
     
+    TEST(alltypes.req_limits.int32_min  == INT32_MIN);
+    TEST(alltypes.req_limits.int32_max  == INT32_MAX);
+    TEST(alltypes.req_limits.uint32_min == 0);
+    TEST(alltypes.req_limits.uint32_max == UINT32_MAX);
+    TEST(alltypes.req_limits.int64_min  == INT64_MIN);
+    TEST(alltypes.req_limits.int64_max  == INT64_MAX);
+    TEST(alltypes.req_limits.uint64_min == 0);
+    TEST(alltypes.req_limits.uint64_max == UINT64_MAX);
+    TEST(alltypes.req_limits.enum_min   == HugeEnum_Negative);
+    TEST(alltypes.req_limits.enum_max   == HugeEnum_Positive);
+    
     TEST(alltypes.end == 1099);
     
     return true;
@@ -173,15 +188,19 @@ bool check_alltypes(pb_istream_t *stream, int mode)
 
 int main(int argc, char **argv)
 {
+    uint8_t buffer[1024];
+    size_t count;
+    pb_istream_t stream;
+
     /* Whether to expect the optional values or the default values. */
     int mode = (argc > 1) ? atoi(argv[1]) : 0;
     
     /* Read the data into buffer */
-    uint8_t buffer[1024];
+    SET_BINARY_MODE(stdin);
-    size_t count = fread(buffer, 1, sizeof(buffer), stdin);
+    count = fread(buffer, 1, sizeof(buffer), stdin);
     
     /* Construct a pb_istream_t for reading from the buffer */
-    pb_istream_t stream = pb_istream_from_buffer(buffer, count);
+    stream = pb_istream_from_buffer(buffer, count);
     
     /* Decode and print out the stuff */
     if (!check_alltypes(&stream, mode))

tests/alltypes/encode_alltypes.c

@@ -6,6 +6,7 @@
 #include <string.h>
 #include <pb_encode.h>
 #include "alltypes.pb.h"
+#include "test_helpers.h"
 
 int main(int argc, char **argv)
 {
@@ -64,6 +65,18 @@ int main(int argc, char **argv)
     alltypes.rep_submsg[4].substuff3 = 2016;
     
     alltypes.rep_enum_count = 5; alltypes.rep_enum[4] = MyEnum_Truth;
+    alltypes.rep_emptymsg_count = 5;
+    
+    alltypes.req_limits.int32_min  = INT32_MIN;
+    alltypes.req_limits.int32_max  = INT32_MAX;
+    alltypes.req_limits.uint32_min = 0;
+    alltypes.req_limits.uint32_max = UINT32_MAX;
+    alltypes.req_limits.int64_min  = INT64_MIN;
+    alltypes.req_limits.int64_max  = INT64_MAX;
+    alltypes.req_limits.uint64_min = 0;
+    alltypes.req_limits.uint64_max = UINT64_MAX;
+    alltypes.req_limits.enum_min   = HugeEnum_Negative;
+    alltypes.req_limits.enum_max   = HugeEnum_Positive;
     
     if (mode != 0)
     {
@@ -107,22 +120,26 @@ int main(int argc, char **argv)
         alltypes.opt_submsg.substuff2 = 3056;
         alltypes.has_opt_enum = true;
         alltypes.opt_enum = MyEnum_Truth;
+        alltypes.has_opt_emptymsg = true;
     }
     
     alltypes.end = 1099;
     
-    uint8_t buffer[1024];
+    {
+        uint8_t buffer[AllTypes_size];
         pb_ostream_t stream = pb_ostream_from_buffer(buffer, sizeof(buffer));
         
         /* Now encode it and check if we succeeded. */
         if (pb_encode(&stream, AllTypes_fields, &alltypes))
         {
+            SET_BINARY_MODE(stdout);
             fwrite(buffer, 1, stream.bytes_written, stdout);
             return 0; /* Success */
         }
         else
         {
-        fprintf(stderr, "Encoding failed!\n");
+            fprintf(stderr, "Encoding failed: %s\n", PB_GET_ERROR(&stream));
             return 1; /* Failure */
         }
     }
+}

tests/alltypes_callback/SConscript

@@ -0,0 +1,23 @@
+# Test the AllTypes encoding & decoding using callbacks for all fields.
+
+Import("env")
+
+c = Copy("$TARGET", "$SOURCE")
+env.Command("alltypes.proto", "#alltypes/alltypes.proto", c)
+
+env.NanopbProto(["alltypes", "alltypes.options"])
+enc = env.Program(["encode_alltypes_callback.c", "alltypes.pb.c", "$COMMON/pb_encode.o"])
+dec = env.Program(["decode_alltypes_callback.c", "alltypes.pb.c", "$COMMON/pb_decode.o"])
+
+refdec = "$BUILD/alltypes/decode_alltypes$PROGSUFFIX"
+
+# Encode and compare results
+env.RunTest(enc)
+env.RunTest("decode_alltypes.output", [refdec, "encode_alltypes_callback.output"])
+env.RunTest("decode_alltypes_callback.output", [dec, "encode_alltypes_callback.output"])
+
+# Do the same thing with the optional fields present
+env.RunTest("optionals.output", enc, ARGS = ['1'])
+env.RunTest("optionals.refdecout", [refdec, "optionals.output"], ARGS = ['1'])
+env.RunTest("optionals.decout", [dec, "optionals.output"], ARGS = ['1'])
+

tests/alltypes_callback/alltypes.options

@@ -0,0 +1,3 @@
+# Generate all fields as callbacks.
+AllTypes.*              type:FT_CALLBACK
+SubMessage.substuff1    max_size:16

tests/alltypes_callback/decode_alltypes_callback.c

@@ -0,0 +1,423 @@
+/* Attempts to test all the datatypes supported by ProtoBuf when used as callback fields.
+ * Note that normally there would be no reason to use callback fields for this,
+ * because each encoder defined here only gives a single field.
+ */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdlib.h>
+#include <pb_decode.h>
+#include "alltypes.pb.h"
+#include "test_helpers.h"
+
+#define TEST(x) if (!(x)) { \
+    printf("Test " #x " failed (in field %d).\n", field->tag); \
+    return false; \
+    }
+
+static bool read_varint(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    uint64_t value;
+    if (!pb_decode_varint(stream, &value))
+        return false;
+    
+    TEST((int64_t)value == (long)*arg);
+    return true;
+}
+
+static bool read_svarint(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    int64_t value;
+    if (!pb_decode_svarint(stream, &value))
+        return false;
+    
+    TEST(value == (long)*arg);
+    return true;
+}
+
+static bool read_fixed32(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    uint32_t value;
+    if (!pb_decode_fixed32(stream, &value))
+        return false;
+    
+    TEST(value == *(uint32_t*)*arg);
+    return true;
+}
+
+static bool read_fixed64(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    uint64_t value;
+    if (!pb_decode_fixed64(stream, &value))
+        return false;
+    
+    TEST(value == *(uint64_t*)*arg);
+    return true;
+}
+
+static bool read_string(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    uint8_t buf[16] = {0};
+    size_t len = stream->bytes_left;
+    
+    if (len > sizeof(buf) - 1 || !pb_read(stream, buf, len))
+        return false;
+    
+    TEST(strcmp((char*)buf, *arg) == 0);
+    return true;
+}
+
+static bool read_submsg(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    SubMessage submsg = {""};
+    
+    if (!pb_decode(stream, SubMessage_fields, &submsg))
+        return false;
+    
+    TEST(memcmp(&submsg, *arg, sizeof(submsg)));
+    return true;
+}
+
+static bool read_emptymsg(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    EmptyMessage emptymsg = {0};
+    return pb_decode(stream, EmptyMessage_fields, &emptymsg);
+}
+
+static bool read_repeated_varint(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    int32_t** expected = (int32_t**)arg;
+    uint64_t value;
+    if (!pb_decode_varint(stream, &value))
+        return false;
+
+    TEST(*(*expected)++ == value);
+    return true;
+}
+
+static bool read_repeated_svarint(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    int32_t** expected = (int32_t**)arg;
+    int64_t value;
+    if (!pb_decode_svarint(stream, &value))
+        return false;
+
+    TEST(*(*expected)++ == value);
+    return true;
+}
+
+static bool read_repeated_fixed32(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    uint32_t** expected = (uint32_t**)arg;
+    uint32_t value;
+    if (!pb_decode_fixed32(stream, &value))
+        return false;
+
+    TEST(*(*expected)++ == value);
+    return true;
+}
+
+static bool read_repeated_fixed64(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    uint64_t** expected = (uint64_t**)arg;
+    uint64_t value;
+    if (!pb_decode_fixed64(stream, &value))
+        return false;
+
+    TEST(*(*expected)++ == value);
+    return true;
+}
+
+static bool read_repeated_string(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    uint8_t*** expected = (uint8_t***)arg;
+    uint8_t buf[16] = {0};
+    size_t len = stream->bytes_left;
+    
+    if (len > sizeof(buf) - 1 || !pb_read(stream, buf, len))
+        return false;
+    
+    TEST(strcmp((char*)*(*expected)++, (char*)buf) == 0);
+    return true;
+}
+
+static bool read_repeated_submsg(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    SubMessage** expected = (SubMessage**)arg;
+    SubMessage decoded = {""};
+    if (!pb_decode(stream, SubMessage_fields, &decoded))
+        return false;
+
+    TEST(memcmp((*expected)++, &decoded, sizeof(decoded)) == 0);
+    return true;
+}
+
+static bool read_limits(pb_istream_t *stream, const pb_field_t *field, void **arg)
+{
+    Limits decoded = {0};
+    if (!pb_decode(stream, Limits_fields, &decoded))
+        return false;
+
+    TEST(decoded.int32_min  == INT32_MIN);
+    TEST(decoded.int32_max  == INT32_MAX);
+    TEST(decoded.uint32_min == 0);
+    TEST(decoded.uint32_max == UINT32_MAX);
+    TEST(decoded.int64_min  == INT64_MIN);
+    TEST(decoded.int64_max  == INT64_MAX);
+    TEST(decoded.uint64_min == 0);
+    TEST(decoded.uint64_max == UINT64_MAX);
+    TEST(decoded.enum_min   == HugeEnum_Negative);
+    TEST(decoded.enum_max   == HugeEnum_Positive);
+    
+    return true;
+}
+
+/* This function is called once from main(), it handles
+   the decoding and checks the fields. */
+bool check_alltypes(pb_istream_t *stream, int mode)
+{
+    /* Values for use from callbacks through pointers. */
+    uint32_t    req_fixed32     = 1008;
+    int32_t     req_sfixed32    = -1009;
+    float       req_float       = 1010.0f;
+    uint64_t    req_fixed64     = 1011;
+    int64_t     req_sfixed64    = -1012;
+    double      req_double      = 1013.0;
+    SubMessage  req_submsg      = {"1016", 1016};
+    
+    int32_t     rep_int32[5]    = {0, 0, 0, 0, -2001};
+    int32_t     rep_int64[5]    = {0, 0, 0, 0, -2002};
+    int32_t     rep_uint32[5]   = {0, 0, 0, 0,  2003};
+    int32_t     rep_uint64[5]   = {0, 0, 0, 0,  2004};
+    int32_t     rep_sint32[5]   = {0, 0, 0, 0, -2005};
+    int32_t     rep_sint64[5]   = {0, 0, 0, 0, -2006};
+    int32_t     rep_bool[5]     = {false, false, false, false, true};
+    uint32_t    rep_fixed32[5]  = {0, 0, 0, 0,  2008};
+    int32_t     rep_sfixed32[5] = {0, 0, 0, 0, -2009};
+    float       rep_float[5]    = {0, 0, 0, 0,  2010.0f};
+    uint64_t    rep_fixed64[5]  = {0, 0, 0, 0,  2011};
+    int64_t     rep_sfixed64[5] = {0, 0, 0, 0, -2012};
+    double      rep_double[5]   = {0, 0, 0, 0,  2013.0};
+    char*       rep_string[5]   = {"", "", "", "", "2014"};
+    char*       rep_bytes[5]    = {"", "", "", "", "2015"};
+    SubMessage  rep_submsg[5]   = {{"", 0, 0, 3},
+                                   {"", 0, 0, 3},
+                                   {"", 0, 0, 3},
+                                   {"", 0, 0, 3},
+                                   {"2016", 2016, true, 2016}};
+    int32_t     rep_enum[5]     = {0, 0, 0, 0, MyEnum_Truth};
+    
+    uint32_t    opt_fixed32     = 3048;
+    int32_t     opt_sfixed32    = 3049;
+    float       opt_float       = 3050.0f;
+    uint64_t    opt_fixed64     = 3051;
+    int64_t     opt_sfixed64    = 3052;
+    double      opt_double      = 3053.0f;
+    SubMessage  opt_submsg      = {"3056", 3056};
+    
+    /* Bind callbacks for required fields */
+    AllTypes alltypes;
+    
+    /* Fill with garbage to better detect initialization errors */
+    memset(&alltypes, 0xAA, sizeof(alltypes));
+    
+    alltypes.req_int32.funcs.decode = &read_varint;
+    alltypes.req_int32.arg = (void*)-1001;
+    
+    alltypes.req_int64.funcs.decode = &read_varint;
+    alltypes.req_int64.arg = (void*)-1002;
+    
+    alltypes.req_uint32.funcs.decode = &read_varint;
+    alltypes.req_uint32.arg = (void*)1003;
+
+    alltypes.req_uint32.funcs.decode = &read_varint;
+    alltypes.req_uint32.arg = (void*)1003;    
+    
+    alltypes.req_uint64.funcs.decode = &read_varint;
+    alltypes.req_uint64.arg = (void*)1004;
+    
+    alltypes.req_sint32.funcs.decode = &read_svarint;
+    alltypes.req_sint32.arg = (void*)-1005;   
+    
+    alltypes.req_sint64.funcs.decode = &read_svarint;
+    alltypes.req_sint64.arg = (void*)-1006;   
+    
+    alltypes.req_bool.funcs.decode = &read_varint;
+    alltypes.req_bool.arg = (void*)true;   
+    
+    alltypes.req_fixed32.funcs.decode = &read_fixed32;
+    alltypes.req_fixed32.arg = &req_fixed32;
+    
+    alltypes.req_sfixed32.funcs.decode = &read_fixed32;
+    alltypes.req_sfixed32.arg = &req_sfixed32;
+    
+    alltypes.req_float.funcs.decode = &read_fixed32;
+    alltypes.req_float.arg = &req_float;
+    
+    alltypes.req_fixed64.funcs.decode = &read_fixed64;
+    alltypes.req_fixed64.arg = &req_fixed64;
+    
+    alltypes.req_sfixed64.funcs.decode = &read_fixed64;
+    alltypes.req_sfixed64.arg = &req_sfixed64;
+    
+    alltypes.req_double.funcs.decode = &read_fixed64;
+    alltypes.req_double.arg = &req_double;
+    
+    alltypes.req_string.funcs.decode = &read_string;
+    alltypes.req_string.arg = "1014";
+    
+    alltypes.req_bytes.funcs.decode = &read_string;
+    alltypes.req_bytes.arg = "1015";
+    
+    alltypes.req_submsg.funcs.decode = &read_submsg;
+    alltypes.req_submsg.arg = &req_submsg;
+    
+    alltypes.req_enum.funcs.decode = &read_varint;
+    alltypes.req_enum.arg = (void*)MyEnum_Truth;
+    
+    alltypes.req_emptymsg.funcs.decode = &read_emptymsg;
+    
+    /* Bind callbacks for repeated fields */
+    alltypes.rep_int32.funcs.decode = &read_repeated_varint;
+    alltypes.rep_int32.arg = rep_int32;
+    
+    alltypes.rep_int64.funcs.decode = &read_repeated_varint;
+    alltypes.rep_int64.arg = rep_int64;
+    
+    alltypes.rep_uint32.funcs.decode = &read_repeated_varint;
+    alltypes.rep_uint32.arg = rep_uint32;
+    
+    alltypes.rep_uint64.funcs.decode = &read_repeated_varint;
+    alltypes.rep_uint64.arg = rep_uint64;
+    
+    alltypes.rep_sint32.funcs.decode = &read_repeated_svarint;
+    alltypes.rep_sint32.arg = rep_sint32;
+    
+    alltypes.rep_sint64.funcs.decode = &read_repeated_svarint;
+    alltypes.rep_sint64.arg = rep_sint64;
+    
+    alltypes.rep_bool.funcs.decode = &read_repeated_varint;
+    alltypes.rep_bool.arg = rep_bool;
+    
+    alltypes.rep_fixed32.funcs.decode = &read_repeated_fixed32;
+    alltypes.rep_fixed32.arg = rep_fixed32;
+    
+    alltypes.rep_sfixed32.funcs.decode = &read_repeated_fixed32;
+    alltypes.rep_sfixed32.arg = rep_sfixed32;
+    
+    alltypes.rep_float.funcs.decode = &read_repeated_fixed32;
+    alltypes.rep_float.arg = rep_float;
+    
+    alltypes.rep_fixed64.funcs.decode = &read_repeated_fixed64;
+    alltypes.rep_fixed64.arg = rep_fixed64;
+    
+    alltypes.rep_sfixed64.funcs.decode = &read_repeated_fixed64;
+    alltypes.rep_sfixed64.arg = rep_sfixed64;
+    
+    alltypes.rep_double.funcs.decode = &read_repeated_fixed64;
+    alltypes.rep_double.arg = rep_double;
+    
+    alltypes.rep_string.funcs.decode = &read_repeated_string;
+    alltypes.rep_string.arg = rep_string;
+    
+    alltypes.rep_bytes.funcs.decode = &read_repeated_string;
+    alltypes.rep_bytes.arg = rep_bytes;
+    
+    alltypes.rep_submsg.funcs.decode = &read_repeated_submsg;
+    alltypes.rep_submsg.arg = rep_submsg;
+    
+    alltypes.rep_enum.funcs.decode = &read_repeated_varint;
+    alltypes.rep_enum.arg = rep_enum;
+    
+    alltypes.rep_emptymsg.funcs.decode = &read_emptymsg;
+    
+    alltypes.req_limits.funcs.decode = &read_limits;
+    
+    alltypes.end.funcs.decode = &read_varint;
+    alltypes.end.arg = (void*)1099;
+    
+    /* Bind callbacks for optional fields */
+    if (mode == 1)
+    {
+        alltypes.opt_int32.funcs.decode = &read_varint;
+        alltypes.opt_int32.arg = (void*)3041;
+        
+        alltypes.opt_int64.funcs.decode = &read_varint;
+        alltypes.opt_int64.arg = (void*)3042;
+        
+        alltypes.opt_uint32.funcs.decode = &read_varint;
+        alltypes.opt_uint32.arg = (void*)3043;
+        
+        alltypes.opt_uint64.funcs.decode = &read_varint;
+        alltypes.opt_uint64.arg = (void*)3044;
+        
+        alltypes.opt_sint32.funcs.decode = &read_svarint;
+        alltypes.opt_sint32.arg = (void*)3045;
+        
+        alltypes.opt_sint64.funcs.decode = &read_svarint;
+        alltypes.opt_sint64.arg = (void*)3046;
+        
+        alltypes.opt_bool.funcs.decode = &read_varint;
+        alltypes.opt_bool.arg = (void*)true;
+
+        alltypes.opt_fixed32.funcs.decode = &read_fixed32;
+        alltypes.opt_fixed32.arg = &opt_fixed32;
+        
+        alltypes.opt_sfixed32.funcs.decode = &read_fixed32;
+        alltypes.opt_sfixed32.arg = &opt_sfixed32;
+        
+        alltypes.opt_float.funcs.decode = &read_fixed32;
+        alltypes.opt_float.arg = &opt_float;
+        
+        alltypes.opt_fixed64.funcs.decode = &read_fixed64;
+        alltypes.opt_fixed64.arg = &opt_fixed64;
+        
+        alltypes.opt_sfixed64.funcs.decode = &read_fixed64;
+        alltypes.opt_sfixed64.arg = &opt_sfixed64;
+        
+        alltypes.opt_double.funcs.decode = &read_fixed64;
+        alltypes.opt_double.arg = &opt_double;
+        
+        alltypes.opt_string.funcs.decode = &read_string;
+        alltypes.opt_string.arg = "3054";
+        
+        alltypes.opt_bytes.funcs.decode = &read_string;
+        alltypes.opt_bytes.arg = "3055";
+        
+        alltypes.opt_submsg.funcs.decode = &read_submsg;
+        alltypes.opt_submsg.arg = &opt_submsg;
+        
+        alltypes.opt_enum.funcs.decode = &read_varint;
+        alltypes.opt_enum.arg = (void*)MyEnum_Truth;
+        
+        alltypes.opt_emptymsg.funcs.decode = &read_emptymsg;
+    }
+    
+    return pb_decode(stream, AllTypes_fields, &alltypes);
+}
+
+int main(int argc, char **argv)
+{
+    uint8_t buffer[1024];
+    size_t count;
+    pb_istream_t stream;
+
+    /* Whether to expect the optional values or the default values. */
+    int mode = (argc > 1) ? atoi(argv[1]) : 0;
+    
+    /* Read the data into buffer */
+    SET_BINARY_MODE(stdin);
+    count = fread(buffer, 1, sizeof(buffer), stdin);
+    
+    /* Construct a pb_istream_t for reading from the buffer */
+    stream = pb_istream_from_buffer(buffer, count);
+    
+    /* Decode and print out the stuff */
+    if (!check_alltypes(&stream, mode))
+    {
+        printf("Parsing failed: %s\n", PB_GET_ERROR(&stream));
+        return 1;
+    } else {
+        return 0;
+    }
+}

tests/alltypes_callback/encode_alltypes_callback.c

@@ -0,0 +1,397 @@
+/* Attempts to test all the datatypes supported by ProtoBuf when used as callback fields.
+ * Note that normally there would be no reason to use callback fields for this,
+ * because each encoder defined here only gives a single field.
+ */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <pb_encode.h>
+#include "alltypes.pb.h"
+#include "test_helpers.h"
+
+static bool write_varint(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    return pb_encode_tag_for_field(stream, field) &&
+           pb_encode_varint(stream, (long)*arg);
+}
+
+static bool write_svarint(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    return pb_encode_tag_for_field(stream, field) &&
+           pb_encode_svarint(stream, (long)*arg);
+}
+
+static bool write_fixed32(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    return pb_encode_tag_for_field(stream, field) &&
+           pb_encode_fixed32(stream, *arg);
+}
+
+static bool write_fixed64(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    return pb_encode_tag_for_field(stream, field) &&
+           pb_encode_fixed64(stream, *arg);
+}
+
+static bool write_string(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    return pb_encode_tag_for_field(stream, field) &&
+           pb_encode_string(stream, *arg, strlen(*arg));
+}
+
+static bool write_submsg(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+   
+    return pb_encode_tag_for_field(stream, field) &&
+           pb_encode_submessage(stream, SubMessage_fields, *arg);
+}
+
+static bool write_emptymsg(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    EmptyMessage emptymsg = {0};
+    return pb_encode_tag_for_field(stream, field) &&
+           pb_encode_submessage(stream, EmptyMessage_fields, &emptymsg);
+}
+
+static bool write_repeated_varint(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    return pb_encode_tag_for_field(stream, field) &&
+           pb_encode_varint(stream, 0) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_varint(stream, 0) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_varint(stream, 0) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_varint(stream, 0) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_varint(stream, (long)*arg);
+}
+
+static bool write_repeated_svarint(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    return pb_encode_tag_for_field(stream, field) &&
+           pb_encode_svarint(stream, 0) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_svarint(stream, 0) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_svarint(stream, 0) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_svarint(stream, 0) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_svarint(stream, (long)*arg);
+}
+
+static bool write_repeated_fixed32(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    uint32_t dummy = 0;
+
+    /* Make it a packed field */
+    return pb_encode_tag(stream, PB_WT_STRING, field->tag) &&
+           pb_encode_varint(stream, 5 * 4) && /* Number of bytes */
+           pb_encode_fixed32(stream, &dummy) &&
+           pb_encode_fixed32(stream, &dummy) &&
+           pb_encode_fixed32(stream, &dummy) &&
+           pb_encode_fixed32(stream, &dummy) &&
+           pb_encode_fixed32(stream, *arg);
+}
+
+static bool write_repeated_fixed64(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    uint64_t dummy = 0;
+
+    /* Make it a packed field */
+    return pb_encode_tag(stream, PB_WT_STRING, field->tag) &&
+           pb_encode_varint(stream, 5 * 8) && /* Number of bytes */
+           pb_encode_fixed64(stream, &dummy) &&
+           pb_encode_fixed64(stream, &dummy) &&
+           pb_encode_fixed64(stream, &dummy) &&
+           pb_encode_fixed64(stream, &dummy) &&
+           pb_encode_fixed64(stream, *arg);
+}
+
+static bool write_repeated_string(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    return pb_encode_tag_for_field(stream, field) &&
+           pb_encode_string(stream, 0, 0) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_string(stream, 0, 0) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_string(stream, 0, 0) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_string(stream, 0, 0) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_string(stream, *arg, strlen(*arg));
+}
+
+static bool write_repeated_submsg(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    SubMessage dummy = {""};
+
+    return pb_encode_tag_for_field(stream, field) &&
+           pb_encode_submessage(stream, SubMessage_fields, &dummy) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_submessage(stream, SubMessage_fields, &dummy) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_submessage(stream, SubMessage_fields, &dummy) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_submessage(stream, SubMessage_fields, &dummy) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_submessage(stream, SubMessage_fields, *arg);
+}
+
+static bool write_limits(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    Limits limits = {0};
+    limits.int32_min  = INT32_MIN;
+    limits.int32_max  = INT32_MAX;
+    limits.uint32_min = 0;
+    limits.uint32_max = UINT32_MAX;
+    limits.int64_min  = INT64_MIN;
+    limits.int64_max  = INT64_MAX;
+    limits.uint64_min = 0;
+    limits.uint64_max = UINT64_MAX;
+    limits.enum_min   = HugeEnum_Negative;
+    limits.enum_max   = HugeEnum_Positive;
+   
+    return pb_encode_tag_for_field(stream, field) &&
+           pb_encode_submessage(stream, Limits_fields, &limits);
+}
+
+static bool write_repeated_emptymsg(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    EmptyMessage emptymsg = {0};
+    return pb_encode_tag_for_field(stream, field) &&
+           pb_encode_submessage(stream, EmptyMessage_fields, &emptymsg) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_submessage(stream, EmptyMessage_fields, &emptymsg) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_submessage(stream, EmptyMessage_fields, &emptymsg) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_submessage(stream, EmptyMessage_fields, &emptymsg) &&
+           pb_encode_tag_for_field(stream, field) &&
+           pb_encode_submessage(stream, EmptyMessage_fields, &emptymsg);
+}
+
+int main(int argc, char **argv)
+{
+    int mode = (argc > 1) ? atoi(argv[1]) : 0;
+
+    /* Values for use from callbacks through pointers. */
+    uint32_t    req_fixed32     = 1008;
+    int32_t     req_sfixed32    = -1009;
+    float       req_float       = 1010.0f;
+    uint64_t    req_fixed64     = 1011;
+    int64_t     req_sfixed64    = -1012;
+    double      req_double      = 1013.0;
+    SubMessage  req_submsg      = {"1016", 1016};
+    
+    uint32_t    rep_fixed32     = 2008;
+    int32_t     rep_sfixed32    = -2009;
+    float       rep_float       = 2010.0f;
+    uint64_t    rep_fixed64     = 2011;
+    int64_t     rep_sfixed64    = -2012;
+    double      rep_double      = 2013.0;
+    SubMessage  rep_submsg      = {"2016", 2016, true, 2016};
+    
+    uint32_t    opt_fixed32     = 3048;
+    int32_t     opt_sfixed32    = 3049;
+    float       opt_float       = 3050.0f;
+    uint64_t    opt_fixed64     = 3051;
+    int64_t     opt_sfixed64    = 3052;
+    double      opt_double      = 3053.0f;
+    SubMessage  opt_submsg      = {"3056", 3056};
+    
+    /* Bind callbacks for required fields */
+    AllTypes alltypes = {{{0}}};
+    
+    alltypes.req_int32.funcs.encode = &write_varint;
+    alltypes.req_int32.arg = (void*)-1001;
+    
+    alltypes.req_int64.funcs.encode = &write_varint;
+    alltypes.req_int64.arg = (void*)-1002;
+    
+    alltypes.req_uint32.funcs.encode = &write_varint;
+    alltypes.req_uint32.arg = (void*)1003;
+
+    alltypes.req_uint32.funcs.encode = &write_varint;
+    alltypes.req_uint32.arg = (void*)1003;    
+    
+    alltypes.req_uint64.funcs.encode = &write_varint;
+    alltypes.req_uint64.arg = (void*)1004;
+    
+    alltypes.req_sint32.funcs.encode = &write_svarint;
+    alltypes.req_sint32.arg = (void*)-1005;   
+    
+    alltypes.req_sint64.funcs.encode = &write_svarint;
+    alltypes.req_sint64.arg = (void*)-1006;   
+    
+    alltypes.req_bool.funcs.encode = &write_varint;
+    alltypes.req_bool.arg = (void*)true;   
+    
+    alltypes.req_fixed32.funcs.encode = &write_fixed32;
+    alltypes.req_fixed32.arg = &req_fixed32;
+    
+    alltypes.req_sfixed32.funcs.encode = &write_fixed32;
+    alltypes.req_sfixed32.arg = &req_sfixed32;
+    
+    alltypes.req_float.funcs.encode = &write_fixed32;
+    alltypes.req_float.arg = &req_float;
+    
+    alltypes.req_fixed64.funcs.encode = &write_fixed64;
+    alltypes.req_fixed64.arg = &req_fixed64;
+    
+    alltypes.req_sfixed64.funcs.encode = &write_fixed64;
+    alltypes.req_sfixed64.arg = &req_sfixed64;
+    
+    alltypes.req_double.funcs.encode = &write_fixed64;
+    alltypes.req_double.arg = &req_double;
+    
+    alltypes.req_string.funcs.encode = &write_string;
+    alltypes.req_string.arg = "1014";
+    
+    alltypes.req_bytes.funcs.encode = &write_string;
+    alltypes.req_bytes.arg = "1015";
+    
+    alltypes.req_submsg.funcs.encode = &write_submsg;
+    alltypes.req_submsg.arg = &req_submsg;
+    
+    alltypes.req_enum.funcs.encode = &write_varint;
+    alltypes.req_enum.arg = (void*)MyEnum_Truth;
+    
+    alltypes.req_emptymsg.funcs.encode = &write_emptymsg;
+    
+    /* Bind callbacks for repeated fields */
+    alltypes.rep_int32.funcs.encode = &write_repeated_varint;
+    alltypes.rep_int32.arg = (void*)-2001;
+    
+    alltypes.rep_int64.funcs.encode = &write_repeated_varint;
+    alltypes.rep_int64.arg = (void*)-2002;
+    
+    alltypes.rep_uint32.funcs.encode = &write_repeated_varint;
+    alltypes.rep_uint32.arg = (void*)2003;
+    
+    alltypes.rep_uint64.funcs.encode = &write_repeated_varint;
+    alltypes.rep_uint64.arg = (void*)2004;
+    
+    alltypes.rep_sint32.funcs.encode = &write_repeated_svarint;
+    alltypes.rep_sint32.arg = (void*)-2005;
+    
+    alltypes.rep_sint64.funcs.encode = &write_repeated_svarint;
+    alltypes.rep_sint64.arg = (void*)-2006;
+    
+    alltypes.rep_bool.funcs.encode = &write_repeated_varint;
+    alltypes.rep_bool.arg = (void*)true;
+    
+    alltypes.rep_fixed32.funcs.encode = &write_repeated_fixed32;
+    alltypes.rep_fixed32.arg = &rep_fixed32;
+
+    alltypes.rep_sfixed32.funcs.encode = &write_repeated_fixed32;
+    alltypes.rep_sfixed32.arg = &rep_sfixed32;
+    
+    alltypes.rep_float.funcs.encode = &write_repeated_fixed32;
+    alltypes.rep_float.arg = &rep_float;
+    
+    alltypes.rep_fixed64.funcs.encode = &write_repeated_fixed64;
+    alltypes.rep_fixed64.arg = &rep_fixed64;
+
+    alltypes.rep_sfixed64.funcs.encode = &write_repeated_fixed64;
+    alltypes.rep_sfixed64.arg = &rep_sfixed64;
+    
+    alltypes.rep_double.funcs.encode = &write_repeated_fixed64;
+    alltypes.rep_double.arg = &rep_double;
+    
+    alltypes.rep_string.funcs.encode = &write_repeated_string;
+    alltypes.rep_string.arg = "2014";
+    
+    alltypes.rep_bytes.funcs.encode = &write_repeated_string;
+    alltypes.rep_bytes.arg = "2015";
+    
+    alltypes.rep_submsg.funcs.encode = &write_repeated_submsg;
+    alltypes.rep_submsg.arg = &rep_submsg;
+    
+    alltypes.rep_enum.funcs.encode = &write_repeated_varint;
+    alltypes.rep_enum.arg = (void*)MyEnum_Truth;
+    
+    alltypes.rep_emptymsg.funcs.encode = &write_repeated_emptymsg;
+    
+    alltypes.req_limits.funcs.encode = &write_limits;
+    
+    /* Bind callbacks for optional fields */
+    if (mode != 0)
+    {
+        alltypes.opt_int32.funcs.encode = &write_varint;
+        alltypes.opt_int32.arg = (void*)3041;
+        
+        alltypes.opt_int64.funcs.encode = &write_varint;
+        alltypes.opt_int64.arg = (void*)3042;
+        
+        alltypes.opt_uint32.funcs.encode = &write_varint;
+        alltypes.opt_uint32.arg = (void*)3043;
+        
+        alltypes.opt_uint64.funcs.encode = &write_varint;
+        alltypes.opt_uint64.arg = (void*)3044;
+        
+        alltypes.opt_sint32.funcs.encode = &write_svarint;
+        alltypes.opt_sint32.arg = (void*)3045;
+        
+        alltypes.opt_sint64.funcs.encode = &write_svarint;
+        alltypes.opt_sint64.arg = (void*)3046;
+        
+        alltypes.opt_bool.funcs.encode = &write_varint;
+        alltypes.opt_bool.arg = (void*)true;
+
+        alltypes.opt_fixed32.funcs.encode = &write_fixed32;
+        alltypes.opt_fixed32.arg = &opt_fixed32;
+        
+        alltypes.opt_sfixed32.funcs.encode = &write_fixed32;
+        alltypes.opt_sfixed32.arg = &opt_sfixed32;
+        
+        alltypes.opt_float.funcs.encode = &write_fixed32;
+        alltypes.opt_float.arg = &opt_float;
+        
+        alltypes.opt_fixed64.funcs.encode = &write_fixed64;
+        alltypes.opt_fixed64.arg = &opt_fixed64;
+        
+        alltypes.opt_sfixed64.funcs.encode = &write_fixed64;
+        alltypes.opt_sfixed64.arg = &opt_sfixed64;
+        
+        alltypes.opt_double.funcs.encode = &write_fixed64;
+        alltypes.opt_double.arg = &opt_double;
+        
+        alltypes.opt_string.funcs.encode = &write_string;
+        alltypes.opt_string.arg = "3054";
+        
+        alltypes.opt_bytes.funcs.encode = &write_string;
+        alltypes.opt_bytes.arg = "3055";
+        
+        alltypes.opt_submsg.funcs.encode = &write_submsg;
+        alltypes.opt_submsg.arg = &opt_submsg;
+        
+        alltypes.opt_enum.funcs.encode = &write_varint;
+        alltypes.opt_enum.arg = (void*)MyEnum_Truth;
+        
+        alltypes.opt_emptymsg.funcs.encode = &write_emptymsg;
+    }
+    
+    alltypes.end.funcs.encode = &write_varint;
+    alltypes.end.arg = (void*)1099;
+    
+    {
+        uint8_t buffer[2048];
+        pb_ostream_t stream = pb_ostream_from_buffer(buffer, sizeof(buffer));
+        
+        /* Now encode it and check if we succeeded. */
+        if (pb_encode(&stream, AllTypes_fields, &alltypes))
+        {
+            SET_BINARY_MODE(stdout);
+            fwrite(buffer, 1, stream.bytes_written, stdout);
+            return 0; /* Success */
+        }
+        else
+        {
+            fprintf(stderr, "Encoding failed: %s\n", PB_GET_ERROR(&stream));
+            return 1; /* Failure */
+        }
+    }
+}

tests/alltypes_pointer/SConscript

@@ -0,0 +1,48 @@
+# Encode the AllTypes message using pointers for all fields, and verify the
+# output against the normal AllTypes test case.
+
+Import("env")
+
+# We need our own pb_decode.o for the malloc support
+env = env.Clone()
+env.Append(CPPDEFINES = {'PB_ENABLE_MALLOC': 1});
+
+# Disable libmudflap, because it will confuse valgrind
+# and other memory leak detection tools.
+if '-fmudflap' in env["CCFLAGS"]:
+    env["CCFLAGS"].remove("-fmudflap")
+    env["LINKFLAGS"].remove("-fmudflap")
+    env["LIBS"].remove("mudflap")
+
+strict = env.Clone()
+strict.Append(CFLAGS = strict['CORECFLAGS'])
+strict.Object("pb_decode_with_malloc.o", "$NANOPB/pb_decode.c")
+strict.Object("pb_encode_with_malloc.o", "$NANOPB/pb_encode.c")
+
+c = Copy("$TARGET", "$SOURCE")
+env.Command("alltypes.proto", "#alltypes/alltypes.proto", c)
+
+env.NanopbProto(["alltypes", "alltypes.options"])
+enc = env.Program(["encode_alltypes_pointer.c", "alltypes.pb.c", "pb_encode_with_malloc.o"])
+dec = env.Program(["decode_alltypes_pointer.c", "alltypes.pb.c", "pb_decode_with_malloc.o"])
+
+# Encode and compare results to non-pointer alltypes test case
+env.RunTest(enc)
+env.Compare(["encode_alltypes_pointer.output", "$BUILD/alltypes/encode_alltypes.output"])
+
+# Decode (under valgrind if available)
+valgrind = env.WhereIs('valgrind')
+kwargs = {}
+if valgrind:
+    kwargs['COMMAND'] = valgrind
+    kwargs['ARGS'] = ["-q", dec[0].abspath]
+
+env.RunTest("decode_alltypes.output", [dec, "encode_alltypes_pointer.output"], **kwargs)
+
+# Do the same thing with the optional fields present
+env.RunTest("optionals.output", enc, ARGS = ['1'])
+env.Compare(["optionals.output", "$BUILD/alltypes/optionals.output"])
+
+kwargs['ARGS'] = kwargs.get('ARGS', []) + ['1']
+env.RunTest("optionals.decout", [dec, "optionals.output"], **kwargs)
+

tests/alltypes_pointer/alltypes.options

@@ -0,0 +1,3 @@
+# Generate all fields as pointers.
+* type:FT_POINTER
+

tests/alltypes_pointer/decode_alltypes_pointer.c

@@ -0,0 +1,173 @@
+#include <stdio.h>
+#include <string.h>
+#include <stdlib.h>
+#include <pb_decode.h>
+#include "alltypes.pb.h"
+#include "test_helpers.h"
+
+#define TEST(x) if (!(x)) { \
+    fprintf(stderr, "Test " #x " failed.\n"); \
+    status = false; \
+    }
+
+/* This function is called once from main(), it handles
+   the decoding and checks the fields. */
+bool check_alltypes(pb_istream_t *stream, int mode)
+{
+    bool status = true;
+    AllTypes alltypes;
+    
+    /* Fill with garbage to better detect initialization errors */
+    memset(&alltypes, 0xAA, sizeof(alltypes));
+    
+    if (!pb_decode(stream, AllTypes_fields, &alltypes))
+        return false;
+    
+    TEST(alltypes.req_int32     && *alltypes.req_int32         == -1001);
+    TEST(alltypes.req_int64     && *alltypes.req_int64         == -1002);
+    TEST(alltypes.req_uint32    && *alltypes.req_uint32        == 1003);
+    TEST(alltypes.req_uint64    && *alltypes.req_uint64        == 1004);
+    TEST(alltypes.req_sint32    && *alltypes.req_sint32        == -1005);
+    TEST(alltypes.req_sint64    && *alltypes.req_sint64        == -1006);
+    TEST(alltypes.req_bool      && *alltypes.req_bool          == true);
+    
+    TEST(alltypes.req_fixed32   && *alltypes.req_fixed32       == 1008);
+    TEST(alltypes.req_sfixed32  && *alltypes.req_sfixed32      == -1009);
+    TEST(alltypes.req_float     && *alltypes.req_float         == 1010.0f);
+    
+    TEST(alltypes.req_fixed64   && *alltypes.req_fixed64       == 1011);
+    TEST(alltypes.req_sfixed64  && *alltypes.req_sfixed64      == -1012);
+    TEST(alltypes.req_double    && *alltypes.req_double        == 1013.0f);
+    
+    TEST(alltypes.req_string    && strcmp(alltypes.req_string, "1014") == 0);
+    TEST(alltypes.req_bytes     && alltypes.req_bytes->size == 4);
+    TEST(alltypes.req_bytes     && memcmp(&alltypes.req_bytes->bytes, "1015", 4) == 0);
+    TEST(alltypes.req_submsg    && alltypes.req_submsg->substuff1
+                                && strcmp(alltypes.req_submsg->substuff1, "1016") == 0);
+    TEST(alltypes.req_submsg    && alltypes.req_submsg->substuff2
+                                && *alltypes.req_submsg->substuff2 == 1016);
+    TEST(*alltypes.req_enum == MyEnum_Truth);
+
+    TEST(alltypes.rep_int32_count == 5 && alltypes.rep_int32[4] == -2001 && alltypes.rep_int32[0] == 0);
+    TEST(alltypes.rep_int64_count == 5 && alltypes.rep_int64[4] == -2002 && alltypes.rep_int64[0] == 0);
+    TEST(alltypes.rep_uint32_count == 5 && alltypes.rep_uint32[4] == 2003 && alltypes.rep_uint32[0] == 0);
+    TEST(alltypes.rep_uint64_count == 5 && alltypes.rep_uint64[4] == 2004 && alltypes.rep_uint64[0] == 0);
+    TEST(alltypes.rep_sint32_count == 5 && alltypes.rep_sint32[4] == -2005 && alltypes.rep_sint32[0] == 0);
+    TEST(alltypes.rep_sint64_count == 5 && alltypes.rep_sint64[4] == -2006 && alltypes.rep_sint64[0] == 0);
+    TEST(alltypes.rep_bool_count == 5 && alltypes.rep_bool[4] == true && alltypes.rep_bool[0] == false);
+    
+    TEST(alltypes.rep_fixed32_count == 5 && alltypes.rep_fixed32[4] == 2008 && alltypes.rep_fixed32[0] == 0);
+    TEST(alltypes.rep_sfixed32_count == 5 && alltypes.rep_sfixed32[4] == -2009 && alltypes.rep_sfixed32[0] == 0);
+    TEST(alltypes.rep_float_count == 5 && alltypes.rep_float[4] == 2010.0f && alltypes.rep_float[0] == 0.0f);
+    
+    TEST(alltypes.rep_fixed64_count == 5 && alltypes.rep_fixed64[4] == 2011 && alltypes.rep_fixed64[0] == 0);
+    TEST(alltypes.rep_sfixed64_count == 5 && alltypes.rep_sfixed64[4] == -2012 && alltypes.rep_sfixed64[0] == 0);
+    TEST(alltypes.rep_double_count == 5 && alltypes.rep_double[4] == 2013.0 && alltypes.rep_double[0] == 0.0);
+    
+    TEST(alltypes.rep_string_count == 5 && strcmp(alltypes.rep_string[4], "2014") == 0 && alltypes.rep_string[0][0] == '\0');
+    TEST(alltypes.rep_bytes_count == 5 && alltypes.rep_bytes[4]->size == 4 && alltypes.rep_bytes[0]->size == 0);
+    TEST(memcmp(&alltypes.rep_bytes[4]->bytes, "2015", 4) == 0);
+
+    TEST(alltypes.rep_submsg_count == 5);
+    TEST(strcmp(alltypes.rep_submsg[4].substuff1, "2016") == 0 && alltypes.rep_submsg[0].substuff1[0] == '\0');
+    TEST(*alltypes.rep_submsg[4].substuff2 == 2016 && *alltypes.rep_submsg[0].substuff2 == 0);
+    TEST(*alltypes.rep_submsg[4].substuff3 == 2016 && alltypes.rep_submsg[0].substuff3 == NULL);
+    
+    TEST(alltypes.rep_enum_count == 5 && alltypes.rep_enum[4] == MyEnum_Truth && alltypes.rep_enum[0] == MyEnum_Zero);
+    TEST(alltypes.rep_emptymsg_count == 5);
+
+    if (mode == 0)
+    {
+        /* Expect that optional values are not present */
+        TEST(alltypes.opt_int32         == NULL);
+        TEST(alltypes.opt_int64         == NULL);
+        TEST(alltypes.opt_uint32        == NULL);
+        TEST(alltypes.opt_uint64        == NULL);
+        TEST(alltypes.opt_sint32        == NULL);
+        TEST(alltypes.opt_sint64        == NULL);
+        TEST(alltypes.opt_bool          == NULL);
+        
+        TEST(alltypes.opt_fixed32       == NULL);
+        TEST(alltypes.opt_sfixed32      == NULL);
+        TEST(alltypes.opt_float         == NULL);
+        TEST(alltypes.opt_fixed64       == NULL);
+        TEST(alltypes.opt_sfixed64      == NULL);
+        TEST(alltypes.opt_double        == NULL);
+        
+        TEST(alltypes.opt_string        == NULL);
+        TEST(alltypes.opt_bytes         == NULL);
+        TEST(alltypes.opt_submsg        == NULL);
+        TEST(alltypes.opt_enum          == NULL);
+    }
+    else
+    {
+        /* Expect filled-in values */
+        TEST(alltypes.opt_int32 && *alltypes.opt_int32      == 3041);
+        TEST(alltypes.opt_int64 && *alltypes.opt_int64      == 3042);
+        TEST(alltypes.opt_uint32 && *alltypes.opt_uint32    == 3043);
+        TEST(alltypes.opt_uint64 && *alltypes.opt_uint64    == 3044);
+        TEST(alltypes.opt_sint32 && *alltypes.opt_sint32    == 3045);
+        TEST(alltypes.opt_sint64 && *alltypes.opt_sint64    == 3046);
+        TEST(alltypes.opt_bool && *alltypes.opt_bool        == true);
+        
+        TEST(alltypes.opt_fixed32 && *alltypes.opt_fixed32  == 3048);
+        TEST(alltypes.opt_sfixed32 && *alltypes.opt_sfixed32== 3049);
+        TEST(alltypes.opt_float && *alltypes.opt_float      == 3050.0f);
+        TEST(alltypes.opt_fixed64 && *alltypes.opt_fixed64  == 3051);
+        TEST(alltypes.opt_sfixed64 && *alltypes.opt_sfixed64== 3052);
+        TEST(alltypes.opt_double && *alltypes.opt_double    == 3053.0);
+        
+        TEST(alltypes.opt_string && strcmp(alltypes.opt_string, "3054") == 0);
+        TEST(alltypes.opt_bytes && alltypes.opt_bytes->size == 4);
+        TEST(alltypes.opt_bytes && memcmp(&alltypes.opt_bytes->bytes, "3055", 4) == 0);
+        TEST(alltypes.opt_submsg && strcmp(alltypes.opt_submsg->substuff1, "3056") == 0);
+        TEST(alltypes.opt_submsg && *alltypes.opt_submsg->substuff2 == 3056);
+        TEST(alltypes.opt_enum && *alltypes.opt_enum == MyEnum_Truth);
+        TEST(alltypes.opt_emptymsg);
+    }
+    
+    TEST(alltypes.req_limits->int32_min && *alltypes.req_limits->int32_min   == INT32_MIN);
+    TEST(alltypes.req_limits->int32_max && *alltypes.req_limits->int32_max   == INT32_MAX);
+    TEST(alltypes.req_limits->uint32_min && *alltypes.req_limits->uint32_min == 0);
+    TEST(alltypes.req_limits->uint32_max && *alltypes.req_limits->uint32_max == UINT32_MAX);
+    TEST(alltypes.req_limits->int64_min && *alltypes.req_limits->int64_min   == INT64_MIN);
+    TEST(alltypes.req_limits->int64_max && *alltypes.req_limits->int64_max   == INT64_MAX);
+    TEST(alltypes.req_limits->uint64_min && *alltypes.req_limits->uint64_min == 0);
+    TEST(alltypes.req_limits->uint64_max && *alltypes.req_limits->uint64_max == UINT64_MAX);
+    TEST(alltypes.req_limits->enum_min && *alltypes.req_limits->enum_min     == HugeEnum_Negative);
+    TEST(alltypes.req_limits->enum_max && *alltypes.req_limits->enum_max     == HugeEnum_Positive);
+    
+    TEST(alltypes.end && *alltypes.end == 1099);
+
+    pb_release(AllTypes_fields, &alltypes);
+
+    return status;
+}
+
+int main(int argc, char **argv)
+{
+    uint8_t buffer[1024];
+    size_t count;
+    pb_istream_t stream;
+    
+    /* Whether to expect the optional values or the default values. */
+    int mode = (argc > 1) ? atoi(argv[1]) : 0;
+    
+    /* Read the data into buffer */
+    SET_BINARY_MODE(stdin);
+    count = fread(buffer, 1, sizeof(buffer), stdin);
+    
+    /* Construct a pb_istream_t for reading from the buffer */
+    stream = pb_istream_from_buffer(buffer, count);
+    
+    /* Decode and verify the message */
+    if (!check_alltypes(&stream, mode))
+    {
+        fprintf(stderr, "Test failed: %s\n", PB_GET_ERROR(&stream));
+        return 1;
+    }
+    else
+    {
+        return 0;
+    }
+}

tests/alltypes_pointer/encode_alltypes_pointer.c

@@ -0,0 +1,188 @@
+/* Attempts to test all the datatypes supported by ProtoBuf.
+ */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <pb_encode.h>
+#include "alltypes.pb.h"
+#include "test_helpers.h"
+
+int main(int argc, char **argv)
+{
+    int mode = (argc > 1) ? atoi(argv[1]) : 0;
+    
+    /* Values for required fields */
+    int32_t     req_int32         = -1001;
+    int64_t     req_int64         = -1002;
+    uint32_t    req_uint32        = 1003;
+    uint64_t    req_uint64        = 1004;
+    int32_t     req_sint32        = -1005;
+    int64_t     req_sint64        = -1006;
+    bool        req_bool          = true;
+    uint32_t    req_fixed32       = 1008;
+    int32_t     req_sfixed32      = -1009;
+    float       req_float         = 1010.0f;
+    uint64_t    req_fixed64       = 1011;
+    int64_t     req_sfixed64      = -1012;
+    double      req_double        = 1013.0;
+    char*       req_string        = "1014";
+    PB_BYTES_ARRAY_T(4) req_bytes = {4, {'1', '0', '1', '5'}};
+    static int32_t req_substuff   = 1016;
+    SubMessage  req_submsg        = {"1016", &req_substuff};
+    MyEnum      req_enum          = MyEnum_Truth;
+    EmptyMessage req_emptymsg     = {0};
+    
+    int32_t     end               = 1099;
+
+    /* Values for repeated fields */
+    int32_t     rep_int32[5]      = {0, 0, 0, 0, -2001};
+    int64_t     rep_int64[5]      = {0, 0, 0, 0, -2002};
+    uint32_t    rep_uint32[5]     = {0, 0, 0, 0, 2003};
+    uint64_t    rep_uint64[5]     = {0, 0, 0, 0, 2004};
+    int32_t     rep_sint32[5]     = {0, 0, 0, 0, -2005};
+    int64_t     rep_sint64[5]     = {0, 0, 0, 0, -2006};
+    bool        rep_bool[5]       = {false, false, false, false, true};
+    uint32_t    rep_fixed32[5]    = {0, 0, 0, 0, 2008};
+    int32_t     rep_sfixed32[5]   = {0, 0, 0, 0, -2009};
+    float       rep_float[5]      = {0, 0, 0, 0, 2010.0f};
+    uint64_t    rep_fixed64[5]    = {0, 0, 0, 0, 2011};
+    int64_t     rep_sfixed64[5]   = {0, 0, 0, 0, -2012};
+    double      rep_double[5]     = {0, 0, 0, 0, 2013.0f};
+    char*       rep_string[5]     = {"", "", "", "", "2014"};
+    static PB_BYTES_ARRAY_T(4) rep_bytes_4 = {4, {'2', '0', '1', '5'}};
+    pb_bytes_array_t *rep_bytes[5]= {NULL, NULL, NULL, NULL, (pb_bytes_array_t*)&rep_bytes_4};
+    static int32_t rep_sub2zero   = 0;
+    static int32_t rep_substuff2  = 2016;
+    static uint32_t rep_substuff3 = 2016;
+    SubMessage  rep_submsg[5]     = {{"", &rep_sub2zero},
+                                     {"", &rep_sub2zero},
+                                     {"", &rep_sub2zero},
+                                     {"", &rep_sub2zero},
+                                     {"2016", &rep_substuff2, &rep_substuff3}};
+    MyEnum      rep_enum[5]       = {0, 0, 0, 0, MyEnum_Truth};
+    EmptyMessage rep_emptymsg[5]  = {{0}, {0}, {0}, {0}, {0}};
+
+    /* Values for optional fields */
+    int32_t     opt_int32         = 3041;
+    int64_t     opt_int64         = 3042;
+    uint32_t    opt_uint32        = 3043;
+    uint64_t    opt_uint64        = 3044;
+    int32_t     opt_sint32        = 3045;
+    int64_t     opt_sint64        = 3046;
+    bool        opt_bool          = true;
+    uint32_t    opt_fixed32       = 3048;
+    int32_t     opt_sfixed32      = 3049;
+    float       opt_float         = 3050.0f;
+    uint64_t    opt_fixed64       = 3051;
+    int64_t     opt_sfixed64      = 3052;
+    double      opt_double        = 3053.0;
+    char*       opt_string        = "3054";
+    PB_BYTES_ARRAY_T(4) opt_bytes = {4, {'3', '0', '5', '5'}};
+    static int32_t opt_substuff   = 3056;
+    SubMessage  opt_submsg        = {"3056", &opt_substuff};
+    MyEnum      opt_enum          = MyEnum_Truth;
+    EmptyMessage opt_emptymsg     = {0};
+
+    /* Values for the Limits message. */
+    static int32_t  int32_min  = INT32_MIN;
+    static int32_t  int32_max  = INT32_MAX;
+    static uint32_t uint32_min = 0;
+    static uint32_t uint32_max = UINT32_MAX;
+    static int64_t  int64_min  = INT64_MIN;
+    static int64_t  int64_max  = INT64_MAX;
+    static uint64_t uint64_min = 0;
+    static uint64_t uint64_max = UINT64_MAX;
+    static HugeEnum enum_min   = HugeEnum_Negative;
+    static HugeEnum enum_max   = HugeEnum_Positive;
+    Limits req_limits = {&int32_min,    &int32_max,
+                         &uint32_min,   &uint32_max,
+                         &int64_min,    &int64_max,
+                         &uint64_min,   &uint64_max,
+                         &enum_min,     &enum_max};
+
+    /* Initialize the message struct with pointers to the fields. */
+    AllTypes alltypes = {0};
+
+    alltypes.req_int32         = &req_int32;
+    alltypes.req_int64         = &req_int64;
+    alltypes.req_uint32        = &req_uint32;
+    alltypes.req_uint64        = &req_uint64;
+    alltypes.req_sint32        = &req_sint32;
+    alltypes.req_sint64        = &req_sint64;
+    alltypes.req_bool          = &req_bool;
+    alltypes.req_fixed32       = &req_fixed32;
+    alltypes.req_sfixed32      = &req_sfixed32;
+    alltypes.req_float         = &req_float;
+    alltypes.req_fixed64       = &req_fixed64;
+    alltypes.req_sfixed64      = &req_sfixed64;
+    alltypes.req_double        = &req_double;
+    alltypes.req_string        = req_string;
+    alltypes.req_bytes         = (pb_bytes_array_t*)&req_bytes;
+    alltypes.req_submsg        = &req_submsg;
+    alltypes.req_enum          = &req_enum;
+    alltypes.req_emptymsg      = &req_emptymsg;
+    alltypes.req_limits        = &req_limits;
+    
+    alltypes.rep_int32_count    = 5; alltypes.rep_int32     = rep_int32;
+    alltypes.rep_int64_count    = 5; alltypes.rep_int64     = rep_int64;
+    alltypes.rep_uint32_count   = 5; alltypes.rep_uint32    = rep_uint32;
+    alltypes.rep_uint64_count   = 5; alltypes.rep_uint64    = rep_uint64;
+    alltypes.rep_sint32_count   = 5; alltypes.rep_sint32    = rep_sint32;
+    alltypes.rep_sint64_count   = 5; alltypes.rep_sint64    = rep_sint64;
+    alltypes.rep_bool_count     = 5; alltypes.rep_bool      = rep_bool;
+    alltypes.rep_fixed32_count  = 5; alltypes.rep_fixed32   = rep_fixed32;
+    alltypes.rep_sfixed32_count = 5; alltypes.rep_sfixed32  = rep_sfixed32;
+    alltypes.rep_float_count    = 5; alltypes.rep_float     = rep_float;
+    alltypes.rep_fixed64_count  = 5; alltypes.rep_fixed64   = rep_fixed64;
+    alltypes.rep_sfixed64_count = 5; alltypes.rep_sfixed64  = rep_sfixed64;
+    alltypes.rep_double_count   = 5; alltypes.rep_double    = rep_double;
+    alltypes.rep_string_count   = 5; alltypes.rep_string    = rep_string;
+    alltypes.rep_bytes_count    = 5; alltypes.rep_bytes     = rep_bytes;
+    alltypes.rep_submsg_count   = 5; alltypes.rep_submsg    = rep_submsg;
+    alltypes.rep_enum_count     = 5; alltypes.rep_enum      = rep_enum;
+    alltypes.rep_emptymsg_count = 5; alltypes.rep_emptymsg  = rep_emptymsg;
+    
+    if (mode != 0)
+    {
+        /* Fill in values for optional fields */
+        alltypes.opt_int32         = &opt_int32;
+        alltypes.opt_int64         = &opt_int64;
+        alltypes.opt_uint32        = &opt_uint32;
+        alltypes.opt_uint64        = &opt_uint64;
+        alltypes.opt_sint32        = &opt_sint32;
+        alltypes.opt_sint64        = &opt_sint64;
+        alltypes.opt_bool          = &opt_bool;
+        alltypes.opt_fixed32       = &opt_fixed32;
+        alltypes.opt_sfixed32      = &opt_sfixed32;
+        alltypes.opt_float         = &opt_float;
+        alltypes.opt_fixed64       = &opt_fixed64;
+        alltypes.opt_sfixed64      = &opt_sfixed64;
+        alltypes.opt_double        = &opt_double;
+        alltypes.opt_string        = opt_string;
+        alltypes.opt_bytes         = (pb_bytes_array_t*)&opt_bytes;
+        alltypes.opt_submsg        = &opt_submsg;
+        alltypes.opt_enum          = &opt_enum;
+        alltypes.opt_emptymsg      = &opt_emptymsg;
+    }
+    
+    alltypes.end = &end;
+    
+    {
+        uint8_t buffer[4096];
+        pb_ostream_t stream = pb_ostream_from_buffer(buffer, sizeof(buffer));
+        
+        /* Now encode it and check if we succeeded. */
+        if (pb_encode(&stream, AllTypes_fields, &alltypes))
+        {
+            SET_BINARY_MODE(stdout);
+            fwrite(buffer, 1, stream.bytes_written, stdout);
+            return 0; /* Success */
+        }
+        else
+        {
+            fprintf(stderr, "Encoding failed: %s\n", PB_GET_ERROR(&stream));
+            return 1; /* Failure */
+        }
+    }
+}

tests/backwards_compatibility/SConscript

@@ -0,0 +1,11 @@
+# Check that the old generated .pb.c/.pb.h files are still compatible with the
+# current version of nanopb.
+
+Import("env")
+
+enc = env.Program(["encode_legacy.c", "alltypes_legacy.c", "$COMMON/pb_encode.o"])
+dec = env.Program(["decode_legacy.c", "alltypes_legacy.c", "$COMMON/pb_decode.o"])
+
+env.RunTest(enc)
+env.RunTest([dec, "encode_legacy.output"])
+

tests/backwards_compatibility/alltypes_legacy.c

@@ -5,7 +5,7 @@
  * incompatible changes made to the generator in future versions.
  */
 
-#include "alltypes.pb.h"
+#include "alltypes_legacy.h"
 
 const char SubMessage_substuff1_default[16] = "1";
 const int32_t SubMessage_substuff2_default = 2;

tests/backwards_compatibility/decode_legacy.c

@@ -1,16 +1,17 @@
 /* Tests the decoding of all types.
- * This is a backwards-compatibility test, using bc_alltypes.pb.h.
+ * This is a backwards-compatibility test, using alltypes_legacy.h.
- * It is similar to test_decode3, but duplicated in order to allow
+ * It is similar to decode_alltypes, but duplicated in order to allow
- * test_decode3 to test any new features introduced later.
+ * decode_alltypes to test any new features introduced later.
  *
- * Run e.g. ./bc_encode | ./bc_decode
+ * Run e.g. ./encode_legacy | ./decode_legacy
  */
 
 #include <stdio.h>
 #include <string.h>
 #include <stdlib.h>
 #include <pb_decode.h>
-#include "bc_alltypes.pb.h"
+#include "alltypes_legacy.h"
+#include "test_helpers.h"
 
 #define TEST(x) if (!(x)) { \
     printf("Test " #x " failed.\n"); \
@@ -176,15 +177,19 @@ bool check_alltypes(pb_istream_t *stream, int mode)
 
 int main(int argc, char **argv)
 {
+    uint8_t buffer[1024];
+    size_t count;
+    pb_istream_t stream;
+
     /* Whether to expect the optional values or the default values. */
     int mode = (argc > 1) ? atoi(argv[1]) : 0;
     
     /* Read the data into buffer */
-    uint8_t buffer[1024];
+    SET_BINARY_MODE(stdin);
-    size_t count = fread(buffer, 1, sizeof(buffer), stdin);
+    count = fread(buffer, 1, sizeof(buffer), stdin);
     
     /* Construct a pb_istream_t for reading from the buffer */
-    pb_istream_t stream = pb_istream_from_buffer(buffer, count);
+    stream = pb_istream_from_buffer(buffer, count);
     
     /* Decode and print out the stuff */
     if (!check_alltypes(&stream, mode))

tests/backwards_compatibility/encode_legacy.c

@@ -1,14 +1,15 @@
 /* Attempts to test all the datatypes supported by ProtoBuf.
- * This is a backwards-compatibility test, using bc_alltypes.pb.h.
+ * This is a backwards-compatibility test, using alltypes_legacy.h.
- * It is similar to test_encode3, but duplicated in order to allow
+ * It is similar to encode_alltypes, but duplicated in order to allow
- * test_encode3 to test any new features introduced later.
+ * encode_alltypes to test any new features introduced later.
  */
 
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
 #include <pb_encode.h>
-#include "bc_alltypes.pb.h"
+#include "alltypes_legacy.h"
+#include "test_helpers.h"
 
 int main(int argc, char **argv)
 {
@@ -114,12 +115,14 @@ int main(int argc, char **argv)
     
     alltypes.end = 1099;
 
+    {    
         uint8_t buffer[1024];
         pb_ostream_t stream = pb_ostream_from_buffer(buffer, sizeof(buffer));
         
         /* Now encode it and check if we succeeded. */
         if (pb_encode(&stream, AllTypes_fields, &alltypes))
         {
+            SET_BINARY_MODE(stdout);
             fwrite(buffer, 1, stream.bytes_written, stdout);
             return 0; /* Success */
         }
@@ -129,3 +132,4 @@ int main(int argc, char **argv)
             return 1; /* Failure */
         }
     }
+}

tests/basic_buffer/SConscript

@@ -0,0 +1,12 @@
+# Build and run a basic round-trip test using memory buffer encoding.
+
+Import("env")
+
+enc = env.Program(["encode_buffer.c", "$COMMON/person.pb.c", "$COMMON/pb_encode.o"])
+dec = env.Program(["decode_buffer.c", "$COMMON/person.pb.c", "$COMMON/pb_decode.o"])
+
+env.RunTest(enc)
+env.RunTest([dec, "encode_buffer.output"])
+env.Decode(["encode_buffer.output", "$COMMON/person.proto"], MESSAGE = "Person")
+env.Compare(["decode_buffer.output", "encode_buffer.decoded"])
+

tests/basic_buffer/decode_buffer.c

@@ -9,6 +9,7 @@
 #include <stdio.h>
 #include <pb_decode.h>
 #include "person.pb.h"
+#include "test_helpers.h"
 
 /* This function is called once from main(), it handles
    the decoding and printing. */
@@ -59,9 +60,13 @@ bool print_person(pb_istream_t *stream)
 
 int main()
 {
+    uint8_t buffer[Person_size];
+    pb_istream_t stream;
+    size_t count;
+    
     /* Read the data into buffer */
-    uint8_t buffer[512];
+    SET_BINARY_MODE(stdin);
-    size_t count = fread(buffer, 1, sizeof(buffer), stdin);
+    count = fread(buffer, 1, sizeof(buffer), stdin);
     
     if (!feof(stdin))
     {
@@ -70,7 +75,7 @@ int main()
     }
     
     /* Construct a pb_istream_t for reading from the buffer */
-    pb_istream_t stream = pb_istream_from_buffer(buffer, count);
+    stream = pb_istream_from_buffer(buffer, count);
     
     /* Decode and print out the stuff */
     if (!print_person(&stream))

tests/basic_buffer/encode_buffer.c

@@ -6,9 +6,13 @@
 #include <stdio.h>
 #include <pb_encode.h>
 #include "person.pb.h"
+#include "test_helpers.h"
 
 int main()
 {
+    uint8_t buffer[Person_size];
+    pb_ostream_t stream;
+    
     /* Initialize the structure with constants */
     Person person = {"Test Person 99", 99, true, "test@person.com",
         3, {{"555-12345678", true, Person_PhoneType_MOBILE},
@@ -16,17 +20,19 @@ int main()
             {"1234-5678", true, Person_PhoneType_WORK},
         }};
 
-    uint8_t buffer[512];
+    stream = pb_ostream_from_buffer(buffer, sizeof(buffer));
-    pb_ostream_t stream = pb_ostream_from_buffer(buffer, sizeof(buffer));
     
     /* Now encode it and check if we succeeded. */
     if (pb_encode(&stream, Person_fields, &person))
     {    
+        /* Write the result data to stdout */
+        SET_BINARY_MODE(stdout);
         fwrite(buffer, 1, stream.bytes_written, stdout);
         return 0; /* Success */
     }
     else
     {
+        fprintf(stderr, "Encoding failed: %s\n", PB_GET_ERROR(&stream));
         return 1; /* Failure */
     }
 }

tests/basic_stream/SConscript

@@ -0,0 +1,12 @@
+# Build and run a basic round-trip test using direct stream encoding.
+
+Import("env")
+
+enc = env.Program(["encode_stream.c", "$COMMON/person.pb.c", "$COMMON/pb_encode.o"])
+dec = env.Program(["decode_stream.c", "$COMMON/person.pb.c", "$COMMON/pb_decode.o"])
+
+env.RunTest(enc)
+env.RunTest([dec, "encode_stream.output"])
+env.Decode(["encode_stream.output", "$COMMON/person.proto"], MESSAGE = "Person")
+env.Compare(["decode_stream.output", "encode_stream.decoded"])
+

tests/basic_stream/decode_stream.c

@@ -4,6 +4,7 @@
 #include <stdio.h>
 #include <pb_decode.h>
 #include "person.pb.h"
+#include "test_helpers.h"
 
 /* This function is called once from main(), it handles
    the decoding and printing.
@@ -69,10 +70,10 @@ bool callback(pb_istream_t *stream, uint8_t *buf, size_t count)
 
 int main()
 {
-    /* Maximum size is specified to prevent infinite length messages from
+    pb_istream_t stream = {&callback, NULL, SIZE_MAX};
-     * hanging this in the fuzz test.
+    stream.state = stdin;
-     */
+    SET_BINARY_MODE(stdin);
-    pb_istream_t stream = {&callback, stdin, 10000};
+
     if (!print_person(&stream))
     {
         printf("Parsing failed: %s\n", PB_GET_ERROR(&stream));

tests/basic_stream/encode_stream.c

@@ -4,6 +4,7 @@
 #include <stdio.h>
 #include <pb_encode.h>
 #include "person.pb.h"
+#include "test_helpers.h"
 
 /* This binds the pb_ostream_t into the stdout stream */
 bool streamcallback(pb_ostream_t *stream, const uint8_t *buf, size_t count)
@@ -22,11 +23,18 @@ int main()
         }};
     
     /* Prepare the stream, output goes directly to stdout */
-    pb_ostream_t stream = {&streamcallback, stdout, SIZE_MAX, 0};
+    pb_ostream_t stream = {&streamcallback, NULL, SIZE_MAX, 0};
+    stream.state = stdout;
+    SET_BINARY_MODE(stdout);
     
     /* Now encode it and check if we succeeded. */
     if (pb_encode(&stream, Person_fields, &person))
+    {
         return 0; /* Success */
+    }
     else
+    {
+        fprintf(stderr, "Encoding failed: %s\n", PB_GET_ERROR(&stream));
         return 1; /* Failure */
     }
+}

tests/buffer_only/SConscript

@@ -0,0 +1,23 @@
+# Run the alltypes test case, but compile with PB_BUFFER_ONLY=1
+
+Import("env")
+
+# Take copy of the files for custom build.
+c = Copy("$TARGET", "$SOURCE")
+env.Command("pb_encode.c", "#../pb_encode.c", c)
+env.Command("pb_decode.c", "#../pb_decode.c", c)
+env.Command("alltypes.pb.h", "$BUILD/alltypes/alltypes.pb.h", c)
+env.Command("alltypes.pb.c", "$BUILD/alltypes/alltypes.pb.c", c)
+env.Command("encode_alltypes.c", "$BUILD/alltypes/encode_alltypes.c", c)
+env.Command("decode_alltypes.c", "$BUILD/alltypes/decode_alltypes.c", c)
+
+# Define the compilation options
+opts = env.Clone()
+opts.Append(CPPDEFINES = {'PB_BUFFER_ONLY': 1})
+
+# Now build and run the test normally.
+enc = opts.Program(["encode_alltypes.c", "alltypes.pb.c", "pb_encode.c"])
+dec = opts.Program(["decode_alltypes.c", "alltypes.pb.c", "pb_decode.c"])
+
+env.RunTest(enc)
+env.RunTest([dec, "encode_alltypes.output"])

tests/callbacks/SConscript

@@ -0,0 +1,14 @@
+# Test the functionality of the callback fields.
+
+Import("env")
+
+env.NanopbProto("callbacks")
+enc = env.Program(["encode_callbacks.c", "callbacks.pb.c", "$COMMON/pb_encode.o"])
+dec = env.Program(["decode_callbacks.c", "callbacks.pb.c", "$COMMON/pb_decode.o"])
+
+env.RunTest(enc)
+env.RunTest([dec, "encode_callbacks.output"])
+
+env.Decode(["encode_callbacks.output", "callbacks.proto"], MESSAGE = "TestMessage")
+env.Compare(["decode_callbacks.output", "encode_callbacks.decoded"])
+

tests/callbacks/callbacks.proto

@@ -11,5 +11,6 @@ message TestMessage {
     repeated fixed32 fixed32value = 3;
     repeated fixed64 fixed64value = 4;
     optional SubMessage submsg = 5;
+    repeated string repeatedstring = 6;
 }
 

tests/callbacks/decode_callbacks.c

@@ -5,8 +5,9 @@
 #include <stdio.h>
 #include <pb_decode.h>
 #include "callbacks.pb.h"
+#include "test_helpers.h"
 
-bool print_string(pb_istream_t *stream, const pb_field_t *field, void *arg)
+bool print_string(pb_istream_t *stream, const pb_field_t *field, void **arg)
 {
     uint8_t buffer[1024] = {0};
     
@@ -20,51 +21,54 @@ bool print_string(pb_istream_t *stream, const pb_field_t *field, void *arg)
     /* Print the string, in format comparable with protoc --decode.
      * Format comes from the arg defined in main().
      */
-    printf((char*)arg, buffer);
+    printf((char*)*arg, buffer);
     return true;
 }
 
-bool print_int32(pb_istream_t *stream, const pb_field_t *field, void *arg)
+bool print_int32(pb_istream_t *stream, const pb_field_t *field, void **arg)
 {
     uint64_t value;
     if (!pb_decode_varint(stream, &value))
         return false;
     
-    printf((char*)arg, (long)value);
+    printf((char*)*arg, (long)value);
     return true;
 }
 
-bool print_fixed32(pb_istream_t *stream, const pb_field_t *field, void *arg)
+bool print_fixed32(pb_istream_t *stream, const pb_field_t *field, void **arg)
 {
     uint32_t value;
     if (!pb_decode_fixed32(stream, &value))
         return false;
     
-    printf((char*)arg, (long)value);
+    printf((char*)*arg, (long)value);
     return true;
 }
 
-bool print_fixed64(pb_istream_t *stream, const pb_field_t *field, void *arg)
+bool print_fixed64(pb_istream_t *stream, const pb_field_t *field, void **arg)
 {
     uint64_t value;
     if (!pb_decode_fixed64(stream, &value))
         return false;
     
-    printf((char*)arg, (long long)value);
+    printf((char*)*arg, (long)value);
     return true;
 }
 
 int main()
 {
     uint8_t buffer[1024];
-    size_t length = fread(buffer, 1, 1024, stdin);
+    size_t length;
-    pb_istream_t stream = pb_istream_from_buffer(buffer, length);
+    pb_istream_t stream;
-    
     /* Note: empty initializer list initializes the struct with all-0.
      * This is recommended so that unused callbacks are set to NULL instead
      * of crashing at runtime.
      */
-    TestMessage testmessage = {};
+    TestMessage testmessage = {{{NULL}}};
+    
+    SET_BINARY_MODE(stdin);
+    length = fread(buffer, 1, 1024, stdin);
+    stream = pb_istream_from_buffer(buffer, length);    
     
     testmessage.submsg.stringvalue.funcs.decode = &print_string;
     testmessage.submsg.stringvalue.arg = "submsg {\n  stringvalue: \"%s\"\n";
@@ -73,7 +77,7 @@ int main()
     testmessage.submsg.fixed32value.funcs.decode = &print_fixed32;
     testmessage.submsg.fixed32value.arg = "  fixed32value: %ld\n";
     testmessage.submsg.fixed64value.funcs.decode = &print_fixed64;
-    testmessage.submsg.fixed64value.arg = "  fixed64value: %lld\n}\n";
+    testmessage.submsg.fixed64value.arg = "  fixed64value: %ld\n}\n";
     
     testmessage.stringvalue.funcs.decode = &print_string;
     testmessage.stringvalue.arg = "stringvalue: \"%s\"\n";
@@ -82,7 +86,9 @@ int main()
     testmessage.fixed32value.funcs.decode = &print_fixed32;
     testmessage.fixed32value.arg = "fixed32value: %ld\n";
     testmessage.fixed64value.funcs.decode = &print_fixed64;
-    testmessage.fixed64value.arg = "fixed64value: %lld\n";
+    testmessage.fixed64value.arg = "fixed64value: %ld\n";
+    testmessage.repeatedstring.funcs.decode = &print_string;
+    testmessage.repeatedstring.arg = "repeatedstring: \"%s\"\n";
     
     if (!pb_decode(&stream, TestMessage_fields, &testmessage))
         return 1;

tests/callbacks/encode_callbacks.c

@@ -4,8 +4,9 @@
 #include <string.h>
 #include <pb_encode.h>
 #include "callbacks.pb.h"
+#include "test_helpers.h"
 
-bool encode_string(pb_ostream_t *stream, const pb_field_t *field, const void *arg)
+bool encode_string(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
 {
     char *str = "Hello world!";
     
@@ -15,7 +16,7 @@ bool encode_string(pb_ostream_t *stream, const pb_field_t *field, const void *ar
     return pb_encode_string(stream, (uint8_t*)str, strlen(str));
 }
 
-bool encode_int32(pb_ostream_t *stream, const pb_field_t *field, const void *arg)
+bool encode_int32(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
 {
     if (!pb_encode_tag_for_field(stream, field))
         return false;
@@ -23,29 +24,49 @@ bool encode_int32(pb_ostream_t *stream, const pb_field_t *field, const void *arg
     return pb_encode_varint(stream, 42);
 }
 
-bool encode_fixed32(pb_ostream_t *stream, const pb_field_t *field, const void *arg)
+bool encode_fixed32(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
 {
+    uint32_t value = 42;
+
     if (!pb_encode_tag_for_field(stream, field))
         return false;
     
-    uint32_t value = 42;
     return pb_encode_fixed32(stream, &value);
 }
 
-bool encode_fixed64(pb_ostream_t *stream, const pb_field_t *field, const void *arg)
+bool encode_fixed64(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    uint64_t value = 42;
+
+    if (!pb_encode_tag_for_field(stream, field))
+        return false;
+    
+    return pb_encode_fixed64(stream, &value);
+}
+
+bool encode_repeatedstring(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
+{
+    char *str[4] = {"Hello world!", "", "Test", "Test2"};
+    int i;
+    
+    for (i = 0; i < 4; i++)
     {
         if (!pb_encode_tag_for_field(stream, field))
             return false;
         
-    uint64_t value = 42;
+        if (!pb_encode_string(stream, (uint8_t*)str[i], strlen(str[i])))
-    return pb_encode_fixed64(stream, &value);
+            return false;
+    }
+    return true;
 }
 
 int main()
 {
     uint8_t buffer[1024];
-    pb_ostream_t stream = pb_ostream_from_buffer(buffer, 1024);
+    pb_ostream_t stream;
-    TestMessage testmessage = {};
+    TestMessage testmessage = {{{NULL}}};
+    
+    stream = pb_ostream_from_buffer(buffer, 1024);
     
     testmessage.stringvalue.funcs.encode = &encode_string;
     testmessage.int32value.funcs.encode = &encode_int32;
@@ -58,9 +79,12 @@ int main()
     testmessage.submsg.fixed32value.funcs.encode = &encode_fixed32;
     testmessage.submsg.fixed64value.funcs.encode = &encode_fixed64;
 
+    testmessage.repeatedstring.funcs.encode = &encode_repeatedstring;
+    
     if (!pb_encode(&stream, TestMessage_fields, &testmessage))
         return 1;
     
+    SET_BINARY_MODE(stdout);
     if (fwrite(buffer, stream.bytes_written, 1, stdout) != 1)
         return 2;
     

tests/common/SConscript

@@ -0,0 +1,17 @@
+# Build the common files needed by multiple test cases
+
+Import('env')
+
+# Protocol definitions for the encode/decode_unittests
+env.NanopbProto("unittestproto")
+
+# Protocol definitions for basic_buffer/stream tests
+env.NanopbProto("person")
+
+# Binaries of the pb_decode.c and pb_encode.c
+# These are built using more strict warning flags.
+strict = env.Clone()
+strict.Append(CFLAGS = strict['CORECFLAGS'])
+strict.Object("pb_decode.o", "$NANOPB/pb_decode.c")
+strict.Object("pb_encode.o", "$NANOPB/pb_encode.c")
+

tests/common/test_helpers.h

@@ -0,0 +1,17 @@
+/* Compatibility helpers for the test programs. */
+
+#ifndef _TEST_HELPERS_H_
+#define _TEST_HELPERS_H_
+
+#ifdef _WIN32
+#include <io.h>
+#include <fcntl.h>
+#define SET_BINARY_MODE(file) setmode(fileno(file), O_BINARY)
+
+#else
+#define SET_BINARY_MODE(file)
+
+#endif
+
+
+#endif

tests/common/unittestproto.proto

@@ -8,6 +8,14 @@ message FloatArray {
     repeated float data = 1 [(nanopb).max_count = 10];
 }
 
+message StringMessage {
+    required string data = 1 [(nanopb).max_size = 10];
+}
+
+message BytesMessage {
+    required bytes data = 1 [(nanopb).max_size = 16];
+}
+
 message CallbackArray {
     // We cheat a bit and use this message for testing other types, too.
     // Nanopb does not care about the actual defined data type for callback

tests/cxx_main_program/SConscript

@@ -0,0 +1,24 @@
+# Run the alltypes test case, but compile it as C++ instead.
+# In fact, compile the entire nanopb using C++ compiler.
+
+Import("env")
+
+# This is needed to get INT32_MIN etc. macros defined
+env = env.Clone()
+env.Append(CPPDEFINES = ['__STDC_LIMIT_MACROS'])
+
+# Copy the files to .cxx extension in order to force C++ build.
+c = Copy("$TARGET", "$SOURCE")
+env.Command("pb_encode.cxx", "#../pb_encode.c", c)
+env.Command("pb_decode.cxx", "#../pb_decode.c", c)
+env.Command("alltypes.pb.h", "$BUILD/alltypes/alltypes.pb.h", c)
+env.Command("alltypes.pb.cxx", "$BUILD/alltypes/alltypes.pb.c", c)
+env.Command("encode_alltypes.cxx", "$BUILD/alltypes/encode_alltypes.c", c)
+env.Command("decode_alltypes.cxx", "$BUILD/alltypes/decode_alltypes.c", c)
+
+# Now build and run the test normally.
+enc = env.Program(["encode_alltypes.cxx", "alltypes.pb.cxx", "pb_encode.cxx"])
+dec = env.Program(["decode_alltypes.cxx", "alltypes.pb.cxx", "pb_decode.cxx"])
+
+env.RunTest(enc)
+env.RunTest([dec, "encode_alltypes.output"])

tests/decode_unittests/SConscript

@@ -0,0 +1,4 @@
+Import('env')
+p = env.Program(["decode_unittests.c", "$COMMON/unittestproto.pb.c"])
+env.RunTest(p)
+

tests/decode_unittests/decode_unittests.c

@@ -1,8 +1,8 @@
-#define NANOPB_INTERNALS
+/* This includes the whole .c file to get access to static functions. */
+#include "pb_decode.c"
 
 #include <stdio.h>
 #include <string.h>
-#include "pb_decode.h"
 #include "unittests.h"
 #include "unittestproto.pb.h"
 
@@ -19,11 +19,11 @@ bool stream_callback(pb_istream_t *stream, uint8_t *buf, size_t count)
 }
 
 /* Verifies that the stream passed to callback matches the byte array pointed to by arg. */
-bool callback_check(pb_istream_t *stream, const pb_field_t *field, void *arg)
+bool callback_check(pb_istream_t *stream, const pb_field_t *field, void **arg)
 {
     int i;
     uint8_t byte;
-    pb_bytes_array_t *ref = (pb_bytes_array_t*) arg;
+    pb_bytes_array_t *ref = (pb_bytes_array_t*) *arg;
     
     for (i = 0; i < ref->size; i++)
     {
@@ -289,6 +289,16 @@ int main()
         TEST((s = S("\x08"), !pb_decode(&s, IntegerArray_fields, &dest)))
     }
     
+    {
+        pb_istream_t s;
+        IntegerContainer dest = {{0}};
+        
+        COMMENT("Testing pb_decode_delimited")
+        TEST((s = S("\x09\x0A\x07\x0A\x05\x01\x02\x03\x04\x05"),
+              pb_decode_delimited(&s, IntegerContainer_fields, &dest)) &&
+              dest.submsg.data_count == 5)
+    }
+    
     if (status != 0)
         fprintf(stdout, "\n\nSome tests FAILED!\n");
     

tests/encode_unittests/SConscript

@@ -0,0 +1,5 @@
+# Build and run the stand-alone unit tests for the nanopb encoder part.
+
+Import('env')
+p = env.Program(["encode_unittests.c", "$COMMON/unittestproto.pb.c"])
+env.RunTest(p)